Notes on training BERT from scratch on an 8GB consumer GPU

I trained a BERT model (Devlin et al, 2019) from scratch on my desktop PC (which has a Nvidia 3060 Ti 8GB GPU). The model architecture, tokenizer, and trainer all came from Hugging Face libraries, and my contribution was mainly setting up the code, setting up the data (~20GB uncompressed text), and leaving my computer running. (And making sure it was working correctly, with good GPU utilization.)

Related Keywords

Karthik Narasimhan ,Aidann Gomez ,Noam Shazeer ,Jacob Devlin ,Alec Radford ,Lukasz Kaiser ,Ashish Vaswani ,Illia Polosukhin ,Niki Parmar ,Ming Wei Chang ,Ilya Sutskever ,Tim Salimans ,Kristina Toutanova ,Kenton Lee ,Jakob Uszkoreit ,Nvidia ,Hugging Face ,Deep Bidirectional Transformers ,Llion Jones ,Attention Is All You ,