Live Breaking News & Updates on Olah Carter

Many new Transformer architecture improvements have been proposed since my last post on “The Transformer Family” about three years ago. Here I did a big refactoring and enrichment of that 2020 post — restructure the hierarchy of sections and improve many sections with more recent papers. Version 2.0 is a superset of the old version, about twice the length.
Notations Symbol Meaning $d$ The model size / hidden state dimension / positional encoding size. ....

Mostafa Dehghani , Olah Carter , Emilio Parisotto , Sainbayar Sukhbaatar , Alex Graves , Longformer Beltagy , Niki Parmar , Ashish Vaswani , Nikita Kitaev , Zihang Dai , Linformer Wang , Rahimi Recht , Aidann Gomez , Adaptive Computation Time For Recurrent Neural Networks , A Survey , Recurrent Neural Networks , Rotary Position Embedding , Memorizing Transformer , Aware Transformer , Linear Biases , Universal Transformer , Adaptive Attention , Adaptive Computation Time , Depth Adaptive Transformer , Confident Adaptive Language Model , Efficient Transformers ,