The Transformer Family Version 2.0 : vimarsana.com

Many new Transformer architecture improvements have been proposed since my last post on “The Transformer Family” about three years ago. Here I did a big refactoring and enrichment of that 2020 post — restructure the hierarchy of sections and improve many sections with more recent papers. Version 2.0 is a superset of the old version, about twice the length.
Notations Symbol Meaning $d$ The model size / hidden state dimension / positional encoding size.

Related Keywords

Mostafa Dehghani , Olah Carter , Emilio Parisotto , Sainbayar Sukhbaatar , Alex Graves , Longformer Beltagy , Niki Parmar , Ashish Vaswani , Nikita Kitaev , Zihang Dai , Linformer Wang , Rahimi Recht , Aidann Gomez , Adaptive Computation Time For Recurrent Neural Networks , A Survey , Recurrent Neural Networks , Rotary Position Embedding , Memorizing Transformer , Aware Transformer , Linear Biases , Universal Transformer , Adaptive Attention , Adaptive Computation Time , Depth Adaptive Transformer , Confident Adaptive Language Model , Efficient Transformers , Image Transformer , Local Attention , Sparse Transformer , Factorized Self Attention , Blockwise Attention , Extended Transformer Construction , Big Bird , Locality Sensitive Hashing , Routing Transformer , Feature Attention , Reinforcement Learning , Gated Recurrent Unit , Augmented Recurrent Neural , Attention Span , Long Sequences , Computation Time , Recurrent Neural , Attentive Language Models Beyond , Fixed Length Context , Reversible Residual Network , Backpropagation Without Storing , Long Range Sequence Modelling , Test Long , Distance Aware , Adaptive Transformer , Adaptive Language , Content Based Sparse Attention , Routing Transformers , Encoding Long , Structured Inputs , Longer Sequences , Linear Complexity , Sinkhorn Attention , Architecture , Attention , Transformer , Foundation , Long Read ,