Transformer Math 101
We present basic math related to computation and memory usage for transformers
Oak Ridge National Lab Summit Computing Costs Oak Ridge National Lab Reducing Activation Recomputation Large Transformer Models Sharded Optimizers
Source: blog.eleuther.ai