Improving LoRA: Implementing Weight-Decomposed Low-Rank Adaptation (DoRA) from Scratch
This article implements LoRA (low-rank adaptation), an parameter-efficient finetuning technique for LLMs from scratch and discussed the newest and most promising variant: DoRA (Weight-Decomposed Low-Rank Adaptation).
Source: sebastianraschka.com