Full fine-tuning updates every weight in the model, which means holding gradients and optimizer state for all of them. LoRA starts from an empirical observation: the update a fine-tune applies to a weight matrix is usually low-rank, meaning it can be approximated well by the product of two much smaller matrices. So instead of learning the full update, you freeze the original weights entirely and learn only that small pair, typically at rank 8 to 64. At inference the two paths are added together, or the adapter is merged into the base weights so there is no latency cost at all.
The numbers are the argument. A 7B model full fine-tune needs on the order of 60 to 80 GB once gradients and optimizer state are counted, which puts it on datacenter hardware. LoRA trains something like 0.1 to 1 percent of the parameters and fits comfortably on a single consumer GPU, and QLoRA, which pairs a 4-bit quantized frozen base with LoRA adapters on top, pushes a 70B fine-tune onto hardware that would not have held the model in fp16 at all. The resulting adapter is measured in megabytes rather than gigabytes.
That size difference changes deployment shape, not just training cost. Because adapters are small and the base is shared, you can serve one copy of a base model and swap adapters per request, which makes per-tenant or per-task customization economically sensible in a way full fine-tuning never was. Hosting a hundred customer-specific fine-tunes as a hundred full models is absurd; hosting them as a hundred adapters over one base is routine.
The limits matter as much as the capability, and they are consistently misunderstood. LoRA is very good at teaching form: output format, tone, domain vocabulary, the shape of a task, adherence to a house style. It is a poor and expensive way to teach facts. Knowledge injected through any fine-tune is diffused across weights, cannot be updated without another training run, cannot be removed on request, and carries no provenance, so nothing can cite it or audit it. The recurring mistake is fine-tuning to install information that changes weekly, then discovering there is no delete operation. Facts belong in retrieval and memory; fine-tuning is for behavior.
Why It Matters
LoRA is what made fine-tuning accessible outside well-funded labs, and it is the reason per-customer model customization is a normal product feature rather than an enterprise-only line item. Knowing where it applies is equally valuable: the choice between fine-tuning and retrieval is not about cost anymore, it is about whether the thing you are adding is a behavior or a fact. Teams that get that boundary right ship faster, and teams that get it wrong end up with models holding stale information they cannot remove.
Example
A legal tech company needs their assistant to draft in a specific firm house style, cite in a particular format, and know the current status of thousands of active matters. They try to do all three with one fine-tune and it goes badly: style improves, matter status is wrong within days, and a client asks for their data to be removed with no mechanism to comply. The split that works is a LoRA adapter for style and citation format, retrained quarterly, plus retrieval over a live matter database for anything factual. Style lives in weights, facts live where they can be updated and deleted.