# Backpropagation

**TL;DR:** The algorithm that computes how much each parameter contributed to a model error, by applying the chain rule backwards through the network.

Training needs to know, for every parameter in the model, whether nudging it up or down would reduce the error. For a 70-billion-parameter model that is 70 billion partial derivatives, and the naive way to get them is to perturb each parameter and re-run the network, which would take 70 billion forward passes per training step. Backpropagation gets all of them in a single backward pass. That efficiency is the entire reason deep learning is computationally possible.

The mechanism is the chain rule applied systematically. A forward pass runs input through the network and produces a loss. The backward pass then walks the computation graph in reverse, and at each operation it takes the gradient flowing in from above and multiplies it by that operation local derivative, passing the result further back. Each layer only needs to know its own derivative and what arrived from downstream. Nothing about the algorithm is specific to neural networks; it is generic reverse-mode automatic differentiation, which is why frameworks like PyTorch implement it once for arbitrary graphs.

The cost is memory, and this is the practical detail most people meet first without knowing its name. The backward pass needs the activations computed during the forward pass, so they all have to be retained until the gradient reaches them. That is why training a model requires several times more memory than running it: weights, gradients, optimizer state, and stored activations all coexist. Gradient checkpointing trades compute back for memory by discarding some activations and recomputing them on the way down, and it is standard practice on anything large.

Backpropagation also had a long-standing failure mode worth knowing about, because the fixes are visible in every modern architecture. Gradients that pass through many layers get multiplied repeatedly and either shrink toward zero or explode, which made deep networks untrainable for years. Residual connections give gradients a direct path backwards, layer normalization keeps activations in a stable range, and gradient clipping caps the magnitude of any single update. Those three are in transformers for exactly this reason.

For application engineers, the useful takeaway is the division of labor and its consequence. Backpropagation finds the direction; gradient descent takes the step. Both happen only during training, both are absent at inference, and together they explain the hardware asymmetry that shapes every fine-tuning decision: serving a model needs room for weights and cache, while training it needs several times that, which is precisely the constraint LoRA was designed around.

## Why it matters

Backpropagation explains the memory wall that makes fine-tuning expensive and inference comparatively cheap, which is the practical fact behind most build decisions in this area. It also makes the training-inference boundary concrete: a model updates its weights only inside this loop, so anything that needs to change after deployment has to live outside the weights entirely. Teams that internalize this stop proposing retraining as a solution to problems that are really about state.

## Example

A team wants to fine-tune a 13B model and provisions a single 24 GB GPU, reasoning that the model runs comfortably in about 26 GB at fp16 and they can shave it. Training immediately runs out of memory, because full fine-tuning also needs gradients for every parameter, optimizer state that is typically twice the parameter count again, and retained activations for the backward pass, landing somewhere north of 200 GB. Switching to 4-bit QLoRA drops trainable parameters by more than two orders of magnitude and the job fits on the card they already have.

## Related terms

- [Gradient Descent](https://www.maximem.ai/glossary/gradient-descent)
- [Loss Function](https://www.maximem.ai/glossary/loss-function)
- [Fine-Tuning](https://www.maximem.ai/glossary/fine-tuning)
- [LoRA (Low-Rank Adaptation)](https://www.maximem.ai/glossary/lora)
- [Transformer Architecture](https://www.maximem.ai/glossary/transformer-architecture)
- [LLM (Large Language Model)](https://www.maximem.ai/glossary/llm)
- [Inference](https://www.maximem.ai/glossary/inference)

---

Source: [https://www.maximem.ai/glossary/backpropagation](https://www.maximem.ai/glossary/backpropagation)