Picture standing on a hillside in dense fog, trying to reach the valley floor. You cannot see the terrain, but you can feel which way the ground slopes under your feet. So you take a step downhill, feel again, step again. That is gradient descent in full. The loss function tells you your altitude, the gradient tells you the slope, and the learning rate decides how big a step you take. Repeat a few million times.
The variants come down to how much data you look at before each step. Batch gradient descent computes the gradient over the entire training set, which is precise and hopelessly slow. Stochastic gradient descent uses a single example, which is fast and extremely noisy. Mini-batch, the version everyone actually uses, takes a few hundred or few thousand examples at a time, and it won mostly because it maps cleanly onto how GPUs work. Layered on top are optimizers that make the steps smarter: momentum carries velocity through flat regions, and Adam and AdamW adapt the step size per parameter, which is why they are the default for training language models. Learning rate schedules with a warmup phase and a cosine decay are similarly standard, because a large early step in a randomly initialized network is a good way to produce a model that never recovers.
The scale is what makes this strange to think about. A 70-billion-parameter model is a point in a 70-billion-dimensional space, and gradient descent adjusts every one of those coordinates on every step, for millions of steps, across trillions of tokens. Backpropagation is what computes the gradient efficiently; gradient descent is what does something with it. When people say a model was trained, this loop is the entirety of what they mean.
The failure modes are practical rather than exotic. Too large a learning rate and the loss diverges, sometimes catastrophically and irreversibly. Too small and training crawls while burning compute. Loss spikes mid-run are common enough at scale that gradient clipping is standard practice and checkpoint rollback is a normal part of operating a large training run. The textbook worry about getting trapped in local minima turns out to matter far less in very high dimensions than the textbooks suggested, though long plateaus are real and expensive.
Here is why this belongs in the vocabulary of engineers who will never train a model. Gradient descent explains what a model actually is: a fixed point in parameter space, found by averaging over a training corpus, frozen the moment training stops. Everything a model knows was compressed into those weights by this process, and nothing that happens at inference time changes a single one of them. That is the whole reason fine-tuning and LoRA exist as separate operations, and it is the reason a model cannot learn a fact from a conversation. Persistence has to be built outside the weights, which is precisely the gap that retrieval and memory systems fill.
Why It Matters
Understanding gradient descent is what makes the boundary between training and inference concrete rather than abstract. It explains why models have knowledge cutoffs, why they cannot remember yesterday, why fine-tuning is expensive and slow to iterate on, and why teams that try to solve a personalization problem by retraining almost always end up rebuilding it as retrieval instead. For anyone deciding between fine-tuning, prompting, and a memory layer, the difference is exactly the difference between changing weights and changing context.
Example
A team wants their assistant to know current inventory levels. Fine-tuning is proposed: gather the catalog, run a training job, ship the new weights. It works for about a day. Inventory changes hourly, and every change would require another training run at real cost and real latency, with no way to remove a stale fact once gradient descent has smeared it across billions of weights. They switch to retrieval, where updating a fact is a database write. Gradient descent is how a model acquires durable general capability; it is a terrible mechanism for anything that changes faster than a training cycle.