Attention is the idea that made transformers work, and the mechanics are simpler than the reputation suggests. Every token produces three vectors: a query representing what it is looking for, a key representing what it offers, and a value carrying its actual content. To process a token, the model compares its query against every key, turns those comparison scores into weights with a softmax, and takes a weighted sum of the corresponding values. Tokens that match strongly contribute a lot; tokens that do not contribute almost nothing. That is the whole operation, repeated across dozens of layers and dozens of parallel heads.
Multiple heads exist because different relationships matter simultaneously. One head might track subject-verb agreement, another coreference across sentences, another the boundary of a code block. Each head learns its own projection into query, key, and value space, and their outputs are concatenated. Nobody assigns these roles; they fall out of training, and untangling what individual heads actually do is most of what mechanistic interpretability spends its time on.
The property that shapes everything downstream is cost. Comparing every token to every other token is quadratic in sequence length, so doubling the context roughly quadruples the attention computation. This single fact explains why long context windows were hard to build, why they remain expensive to serve, and why so much engineering effort goes into working around it. Grouped-query and multi-query attention share key and value projections across heads to shrink the KV cache. Sliding-window attention limits each token to a local neighborhood. FlashAttention does not change the math at all but reorders the computation to avoid writing the full attention matrix to memory, which is where the practical speedups of the last few years mostly came from.
The most useful way to think about attention for application work is as soft retrieval over the prompt. At every layer, every token is querying everything before it and pulling in what looks relevant. That framing immediately explains a failure mode teams hit constantly: attention is a finite budget spread across a softmax, so as the prompt grows, weight gets diluted across more candidates, and a genuinely important sentence buried in 80,000 tokens of mostly irrelevant context can end up with almost no weight on it. The lost-in-the-middle effect and context rot are both descriptions of this. Filling a large window is not the same as being attended to, which is why curating what goes into context beats maximizing how much fits.
Why It Matters
Attention is where the quadratic cost, the KV cache, and the practical limits of long context all originate, so it is the concept that makes those three stop being separate mysteries. It also gives a principled reason to prefer selective context over exhaustive context: not that a big prompt is expensive, though it is, but that the mechanism reading it distributes a fixed amount of weight across everything you include. Every irrelevant token you add takes attention away from the ones that mattered.
Example
A team moves their assistant to a 200,000-token model and starts sending the entire customer history with every request, expecting better answers. Accuracy on specific factual questions drops. The relevant detail is present in the prompt every time, but it is one sentence competing with thousands of others for attention weight, and it sits in the middle of the sequence where models attend least reliably. Replacing the dump with a retrieval step that supplies 2,000 tokens of relevant history restores accuracy and cuts cost by two orders of magnitude.