Most of what you care about in an AI product cannot be measured by comparing strings. Whether a summary is faithful to its source, whether a support reply is actually helpful, whether an answer is grounded in the retrieved documents, whether the tone fits your brand: none of these have a scoring function. Humans can judge them reliably but not on every commit for a thousand cases. LLM-as-a-judge fills that gap by handing the rubric to a model.
The setup has a few standard shapes and they behave differently. Pointwise scoring asks the judge to rate a single output against criteria, which is convenient and the least reliable, because absolute scales drift. Pairwise comparison asks which of two outputs is better, which is far more stable and is what you want for comparing model versions or prompt variants. Reference-based grading gives the judge a gold answer to compare against; reference-free grading gives it only the rubric, which is more flexible and noisier. Across all of them, asking the judge to reason before it scores measurably improves agreement with humans, and asking it to emit a structured verdict rather than prose makes the results usable.
The biases are well documented and, more usefully, they have names and mitigations. Position bias means the judge favors whichever candidate appears first, and the fix is to run each comparison both ways and discard disagreements. Verbosity bias means longer answers score higher independent of quality, which you control by normalizing length or by scoring specific claims rather than overall impression. Self-preference bias means a model rates its own outputs generously, which is an argument for using a different model family as judge than the one under test. And scores drift when the judge model version changes underneath you, which is an argument for pinning it.
The discipline that separates a useful judge from a confident random number generator is calibration. Label a few hundred examples by hand, run the judge against them, and measure agreement with a real statistic such as Cohen kappa rather than eyeballing it. If agreement is poor, the rubric is the problem far more often than the model is, and tightening from a one-to-ten scale down to a binary or three-point decision usually fixes most of it. Keep the labeled set as a permanent fixture and re-run it whenever the rubric or judge model changes. An uncalibrated judge is worse than no judge, because it produces precise-looking numbers that a team will act on.
Cost and latency are real but rarely decisive. A judge run on every example of a large eval set with a frontier model gets expensive, so common practice is a strong judge on a sampled subset plus a cheaper judge on the full set, or a cheap judge that escalates only ambiguous cases. The thing to avoid is quietly downgrading the judge to save money and then comparing this month scores against last month.
Why It Matters
Without an automated judge, subjective quality gets measured by whoever looked at the output last, which means it is not measured. With one, quality becomes a number you can put in CI, track across model upgrades, and use to settle prompt debates with evidence rather than intuition. It is the practical bridge between having evals and having evals for the things that actually matter, which are almost never the things a string comparison can check.
Example
A team evaluates whether their assistant answers are grounded in retrieved context. They start with a one-to-ten helpfulness score and find the judge assigns 7 or 8 to nearly everything, so the metric moves with noise rather than with changes. They rewrite the rubric as three binary questions: is every factual claim supported by the provided context, does the answer address the question asked, does it decline appropriately when context is insufficient. Agreement with human labels on a 200-example calibration set rises from a kappa around 0.3 to above 0.7, and the metric starts detecting real regressions on the same day a bad retrieval change ships.