Agent Evals & Observability
Measuring whether AI agents work: eval design, LLM-as-a-judge, test sets, tracing, observability and regression testing for agents in production.
An agent eval is only as good as its judge, and most teams never measure the judge. A grader that agrees with itself but not with users produces a green dashboard over a product that is getting worse, which is why eval design deserves the same scrutiny as the agent it scores.
These articles cover how to build evals that catch regressions: choosing between rule-based checks and LLM-as-a-judge, calibrating a judge against human labels, building test sets from real traffic, and what to trace once an agent is live.
Start here: Measuring agent memory · Maximem evals · LLM-as-a-judge (glossary)
Which LLM Should Write Your AI Agent's Memory? How to Choose and Test a Memory Extraction Model
Which LLM to use for memory extraction: in Mem0's October 2026 test of 17 writer models on LoCoMo, the writer alone moved accuracy from 84.1% to 93.5% and cost from $0.68 to $158.55 per 1,000 writes, yet the top six sat within 0.9 points. How to test writer models on your own transcripts, the failure modes to measure (rules written as memory, over-extraction, missed updates), and where a small judge model fits with a type-aware write policy.
Agent Memory Consolidation: What a Nightly "Dreaming" Job Should Do to Your Agent's Memory, and How to Check It Did
Agent memory consolidation is a background job that merges duplicate memories, supersedes replaced facts, expires time-bound ones and promotes repeated episodes into durable facts. Run it on a copy, verify the copy against invariants and known answers, then swap. How Anthropic Dreams, Letta and Mem0 Dream do it, the failure modes of unattended runs, and a tested Python job.
Synap Is #1 on LongMemEval: 95.6% on All 500 Questions
Maximem Synap is #1 on LongMemEval: 95.6% on all 500 questions with Gemini 3.8 Flash, at about 7.4K tokens per question. Every answer and verdict is public.
How Do You Prevent Memory Poisoning Attacks on AI Agents?
How to prevent memory poisoning in AI agents: the five ways poisoned content gets into memory, the published attacks on ChatGPT, Bedrock Agents and Claude Code, what OWASP ASI06 requires, eight defences from write-path validation to rollback, which ones still hold when the model is fooled, and a test plan you can run in CI.
How Do You Add Episodic and Semantic Memory to an AI Agent? A Working Python Build With SQLite, Local Embeddings and Tests
Add episodic and semantic memory to an AI agent with two SQLite tables and two code paths: episodes record what happened and when, facts hold what is true now and a new value closes the old one, a write path extracts both after each session, and a read path ranks them into the prompt. A shared playbook table, changed only on evidence attributed to the strategy as in Alibaba's SPAR, carries what works across users. Full Python with local embeddings, thirteen passing tests, no API key.
What Breaks When You Build Your Own Agent Memory? Nine Failure Modes and How to Detect Each
When you build your own agent memory, the write path breaks first: unresolved contradictions, stale facts, duplicates that outvote corrections, entity confusion, deletes that miss derived copies, cross-user leakage and write cost. What each looks like in production, what causes it, and the signal that detects it, grounded in STALE, FEB-30 and HaluMem.
How Do You Test Agent Memory Before You Ship? A Test Plan You Can Run in a Week
A one-week test plan for agent memory: script multi-session histories with a known answer key, then grade the store, retrieval and outcome separately across recall, continuity, contradiction, staleness, deletion, cross-tenant leakage and hallucinated recall, with a metric and a recommended pass bar for each.
AI Agent Evals: The Judge Is The Part Nobody Measures
An LLM judge scoring the same agent output three times will often hand back three different answers, and almost no guide to agent evaluation asks whether the judge agrees with itself or what it costs to run on everything rather than a sample.
How to Evaluate AI Agents (and Test a Model Swap Before You Switch)
How to evaluate an AI agent: start with 20 to 50 tasks drawn from real failures, grade the outcome and the transcript, run several trials per task, gate every change in CI, and check the grader before you trust it. Updated 6 November 2026 with a seven-step method for testing a cheaper model inside your own agent, and why a loop that closed the gap in a paper did not justify a switch in practice.
$4K Courses Will Teach You Agent Evals. Here's a Free Guide.
Move beyond static benchmarks to master the art of AI agent evaluation. This guide explores how to design frameworks that measure reasoning, tool-use, and reliability to bridge the gap between experimental prototypes and production-ready systems.