Published 29 October 2026 by Gaurav Dadhich. Every product detail and quote below was read from its primary source on 29 October 2026, and the code was run the same day on Python 3.10.12 with SQLite 3.37.2; its nine tests passed in 0.17 seconds.
Agent memory consolidation is a background job that runs over an agent's stored memory when no user is waiting, and does what the per-message write path cannot: it merges duplicates, marks facts that a newer fact replaced as superseded, expires time-bound facts, and turns patterns that repeat across several conversations into one durable fact. The job is the easy half. The half that decides whether you can leave it running unattended is how it writes: a consolidation pass should never edit the live store in place, it should build a new version, check that version against a short list of invariants and a few known answers, and only then swap it in, keeping yesterday's store as the rollback.
The idea moved from research into products this year. Anthropic's Dreams for Claude Managed Agents (research preview) and Mem0's Dream (4 August 2026) both run it as a managed job, and Letta ships it as "dreaming". On 28 October 2026 Harrison Chase listed "memory (dreaming)" first among the new capabilities that crons give agent harnesses, and LangChain posted that since it launched dynamic scheduling for Managed Deep Agents in September, "over 70% of MDA runs on LangSmith come from schedules vs human typing a message." A scheduled job that rewrites memory with nobody watching is now a normal thing to build, which makes the checks around it the part worth getting right.
The failure catalogue this job guards against is in what breaks when you build your own agent memory, the episodic and semantic store it runs over is built in how to add episodic and semantic memory to an AI agent, and the pre-launch test plan is in how to test agent memory before you ship. The short definition is in our glossary entry on memory consolidation.
Why the write path cannot do this on its own
Every write decision in a memory system is made with a narrow view. When a message arrives, the extractor pulls out candidate facts, retrieves the handful of stored memories most similar to each one, and decides whether to add, update or ignore. Mem0 states the limit plainly: each write decision "sees only the current conversation and a small set of similar memories retrieved for context", and nothing else in the store. Anthropic's docs describe the result: agent writes "are local and incremental: over many sessions a memory store accumulates duplicates, contradictions, and stale entries."
Three problems are invisible from inside a single write. A fact restated in different words ("prefers window seats" and "always books window seats on long-haul flights", Mem0's example) is a new string, so exact-match deduplication misses it. A fact that changed four months after it was written may never be retrieved alongside its replacement, so both stay current. And a pattern spread over many conversations, such as a user who asks for an aisle seat on every booking, exists in no single conversation, so no single write can record it. Mem0 says that in its own production data "the median active project carries a few hundred memories that duplicate or contradict other memories in the same project."
Write-time resolution still matters, and it should stay. When the old fact is in the write context, deciding there and then is cheaper and keeps the store correct sooner, which is why Mem0 makes its merge and supersede decisions during extraction and runs only synthesis on a schedule. The OpenAI Agents SDK cookbook on context personalization uses the same split, session notes during the run and consolidation afterwards, and calls two-phase processing "more reliable than one-shot". So the honest answer to whether you need a scheduled job at all is: you need one as soon as the store is large enough that the old version of a fact can fall out of the top results at write time.
What a consolidation job should do, in order
| Step | What it does | Guard that keeps it safe |
|---|---|---|
| Expire | Closes facts whose validity window has ended ("in Singapore this week") | Only facts written with an explicit expiry; never age alone |
| Merge | Collapses restatements of the same fact into one record, adding the confirmations | Same user, same attribute, same normalised value |
| Supersede | Marks the older value of a single-valued attribute as replaced, with a pointer to its successor | An agent's inference never supersedes something the user said |
| Promote | Turns a pattern seen across several episodes into one semantic fact | Minimum evidence count, source episodes stored on the fact |
| Resolve entities | Joins "my manager", "Sarah" and "Sarah Chen" | Ambiguous matches go to review, not to a guess |
| Respect forgets | Skips anything the user asked to have forgotten | A tombstone table the job reads before every promotion |
The order matters. Expiring first stops the job from merging a fact that is no longer true; merging before superseding means three copies of "lives in Boston" do not outvote one "moved to Austin"; and promotion runs last, followed by a second supersede pass, because a newly promoted fact can itself conflict with something already stored.
Promotion is the step that turns episodic memory into semantic memory, and it needs the strictest bar. Mem0 describes its synthesis as "deliberately conservative. Single observations, one-off events, and repeated questions about a topic do not produce summaries." The OpenAI cookbook's note-taking rules ask for a memory to be durable, actionable and explicit, "stated or clearly confirmed by the user (not inferred)". In code, that means a minimum number of supporting episodes and the episode IDs stored on the new fact, so anyone can see why the agent believes it.
None of the steps deletes anything. Each one changes a status and records a pointer, which is the design all three managed products converge on: Mem0 says "Nothing is deleted in any of these operations", Amazon Bedrock AgentCore Memory marks outdated memories as INVALID "instead of instantly deleting them", and an Anthropic dream writes a new store and leaves the old one untouched. Hindsight's write-up of the consolidation problem puts the rule well: "eviction is a compliance tool, not a performance tool." Decay belongs at retrieval, as a ranking weight; real deletion belongs to the erasure path, which is a different job with different rules.
Sleep-time compute, dreaming and consolidation are three names for overlapping ideas
Sleep-time compute is the research framing. Lin et al. (arXiv 2504.13171, April 2025) proposed letting models "think" offline about a context before any query arrives, and found that the benefit tracks how predictable the user's next question is from that context. Letta's announcement applied it to memory: a sleep-time agent rewrites the primary agent's memory blocks in the background, and the primary agent is given no tools to edit its own core memory at all.
"Dreaming" is the product name that stuck. Anthropic, Letta and Mem0 all use it for a background pass that reads what the agent has stored (and, for Anthropic and Letta, recent conversations) and writes a cleaner version. Consolidation is the older, more general word, borrowed from neuroscience, and the borrowing is deliberate: Auto-Dreamer (arXiv 2605.20616, May 2026) cites complementary learning systems theory and "decouples fast per-session memory acquisition from slow cross-session consolidation", and LightMem uses an offline "sleep-time update" that separates consolidation from online inference. The analogy holds for the scheduling (cheap fast writes by day, expensive slow reorganisation offline) and breaks on forgetting: a brain discards detail because capacity is finite, while an agent store can keep every superseded fact as history and only hide it from retrieval.
Auto-Dreamer is worth reading for one design choice in particular. Its consolidator "treats the region as read-only evidence" and "synthesizes a fresh compact replacement set" that supersedes the original, which is the same copy-then-replace shape that Anthropic ships as a product, arrived at from the research side.
When to run it, and what it costs
| Where | What triggers consolidation |
|---|---|
| Anthropic Managed Agents Dreams | You create a dream through the API over a memory store and 1 to 100 past sessions; it "typically" takes "minutes to a few hours" |
| Letta (Memory & dreaming) | After a set number of completed agent steps, or when the context window is compacted |
| Mem0 Dream | Synthesis runs weekly per project for eligible users; merge and supersede happen at write time |
| OpenAI Agents SDK cookbook | Asynchronously at the end of each session |
| LangMem | A ReflectionExecutor waits for conversation activity to settle before processing (guide) |
| Your own cron | Whatever you schedule, usually during the product's quiet hours |
Pick the cadence from two numbers: how quickly a stale fact causes a visible mistake in your product, and how much each run costs. Anthropic bills dreams at standard token rates for the model you choose and says cost "scales roughly linearly with the number and length of input sessions", so a nightly run over every active user's day is a line item, not a rounding error. Letta's guidance on models is the useful counterweight: the background agent is not latency-bound, so it can run a stronger, slower model than the conversational one, and "the higher the frequency setting, the more tokens your agent will use."
A sensible default is a light, frequent pass that only expires and merges (cheap, deterministic, no model call) and a heavier nightly pass that promotes episodes and resolves conflicts with a model. Run the heavy pass per user, so one large account does not hold up everyone else, and trigger it only for users with new activity since the last run, which is also what Mem0 does when Dream is first enabled: it "processes new activity from the moment you enable it and does not touch your existing memories retroactively."
Never consolidate in place
The single most useful design decision is that the job writes somewhere new. Anthropic's Dreams API makes it structural: a dream "produces a new, reorganized memory store", and "the input store is never modified, so you can review the output and discard it". If a dream fails, "the output memory store is left as-is with whatever was written before failure", and the input is still intact. Anthropic's memory docs then describe the cut-over: "Switch your sessions over to that output store, then archive or delete the original." Letta's reorganisation workflow does the same with git, backing up the memory repository before it splits files, merges duplicates or restructures the hierarchy.
In-place consolidation turns every bug into data loss. A merge rule that is slightly too loose, an LLM that summarises "was considering a move to Seattle" as "lives in Seattle", or a crash halfway through leaves the live store in a state nobody chose, and the agent reads from it straight away. Working on a copy turns the same bug into a rejected run and a file you can inspect.
History and concurrency are the two details that make the copy safe in practice. On history, Anthropic keeps an immutable version for every change to a memory, retained for 30 days (with recent versions of a live memory kept regardless of age), and there is no dedicated restore endpoint, so rollback means reading the old version and writing it back. If you build your own, keep the pre-run store until the next run succeeds. On concurrency, users keep talking while the job runs, and a swap that ignores those writes silently drops them. Anthropic's memory API offers a content_sha256 precondition so an update only applies if the content has not changed since you read it, and Mem0 applies its updates "with conditional writes, so concurrent operations on the same memory cannot conflict." The code below takes a third route: it remembers the last row it saw, and at swap time it carries every newer row into the copy under a short write lock.
Whether a person approves the result is a product decision, not a technical one. Anthropic's design leaves the swap to you, so a review step is natural; Letta's "Agent reviews before applying" option has a second background agent check proposed updates, and the docs say it "does not ask you for approval"; Mem0 lets you review each change in its dashboard after the fact. For a consumer product with thousands of users, a human cannot read every run, so the automated checks below have to carry the weight, and a human reads only the runs those checks reject.
The append-only markdown trap
The simplest overnight memory is a model that reads the day and appends what it learned to a markdown file. It is easy to inspect, easy to edit and easy to delete, which is why products like Sentient OS build on it: launched on 27 October 2026, it reads messages, files and notes overnight with a local model and writes what it learns into a folder of plain markdown files, according to Rohan Paul's summary, and the launch post says "You own all of Sentient's knowledge", describing it as a folder of markdown on your Mac written by a model you control.
The trap is what happens to a fact that changes. Suppose March's run wrote - Lives in Boston and September's run wrote - Moved to Austin. An append-only writer never retires the first line, so the file now holds both, with nothing marking which is current, and whichever model reads the file next has to guess. It will often guess right, which is the problem: the failure only shows up on the turn where it guesses wrong, and nothing in the file explains why. Markdown is a fine format for memory; append-only is the part that fails. A consolidation pass over markdown has to rewrite the file (move Boston to a history section with a date, keep Austin as current) and, like any other consolidation, it should write a new file and diff it against the old one before replacing it.
A nightly consolidation job you can run
The code below is a complete nightly job over SQLite, using only the Python standard library. Facts carry a source (user or inferred), a validity window and a status; episodes are one-line summaries of past conversations; tombstones record what a user asked to forget. The Consolidator runs the steps from the table above on a copy, and nightly does the copy, the carry-over of late writes, the verification and the swap. Daytime writes are inserts only; status changes happen in the job, which is what makes the carry-over by row ID correct. Promotion takes a summarise function, which in production is an LLM call and in the tests is a deterministic stand-in.
"""A nightly memory consolidation job that works on a copy and only swaps it in if checks pass."""
import os, re, sqlite3, time, uuid
SCHEMA = """
CREATE TABLE IF NOT EXISTS facts (
id TEXT PRIMARY KEY,
user_id TEXT NOT NULL,
attribute TEXT NOT NULL, -- home_city, diet, allergy ...
value TEXT NOT NULL,
multi INTEGER NOT NULL DEFAULT 0,
source TEXT NOT NULL, -- 'user' (said it) or 'inferred' (agent guessed it)
valid_from REAL NOT NULL,
valid_to REAL, -- NULL = current
expires_at REAL, -- for time-bound facts ("in Singapore this week")
status TEXT NOT NULL DEFAULT 'active', -- active | merged | superseded | expired | review
superseded_by TEXT,
evidence TEXT NOT NULL DEFAULT '', -- ids of the episodes or messages behind it
confirmations INTEGER NOT NULL DEFAULT 1
);
CREATE TABLE IF NOT EXISTS episodes (
id TEXT PRIMARY KEY, user_id TEXT NOT NULL, occurred_at REAL NOT NULL, summary TEXT NOT NULL);
CREATE TABLE IF NOT EXISTS tombstones ( -- things a user asked us to forget
user_id TEXT NOT NULL, attribute TEXT NOT NULL, value TEXT NOT NULL);
CREATE TABLE IF NOT EXISTS job_log (
run_id TEXT, at REAL, action TEXT, fact_id TEXT, detail TEXT);
"""
def connect(path):
db = sqlite3.connect(path)
db.row_factory = sqlite3.Row
db.executescript(SCHEMA)
return db
def norm(v):
return re.sub(r"[^a-z0-9]+", " ", v.lower()).strip()
def current(db, where="", args=()):
return db.execute("SELECT * FROM facts WHERE valid_to IS NULL AND status='active' "
+ where + " ORDER BY valid_from, rowid", args).fetchall()
class Consolidator:
"""Runs on the shadow copy only. `summarise` turns a user's episodes into candidate facts."""
def __init__(self, summarise=None, min_support=2):
self.summarise, self.min_support = summarise, min_support
def run(self, db, now, run_id):
log = lambda a, fid, d="": db.execute(
"INSERT INTO job_log VALUES (?,?,?,?,?)", (run_id, now, a, fid, d))
with db:
# 1. retire time-bound facts whose window has passed
for r in current(db, "AND expires_at IS NOT NULL AND expires_at < ?", (now,)):
db.execute("UPDATE facts SET valid_to=expires_at, status='expired' WHERE id=?", (r["id"],))
log("expire", r["id"], r["value"])
# 2. merge duplicates: same user, attribute and normalised value
groups = {}
for r in current(db):
groups.setdefault((r["user_id"], r["attribute"], norm(r["value"])), []).append(r)
for rows in groups.values():
keep, dupes = rows[0], rows[1:]
for d in dupes:
db.execute("UPDATE facts SET status='merged', valid_to=?, superseded_by=? WHERE id=?",
(now, keep["id"], d["id"]))
db.execute("UPDATE facts SET confirmations=confirmations+?, evidence=evidence||' '||? "
"WHERE id=?", (d["confirmations"], d["evidence"], keep["id"]))
log("merge", d["id"], "into " + keep["id"])
# 3. resolve contradictions on single-valued attributes
self.resolve(db, now, log)
# 4. promote repeated episode patterns to facts (episodic -> semantic)
if self.summarise:
for (uid,) in db.execute("SELECT DISTINCT user_id FROM episodes").fetchall():
eps = db.execute("SELECT * FROM episodes WHERE user_id=? ORDER BY occurred_at",
(uid,)).fetchall()
for c in self.summarise(eps):
if len(c["evidence"]) < self.min_support:
continue # one observation is not a pattern
if db.execute("SELECT 1 FROM tombstones WHERE user_id=? AND attribute=? "
"AND value=?", (uid, c["attribute"], c["value"])).fetchone():
log("skip_tombstoned", "", c["attribute"] + "=" + c["value"])
continue # the user asked us to forget this
if any(norm(r["value"]) == norm(c["value"])
for r in current(db, "AND user_id=? AND attribute=?", (uid, c["attribute"]))):
continue
fid = uuid.uuid4().hex
db.execute("INSERT INTO facts (id,user_id,attribute,value,multi,source,valid_from,"
"evidence) VALUES (?,?,?,?,?,'inferred',?,?)",
(fid, uid, c["attribute"], c["value"], int(c.get("multi", 0)), now,
" ".join(c["evidence"])))
log("promote", fid, c["attribute"] + "=" + c["value"])
self.resolve(db, now, log)
@staticmethod
def resolve(db, now, log):
by_key = {}
for r in current(db, "AND multi=0"):
by_key.setdefault((r["user_id"], r["attribute"]), []).append(r)
for rows in by_key.values():
if len(rows) < 2:
continue
newest = rows[-1]
for old in rows[:-1]:
if old["source"] == "user" and newest["source"] == "inferred":
# an agent's guess never overrules what the user said: park it for review
db.execute("UPDATE facts SET status='review' WHERE id=?", (newest["id"],))
log("review", newest["id"], "inferred value conflicts with a user statement")
break
db.execute("UPDATE facts SET status='superseded', valid_to=?, superseded_by=? WHERE id=?",
(newest["valid_from"], newest["id"], old["id"]))
log("supersede", old["id"], old["value"] + " -> " + newest["value"])
def verify(live, shadow, probes=(), max_drop=0.5):
"""Return a list of problems. An empty list means the shadow store may replace the live one."""
problems = []
live_rows = {r["id"]: r for r in live.execute("SELECT * FROM facts")}
shadow_rows = {r["id"]: r for r in shadow.execute("SELECT * FROM facts")}
lost = set(live_rows) - set(shadow_rows)
if lost:
problems.append(f"{len(lost)} facts deleted outright; consolidation may only change status")
for fid, r in live_rows.items():
if fid in shadow_rows and shadow_rows[fid]["user_id"] != r["user_id"]:
problems.append(f"fact {fid} moved from {r['user_id']} to {shadow_rows[fid]['user_id']}")
for r in shadow_rows.values():
if r["superseded_by"] and r["superseded_by"] in shadow_rows \
and shadow_rows[r["superseded_by"]]["user_id"] != r["user_id"]:
problems.append(f"fact {r['id']} superseded by another user's fact")
for r in shadow.execute("SELECT user_id, attribute, COUNT(*) n FROM facts WHERE valid_to IS NULL "
"AND status='active' AND multi=0 GROUP BY 1,2 HAVING n>1"):
problems.append(f"{r['user_id']}.{r['attribute']} still has {r['n']} current values")
for t in shadow.execute("SELECT * FROM tombstones"):
if current(shadow, "AND user_id=? AND attribute=? AND value=?",
(t["user_id"], t["attribute"], t["value"])):
problems.append(f"forgotten value came back: {t['user_id']}.{t['attribute']}")
for (uid,) in live.execute("SELECT DISTINCT user_id FROM facts"):
before = len(current(live, "AND user_id=?", (uid,)))
after = len(current(shadow, "AND user_id=?", (uid,)))
if before >= 4 and after < before * (1 - max_drop):
problems.append(f"{uid}: current facts fell from {before} to {after}")
for uid, attr, expected in probes:
got = [r["value"] for r in current(shadow, "AND user_id=? AND attribute=?", (uid, attr))]
if expected not in got:
problems.append(f"probe failed: {uid}.{attr} expected {expected!r}, got {got}")
return problems
def nightly(live_path, consolidator, probes=(), now=None, during_job=None):
"""Copy, consolidate the copy, carry over writes that landed meanwhile, verify, then swap."""
now, run_id = now or time.time(), uuid.uuid4().hex[:8]
shadow_path = live_path + ".shadow"
if os.path.exists(shadow_path):
os.remove(shadow_path)
live = connect(live_path)
hwm = live.execute("SELECT COALESCE(MAX(rowid),0) FROM facts").fetchone()[0]
shadow = connect(shadow_path)
live.backup(shadow) # consistent snapshot
consolidator.run(shadow, now, run_id)
if during_job:
during_job() # tests use this to simulate daytime writes
live.isolation_level = None # manage the transaction by hand
live.execute("BEGIN IMMEDIATE") # brief write lock for the swap
try:
late = live.execute("SELECT * FROM facts WHERE rowid > ?", (hwm,)).fetchall()
with shadow:
for r in late:
shadow.execute(f"INSERT INTO facts VALUES ({','.join('?' * len(r.keys()))})", tuple(r))
Consolidator.resolve(shadow, now, lambda *a: None)
shadow.execute("DELETE FROM tombstones")
shadow.executemany("INSERT INTO tombstones VALUES (?,?,?)",
[tuple(t) for t in live.execute("SELECT * FROM tombstones")])
problems = verify(live, shadow, probes)
if problems:
shadow.close()
os.replace(shadow_path, f"{live_path}.rejected-{run_id}") # keep it for a human
return {"run_id": run_id, "swapped": False, "problems": problems}
actions = shadow.execute("SELECT action, COUNT(*) FROM job_log WHERE run_id=? GROUP BY 1",
(run_id,)).fetchall()
shadow.close()
os.replace(live_path, f"{live_path}.pre-{run_id}") # yesterday's store is the rollback
os.replace(shadow_path, live_path) # swap while still holding the lock
return {"run_id": run_id, "swapped": True, "actions": dict(map(tuple, actions))}
finally:
live.execute("ROLLBACK")
live.close()
The tests pin the behaviour that matters most when nobody is watching: duplicates merge, a move supersedes the old city and keeps it as history, an inferred value never overrules the user, a time-bound fact expires, a pattern needs more than one episode, a forgotten fact is not rebuilt from old episodes, a buggy job that merges across users is rejected with the live store untouched, a failed probe blocks the swap, and a write that lands during the job survives it.
import sqlite3, uuid, pytest
from consolidate import connect, current, Consolidator, nightly
DAY, T0 = 86400, 1_785_000_000
def add(path, user, attr, value, at, source="user", multi=0, expires=None, evidence=""):
db = connect(path)
with db:
db.execute("INSERT INTO facts (id,user_id,attribute,value,multi,source,valid_from,expires_at,"
"evidence) VALUES (?,?,?,?,?,?,?,?,?)",
(uuid.uuid4().hex, user, attr, value, multi, source, at, expires, evidence))
db.close()
def values(path, user, attr):
db = connect(path)
out = [r["value"] for r in current(db, "AND user_id=? AND attribute=?", (user, attr))]
db.close()
return out
@pytest.fixture
def live(tmp_path):
return str(tmp_path / "memory.db")
def test_duplicates_merge_into_one_with_confirmations(live):
add(live, "u1", "diet", "vegetarian", T0)
add(live, "u1", "diet", "Vegetarian.", T0 + DAY)
out = nightly(live, Consolidator(), now=T0 + 2 * DAY)
assert out["swapped"] and out["actions"] == {"merge": 1}
db = connect(live)
assert [(r["value"], r["confirmations"]) for r in current(db)] == [("vegetarian", 2)]
def test_move_supersedes_old_city_and_keeps_history(live):
add(live, "u1", "home_city", "Boston", T0)
add(live, "u1", "home_city", "Austin", T0 + 30 * DAY)
nightly(live, Consolidator(), now=T0 + 31 * DAY)
assert values(live, "u1", "home_city") == ["Austin"]
db = connect(live)
old = db.execute("SELECT * FROM facts WHERE value='Boston'").fetchone()
assert old["status"] == "superseded" and old["valid_to"] == T0 + 30 * DAY
def test_inferred_value_never_overrules_the_user(live):
add(live, "u1", "home_city", "Boston", T0)
add(live, "u1", "home_city", "Austin", T0 + DAY, source="inferred")
nightly(live, Consolidator(), now=T0 + 2 * DAY)
assert values(live, "u1", "home_city") == ["Boston"]
db = connect(live)
assert db.execute("SELECT status FROM facts WHERE value='Austin'").fetchone()[0] == "review"
def test_time_bound_fact_expires(live):
add(live, "u1", "location_this_week", "Singapore", T0, expires=T0 + 7 * DAY)
nightly(live, Consolidator(), now=T0 + 8 * DAY)
assert values(live, "u1", "location_this_week") == []
def repeated_requests(episodes):
"""Stand-in for an LLM summariser: a request seen in several episodes becomes a preference."""
seen = {}
for e in episodes:
if "aisle seat" in e["summary"]:
seen.setdefault(("seat_preference", "aisle"), []).append(e["id"])
return [{"attribute": a, "value": v, "evidence": ids} for (a, v), ids in seen.items()]
def seed_episodes(path, user, n):
db = connect(path)
with db:
for i in range(n):
db.execute("INSERT INTO episodes VALUES (?,?,?,?)",
(f"ep{i}", user, T0 + i * DAY, "Booked a flight and asked for an aisle seat"))
db.close()
def test_pattern_across_episodes_becomes_a_fact_only_with_support(live, tmp_path):
seed_episodes(live, "u1", 3)
nightly(live, Consolidator(repeated_requests), now=T0 + 5 * DAY)
assert values(live, "u1", "seat_preference") == ["aisle"]
one = str(tmp_path / "one.db")
seed_episodes(one, "u2", 1)
nightly(one, Consolidator(repeated_requests), now=T0 + 5 * DAY)
assert values(one, "u2", "seat_preference") == []
def test_forgotten_fact_is_not_rebuilt_from_old_episodes(live):
seed_episodes(live, "u1", 3)
db = connect(live)
with db:
db.execute("INSERT INTO tombstones VALUES ('u1','seat_preference','aisle')")
db.close()
nightly(live, Consolidator(repeated_requests), now=T0 + 5 * DAY)
assert values(live, "u1", "seat_preference") == []
class CrossUserMerge(Consolidator):
"""A buggy job: merges duplicates without checking the user."""
def run(self, db, now, run_id):
with db:
rows = current(db)
db.execute("UPDATE facts SET status='merged', valid_to=?, superseded_by=? WHERE id=?",
(now, rows[0]["id"], rows[1]["id"]))
def test_bad_job_is_rejected_and_live_store_untouched(live, tmp_path):
add(live, "alice", "employer", "Acme", T0)
add(live, "bob", "employer", "Acme", T0)
out = nightly(live, CrossUserMerge(), now=T0 + DAY)
assert not out["swapped"]
assert any("another user's fact" in p for p in out["problems"])
assert values(live, "bob", "employer") == ["Acme"]
assert any(p.name.startswith("memory.db.rejected-") for p in tmp_path.iterdir())
def test_golden_probe_blocks_a_swap(live):
add(live, "u1", "home_city", "Boston", T0)
add(live, "u1", "home_city", "Austin", T0 + DAY)
out = nightly(live, Consolidator(), probes=[("u1", "home_city", "Boston")], now=T0 + 2 * DAY)
assert not out["swapped"] and out["problems"][0].startswith("probe failed")
def test_write_that_lands_during_the_job_survives_the_swap(live):
add(live, "u1", "home_city", "Boston", T0)
late = lambda: add(live, "u1", "home_city", "Denver", T0 + 3 * DAY)
out = nightly(live, Consolidator(), now=T0 + 3 * DAY, during_job=late)
assert out["swapped"]
assert values(live, "u1", "home_city") == ["Denver"]
test_consolidate.py::test_duplicates_merge_into_one_with_confirmations PASSED [ 11%]
test_consolidate.py::test_move_supersedes_old_city_and_keeps_history PASSED [ 22%]
test_consolidate.py::test_inferred_value_never_overrules_the_user PASSED [ 33%]
test_consolidate.py::test_time_bound_fact_expires PASSED [ 44%]
test_consolidate.py::test_pattern_across_episodes_becomes_a_fact_only_with_support PASSED [ 55%]
test_consolidate.py::test_forgotten_fact_is_not_rebuilt_from_old_episodes PASSED [ 66%]
test_consolidate.py::test_bad_job_is_rejected_and_live_store_untouched PASSED [ 77%]
test_consolidate.py::test_golden_probe_blocks_a_swap PASSED [ 88%]
test_consolidate.py::test_write_that_lands_during_the_job_survives_the_swap PASSED [100%]
============================== 9 passed in 0.17s ===============================
A short demo seeds one user with a duplicate diet, a move from Boston to Austin, a week in Singapore that has ended and three bookings with an aisle seat, then runs a correct job and a buggy one:
import sys
from consolidate import connect, current, Consolidator, nightly
from test_consolidate import add, seed_episodes, repeated_requests, CrossUserMerge, T0, DAY
path = sys.argv[1] # an empty directory, e.g. /tmp/demo/memory.db
add(path, "maya", "home_city", "Boston", T0)
add(path, "maya", "diet", "vegetarian", T0)
add(path, "maya", "diet", "Vegetarian", T0 + 9 * DAY)
add(path, "maya", "location_this_week", "Singapore", T0 + 20 * DAY, expires=T0 + 27 * DAY)
add(path, "maya", "home_city", "Austin", T0 + 40 * DAY)
seed_episodes(path, "maya", 3)
print(nightly(path, Consolidator(repeated_requests), probes=[("maya", "home_city", "Austin")],
now=T0 + 45 * DAY))
db = connect(path)
for r in current(db):
print(f" {r['attribute']:<16} {r['value']:<11} source={r['source']:<8} confirmations={r['confirmations']}")
db.close()
add(path, "sam", "employer", "Acme", T0)
print(nightly(path, CrossUserMerge(), now=T0 + 46 * DAY))
Running python demo.py /tmp/demo/memory.db (the directory must exist and be empty) printed:
{'run_id': '93267676', 'swapped': True, 'actions': {'expire': 1, 'merge': 1, 'promote': 1, 'supersede': 1}}
diet vegetarian source=user confirmations=2
home_city Austin source=user confirmations=1
seat_preference aisle source=inferred confirmations=1
{'run_id': '6dee8414', 'swapped': False, 'problems': ["fact 2c62a04d77b44bdc985894f6ba30e880 superseded by another user's fact"]}
The first run expired Singapore, merged the two vegetarian rows into one with two confirmations, superseded Boston with Austin and promoted the aisle-seat pattern as an inferred fact, then swapped. The second run, a job that merged one user's fact into another's, was rejected; the live store kept serving, and the rejected copy stayed on disk next to the pre-run backup for someone to read in the morning.
Three things in this build are deliberately simple and need work before production. The swap replaces a file, which is fine for one SQLite store per tenant; with a shared database, the equivalent is a version column or a pointer to the current store, which is what Anthropic's output-store design gives you. Entity resolution is not implemented, so "Sarah" and "Sarah Chen" stay two subjects. And the summariser is a stand-in; a real one is an LLM call whose output should be checked against its cited episodes before it is promoted.
How to check stored state after the job runs
A consolidation run is a large, unsupervised write, so check it the way you would check a database migration: compare before and after against rules that must always hold, then run a handful of known answers. The verify function above implements these, and each is cheap enough to run on every user every night:
- Nothing deleted outright. Every row in the live store exists in the new one; only statuses changed. A missing row means a bug, not a cleanup.
- No lineage across users. No fact moved owner, and no fact is superseded by, merged into or promoted from another user's record. This is the check that catches the most dangerous bug a background job can have.
- One current value per single-valued attribute. Home city, plan and preferred channel each have at most one current row per user after the run, the same signal the failure catalogue recommends for contradictions.
- No forgotten value came back. Every tombstone is checked against current facts after promotion.
- Bounded shrink. If a user's current facts fall by more than half in one run, stop and look; aggressive pruning is the failure the OpenAI cookbook lists first.
- Golden probes. A small set of known (user, attribute, expected value) answers, ideally from internal test accounts with scripted histories, must still hold.
Two further checks are worth adding once the basics run. Sample a few promoted facts per night and have a second model confirm each one against the episodes it cites, which catches a summariser that invents. And keep the per-run action counts (expired, merged, superseded, promoted, sent to review): a night where the number of supersedes jumps tenfold is a prompt change or a model change, not a sudden wave of users moving house.
How unattended consolidation fails
A bad write nobody catches. The defining risk of a job that runs at 3am is that its mistakes are served at 9am with full confidence. The OpenAI cookbook calls consolidation "the most sensitive and error-prone stage" and warns that "Poor consolidation can lead to context poisoning, memory loss, or long-term hallucinations." The defence is the structure above: a copy, checks, a rejected run that pages someone, and a rollback.
Cross-user merges. A job that clusters similar memories globally, rather than per user and per tenant, will eventually merge two customers who both "work at Acme". The lineage check above exists for this case.
Lossy summaries. Summarising episodes drops qualifiers, so "was considering" becomes "is", and each nightly pass over the previous night's summary loses a little more. Promote from the source episodes, store their IDs on the new fact, and never summarise a summary without the originals.
An inference overwriting the user. When the job promotes a pattern ("seems to have moved to Austin, based on recent searches"), it creates a fact the user never stated. If that fact can supersede what the user did say, the job has rewritten the user's memory on a guess. Inferred facts should only fill gaps or go to review, which is what the source column enforces.
Resurrecting what the user deleted. A dream that reads past transcripts can rediscover a fact the user already asked you to forget, if the transcript still contains it. Anthropic's dreams take up to 100 past sessions as input, so this applies to any job built the same way. Delete from the transcripts as well, or keep tombstones the job checks before every promotion.
Lost daytime writes. A job that snapshots at 2am and swaps at 3am throws away anything written in between unless it carries those writes over, which is what the last test checks.
Half-finished runs. A crash or timeout mid-run on an in-place job leaves the live store half-consolidated. On a copy, it leaves an incomplete copy and an unchanged live store, which is Anthropic's documented behaviour for a failed dream.
Promoting poison. Anything an attacker gets into a single conversation is one observation; a consolidation job that promotes repeated observations into durable facts is the step that makes a planted instruction permanent if the attacker repeats it across sessions. Scan candidate facts for instructions before promotion, and treat the evidence threshold as a security control, not only a quality one.
Who finds out when last night's run went wrong
A rejected run is the alert. Route it to the same place as a failed migration, with the run ID, the problems list and the path to the rejected copy, and make the default behaviour "keep serving yesterday's store", so a failure costs one night of freshness rather than a morning of wrong answers. A run that succeeds but changes far more than usual deserves a softer alert of its own.
Over weeks, measure whether consolidation is doing its job with the signals from the failure catalogue: near-duplicate clusters per user and users with two current values for a single-valued attribute should trend down after each run, and retrieval precision on a fixed probe set should hold as stores grow. Hindsight adds a cost signal worth keeping: a policy that works "will keep per-turn token usage flat as the index grows."
What the platforms ship, and when to build your own
| Runs as | Writes to | History and rollback | Review | |
|---|---|---|---|---|
| Anthropic Dreams | API job you create; research preview, access by request | A new memory store; input never modified | Memory versions kept 30 days; restore by writing a version back | You choose when to switch sessions to the output |
| Letta | Background subagents after N steps or on compaction | Git-backed memory filesystem (MemFS) | Git history; backup before reorganising | Optional second agent review, no human approval |
| Mem0 Dream | Weekly synthesis; merge and supersede at write time; Pro and Enterprise | Same store, lifecycle states | Merged and superseded memories kept with pointers | Dashboard review after the fact |
| AWS AgentCore Memory | Consolidation at write time, asynchronously | Same store | Outdated memories marked INVALID | Custom prompts to change rules |
| Your own job | Cron or queue worker | Whatever you build | Whatever you keep | Whatever you build |
AWS's write-up describes one choice worth copying and one worth questioning. It handles failure per memory, so one bad consolidation does not stop the rest, and it says that "if consolidation ultimately fails, the memory is added to the system to help prevent potential loss of information." That trades a possible duplicate for no lost data, which is a defensible default, provided something later cleans up the duplicate.
Build your own job when your memory lives in your own database and you need the consolidation rules to match your domain: which attributes are single-valued, what counts as enough evidence, and which sources outrank which. Use a platform's when your memory already lives there; the job needs the same lineage and status model as the store it runs over, and bolting a homegrown consolidator onto a store that was not designed for supersession is where most of the failure modes above come from.
How Maximem Synap handles consolidation
Maximem Synap runs consolidation in the background at three cadences: a light pass every few hours, a deeper one once a day, and the heaviest work, consolidation and conscious forgetting, during your product's quiet hours, so the expensive work happens when your traffic does not need the capacity. Every memory keeps its provenance (where it came from, what it replaced, why it changed and what it used to say), so a change is recorded and the old version stays readable, and a newer statement that is poorer than the one it resembles never replaces it, which means "takes a cholesterol medication" does not quietly overwrite "takes 20mg of Atorvastatin daily". When an entity match is genuinely ambiguous, it goes to a review queue for a person to decide. Writes return immediately and extraction runs in the background, so the agent never waits on any of it.
Consolidation done right is invisible to the user, and that is the point: the agent stops repeating what stopped being true, and the customer stops having to correct it.
Frequently asked questions
What is agent memory consolidation?
It is a background process that reorganises an agent's stored memory outside the conversation: merging duplicates, marking replaced facts as superseded, expiring time-bound facts and turning patterns repeated across conversations into durable facts. It complements write-time resolution, which only sees the few memories similar to each new message.
What is sleep-time compute?
Sleep-time compute, introduced by Lin et al. in April 2025 (arXiv 2504.13171), lets a model reason over a context offline, before queries arrive, so less work is needed when the user is waiting. Applied to memory, it means a background agent rewrites the agent's memory between conversations; Letta calls this a sleep-time agent.
What is memory "dreaming" in AI agents?
Dreaming is the product name for scheduled consolidation. Anthropic's Dreams for Claude Managed Agents read a memory store and up to 100 past sessions and write a new, reorganised store; Mem0's Dream merges, supersedes and synthesises memories in the background; Letta's dreaming uses background subagents to update memory.
How often should memory consolidation run?
Run a cheap deterministic pass (expire and merge) often, and a model-driven pass (promote and resolve) nightly or weekly for users with new activity. Letta triggers after a set number of agent steps or on compaction, Mem0 synthesises weekly, and the OpenAI Agents SDK cookbook consolidates at the end of each session.
Should a consolidation job delete memories?
No. It should change statuses and record a pointer to the replacing memory, keeping old facts as history hidden from default retrieval. Mem0, AWS AgentCore and Anthropic all avoid destructive consolidation. Deletion belongs to the user-erasure path, which should also stop the job from re-deriving deleted facts.
How do you verify a memory consolidation job worked?
Run it on a copy, then check before swapping: no rows deleted outright, no lineage across users, one current value per single-valued attribute, no forgotten value restored, no large drop in a user's facts, and a set of known answers still correct. Reject and alert on any failure, and keep the pre-run store for rollback.
Can I consolidate memory stored as markdown files?
Yes, but the pass has to rewrite the file rather than append to it, moving replaced facts into a dated history section, and it should write a new file and diff it against the old one before replacing it. An append-only markdown memory keeps both "lives in Boston" and "moved to Austin" as current.
Sources
All sources were read on 29 October 2026.
- Dreams, Claude Managed Agents documentation, Anthropic
- Using agent memory, Claude Managed Agents documentation, Anthropic
- Dream: Background memory consolidation for AI agents, Rudraj Mehta, Mem0, 4 August 2026, updated 28 September 2026
- Memory & dreaming, Letta documentation
- Sleep-time Compute, Letta, 21 April 2025
- Sleep-time Compute: Beyond Inference Scaling at Test-time, Lin et al., arXiv 2504.13171
- Auto-Dreamer: Learning Offline Memory Consolidation for Language Agents, arXiv 2605.20616
- LightMem: Lightweight and Efficient Memory-Augmented Generation, arXiv 2510.18866
- The Consolidation Problem in Agent Memory, Ben Bartholomew, Hindsight, 21 May 2026
- Context Engineering for Personalization: State Management with Long-Term Memory Notes, OpenAI cookbook
- Amazon Bedrock AgentCore Memory long-term memory pipeline, AWS Machine Learning blog
- Delayed Background Memory Processing, LangMem documentation
- Harrison Chase on X, 28 October 2026
- LangChain on X, 28 October 2026
- Sentient OS launch post, Aditya Vellanki on X, 27 October 2026 and Rohan Paul's summary, 27 October 2026
