Semantic Kernel Memory: How to Add Memory to an Agent

Agent FrameworksAI Agent MemoryMaximem Team3 June 20269 min read
Semantic Kernel Memory: How to Add Memory to an Agent
On this page
  1. Where Semantic Kernel Excels
  2. How Semantic Kernel Memory Works Today
  3. What Synap Adds
  4. Technical Deep Dive
  5. What Synap Adds to Semantic Kernel
  6. What Production Teams Gain
  7. How to Get Started
  8. If You Are Moving to Microsoft Agent Framework

Semantic Kernel gives an agent ChatHistory, an in-process list of messages for one conversation, with ChatHistoryTruncationReducer and ChatHistorySummarizationReducer to keep it short; saving and reloading it between sessions is your code. Vector store connectors add storage and similarity search over records you write yourself. For memory across threads, the agent framework documents Mem0Provider and WhiteboardProvider: the first stores memories under a user id in the Mem0 service, the second keeps short-term notes, and Microsoft marks both experimental. Maximem Synap adds persistent per-user memory as a kernel plugin with two functions, search_memory and store_memory, scoped at plugin construction by user_id and an optional customer_id.

One thing to know before you invest further: the Semantic Kernel README now says that Semantic Kernel is now Microsoft Agent Framework, which it describes as the enterprise-ready successor, and Microsoft publishes a migration guide. This guide is for teams with Semantic Kernel agents already in production. For a new project, skip to the Agent Framework section near the end; Synap has a separate package for it.


Where Semantic Kernel Excels

Semantic Kernel is Microsoft's orchestration SDK for .NET, Python and Java, and its center is the kernel: a container for AI services and plugins. A plugin is a class whose methods are exposed to the model as kernel functions, and with function calling enabled the model decides which to invoke. The older Stepwise and Handlebars planners have been removed from the package; Microsoft's guidance is to use function calling for planning. If your agent has to call typed business functions inside the Microsoft stack, that plugin model is what the framework does well.

A memory layer attaches at the same point, as one more plugin on the kernel.


How Semantic Kernel Memory Works Today

Semantic Kernel has four memory surfaces, and they differ in how much they do for you.

ChatHistory is the conversation store. You add turns with add_user_message, add_assistant_message and add_message, and the list grows with every exchange. In the Python package it can be serialized and restored (serialize, restore_chat_history), which makes saving it between sessions a matter of choosing where to put the JSON. That choice, and keying it per user, is yours.

History reducers keep the list bounded. ChatHistoryTruncationReducer cuts the history to a target_count once it passes the threshold, and ChatHistorySummarizationReducer replaces older messages with a summary. Both preserve system messages. They manage one conversation's length.

Vector store connectors give you collections and records with vector search over stores such as Qdrant. Microsoft's page labels the feature a preview and points to a separate page for the legacy Memory Stores (in the Python package, VolatileMemoryStore is marked deprecated). A connector retrieves what is similar to a query; what goes into a record is up to your code.

Agent memory providers are the closest thing to built-in long-term memory. Mem0Provider sends each thread message to Mem0 for extraction and queries it on each invocation, so an agent can recall a user's preferences across threads; WhiteboardProvider extracts requirements, decisions and actions into short-term notes. They attach to a ChatHistoryAgentThread through its AIContextProviders collection. The documentation shows them in C# and marks the feature experimental, and we did not find either class in the Python package (1.45.0) that this guide uses.

So the Python developer's position is this: conversation history and its trimming are handled well, and memory about a user that outlives the thread is something to add.


What Synap Adds

Synap is agentic context management. Its Semantic Kernel package, maximem-synap-semantic-kernel 0.2.1, is a Python package that requires semantic-kernel 1.0 or later; the repository has no .NET equivalent. It exports five names.

SynapPlugin is the core. Register it on a kernel and the model gains two kernel functions. search_memory takes a natural-language query and returns a formatted context string drawn from the user's memory. store_memory takes the content to remember, submits it for ingestion and returns a confirmation carrying the ingestion id. Scope is set when you construct the plugin, through user_id and an optional customer_id, so the model only ever supplies the query or the content.

SynapAgentThread is a ChatHistoryAgentThread that records each user and assistant turn to Synap as it is added to the thread, over the SDK's live stream when one is open and through sdk.conversation.record_message when it is not. With it, memory no longer depends on the model deciding to call store_memory.

attach_synap_filters registers an auto function invocation filter on the kernel (built by synap_auto_function_filter) that reports each tool call and its result on Synap's live stream, when one is open. synap_st_chat_message returns an async factory for a system message carrying Synap's compacted summary of the current conversation.

We built this because agents kept stalling in production. The problem was not bad orchestration. It was missing context that lived in a different session last Tuesday. Production testing hit 92% LongMemEval, 93.2% on LoCoMo.

For why context management is infrastructure and not a feature, read What Is Agentic Context Management?. For build-versus-buy numbers, see The Real Cost of DIY Agent Memory.


Technical Deep Dive

LongMemEval Benchmark

LongMemEval tests whether agents recall facts across long, multi-turn conversations spanning multiple sessions. The benchmark simulates production conditions where users return days apart and expect the agent to remember prior context. Synap scores 92% on this benchmark.

Entity Resolution Mechanism

When the same person, place or thing appears across conversations under different labels ("John", "Mr. Smith", "my manager"), Synap's ingestion pipeline links them to one canonical entity. This happens during ingestion, not at query time, and it respects scope: entities resolved for one user do not leak to another.

Compaction and Short-Term Context

Synap keeps a compacted summary plus the recent turns for each conversation. synap_st_chat_message(sdk, conversation_id, system="...") gives you a factory; await it before each invocation and add the returned message to your ChatHistory. The summary is prepended above your own system text, in a style of structured, narrative or bullet_points. If the call to Synap fails, the default is to return your system text alone, so the kernel keeps running.

Fast and Accurate Retrieval Modes

The SDK documents fast mode as vector similarity only. Accurate mode adds a graph layer that traverses relationships between entities, then re-ranks, and it is the slower of the two. search_memory always uses accurate mode in this version of the plugin; there is no argument to change it.

Multi-Tenant Scoping

Three ids set scope: user_id identifies the person and is required, customer_id identifies the tenant on B2B Synap instances (a B2C instance does not accept one), and conversation_id, which must be a UUID, identifies one conversation.

For the Semantic Kernel integration, scope is fixed when the plugin is constructed. Build a fresh kernel and plugin instance for each user's request so that concurrent users never share one.


What Synap Adds to Semantic Kernel

Persistence

Semantic Kernel Native. ChatHistory in process, serializable by your code; experimental agent memory providers documented for .NET. With Synap. Hosted per-user memory, written by store_memory and by SynapAgentThread.


Entity Resolution

Semantic Kernel Native. Not part of chat history or the vector store connectors. With Synap. Mentions of the same entity under different labels are linked during ingestion, inside the user's scope.


Compaction

Semantic Kernel Native. Truncation and summarization reducers on ChatHistory. With Synap. A compacted summary plus recent turns through synap_st_chat_message.


Retrieval

Semantic Kernel Native. Vector search over records you wrote to a connector. With Synap. search_memory, in accurate mode, over memories extracted from the user's conversations.


Long-Term Recall

Semantic Kernel Native. Not benchmarked for cross-session recall. With Synap. 92% on LongMemEval.


Failure Handling

Semantic Kernel Native. During automatic function calling, the Python kernel catches an exception raised by a function, logs it and gives the model an error message as the function result. With Synap. Both plugin functions wrap SDK failures in SynapIntegrationError, logged once at ERROR with the operation name; a failed search raises, it does not return an empty result. SynapAgentThread updates the chat history first and never lets a recording failure reach the agent run.


User Scoping

Semantic Kernel Native. Yours to implement. With Synap. user_id and optional customer_id set at plugin construction.


What Production Teams Gain

Cross-session continuity. Your user chats on Monday and returns on Wednesday. search_memory recalls what was established. store_memory persists new facts the agent learns, and SynapAgentThread records the turns whether or not the model calls it.

Accuracy that ships. 92% LongMemEval, 93.2% on LoCoMo measures whether agents recall facts across long, multi-turn conversations spanning multiple sessions. This benchmark tests the specific failure mode that breaks production agents: accurate recall over distance and time.

Writes that do not hold the turn. store_memory returns as soon as Synap accepts the ingestion job; extraction runs afterwards on the server.


How to Get Started

Step 1: Install

pip install maximem-synap-semantic-kernel

This pulls in maximem-synap and semantic-kernel. The import name is synap_semantic_kernel, and the SDK itself is imported from maximem_synap.

Step 2: Initialize and register the plugin

import asyncio
import os
import uuid

from semantic_kernel import Kernel from semantic_kernel.agents import ChatCompletionAgent from semantic_kernel.connectors.ai.open_ai import OpenAIChatCompletion from maximem_synap import MaximemSynapSDK from synap_semantic_kernel import SynapAgentThread, SynapPlugin

async def main(): sdk = MaximemSynapSDK(api_key=os.environ["SYNAP_API_KEY"]) await sdk.initialize()

user_id = "user_123"

# Create a user-scoped memory plugin and add it to the kernel
kernel = Kernel()
kernel.add_service(OpenAIChatCompletion(ai_model_id="gpt-4o"))
kernel.add_plugin(SynapPlugin(sdk=sdk, user_id=user_id), plugin_name="synap")

# The agent can now call search_memory and store_memory as kernel functions
agent = ChatCompletionAgent(
    kernel=kernel,
    name="support_agent",
    instructions=(
        "Call search_memory before answering questions about the user. "
        "Call store_memory when the user states a durable fact."
    ),
)

# Optional: a thread that records every turn. The conversation id must be a UUID.
thread = SynapAgentThread(
    sdk, conversation_id=str(uuid.uuid4()), user_id=user_id
)

response = await agent.get_response(
    messages="What did I tell you about my billing plan?", thread=thread
)
print(response)

await thread.delete()
await sdk.shutdown()

asyncio.run(main())

Step 3: Deploy

For multi-tenant production, construct the plugin per request with that user's ids and add customer_id= on a B2B instance. Call await sdk.initialize() once at startup; the SDK refuses every operation until it has run.

Full config, scoping rules, and error handling: see the integration docs (https://docs.maximem.ai/integrations/semantic-kernel)


If You Are Moving to Microsoft Agent Framework

Agent Framework has context providers for agent memory, and Synap's package for it is maximem-synap-microsoft-agent 0.3.1, a Python package that requires agent-framework 1.0 or later.

pip install maximem-synap-microsoft-agent
from agent_framework.openai import OpenAIChatClient
from synap_microsoft_agent import SynapContextProvider, SynapHistoryProvider

# sdk is an initialized MaximemSynapSDK, as in Step 2

client = OpenAIChatClient(model="gpt-4o-mini")

agent = client.as_agent( name="MemoryAgent", instructions="You are a helpful assistant.", context_providers=[ SynapContextProvider(sdk=sdk, user_id="user_123"), SynapHistoryProvider(sdk=sdk, user_id="user_123"), ], )

SynapContextProvider fetches Synap context before each run and records the turn after it. SynapHistoryProvider persists the message log and loads it on session resume. The package also backs the memory and file stores of Agent Framework's harness, which needs agent-framework 1.13 or later and is marked experimental upstream.


Memory Is Infrastructure

Semantic Kernel gave .NET and Python developers a framework for orchestrating agents, and the ChatHistory it ships handles one conversation well. Deciding what to remember about each user, keeping it correct as it changes and retrieving it weeks later is a different layer of the stack, and it stays the same layer when you move to Agent Framework.

Memory is infrastructure, not a feature.

Start building Semantic Kernel agents that remember across sessions (https://synap.maximem.ai)

Synap pricing is usage-based. You pay for memory operations: storage, retrieval, compaction. No per-seat or per-framework surcharge. The $49/month starter plan includes a base allocation; usage beyond that is metered by operation. Every new account gets $25 in free credits to test before committing. See the full pricing page (https://synap.maximem.ai/pricing).


Related Posts


Frequently Asked Questions

Does Semantic Kernel have built-in memory?

ChatHistory holds one conversation and can be trimmed with the truncation and summarization reducers. Vector store connectors give you similarity search over records you write. For long-term memory about a user, the agent framework documents Mem0Provider and WhiteboardProvider, shown in C# and marked experimental. See the Semantic Kernel agent memory documentation (https://learn.microsoft.com/en-us/semantic-kernel/frameworks/agent/agent-memory) for details.

How is Synap different from a vector store connector?

A vector store connector stores the records you give it and returns the ones closest to a query. Synap takes conversations, extracts memories from them, links mentions of the same entity, and in accurate mode adds graph traversal and re-ranking to retrieval. Synap hosts the records, so memory survives restarts.

Is Semantic Kernel still the right place to start?

For a new project, Microsoft points to Agent Framework, and Synap's package for it is maximem-synap-microsoft-agent. For agents already running on Semantic Kernel, the plugin in this guide is the shorter path.

Can I use this in production today?

The plugin uses standard kernel functions. Both functions raise SynapIntegrationError when a Synap call fails; inside the function-calling loop the model sees that as an error result, so watch your logs. For multi-tenant deployments, construct a fresh plugin instance per request.

How does pricing work?

Usage-based. Storage, retrieval, and compaction are metered separately. No per-seat or per-framework fee. The $49/month starter plan includes a base allocation; usage beyond that is metered by operation. Every new account gets $25 in free credits. See the full pricing page (https://synap.maximem.ai/pricing).

From the team at Maximem

Stop rebuilding agent memory from scratch

Maximem Synap is the context management layer we built after hitting every problem in this post ourselves. Persistent recall across sessions, entity resolution and conscious forgetting, in Python, TypeScript and REST.

Related posts

Looking for more to read? See our directory of the best engineering blogs to follow.