This is a 4-part series on how agents remember: building short-, medium- and long-term memory that scales across users, agents, and kubernetes clusters.LLMs are stateless by design, and without added memory logic every session starts from zero.A number of dedicated memory layers have emerged (and continue emerging almost daily) to tackle this, each with different approaches and tradeoffs. Which one should you adopt?Recently I spent some time extending the Kubernetes Agent Orchestration System (KAOS) to support multi-tiered memory persistence (aka short-, medium- and long-term memory). Along the way I hit most of the same issues that anyone would whilst building or integrating multi-tiered memory into an agentic system, so I thought it would be useful to compile all the learnings, design choices and examples into this 4-part series.Hopefully this post is useful for anyone looking to do this on their own project. My objective:Let’s make the memory layer BORING, so that the agents can continue to be the fun part.This first part includes the research findings from exploring ~30 tools, together with the taxonomy and the baseline implementations that motivate needing them, including tools like Mem0, Zep/Graphiti, Letta (MemGPT), Cognee, Memobase, Redis Agent Memory Server, as well as native implementations in OpenAI’s products, Claude, LangGraph, CrewAI, and Google ADK, among many others.Throughout the series, I also share the learnings and best practices that came out of navigating through a large number of architecture tradeoffs, and getting my hands dirty on the implementation that now ships as a distributed, highly available, and scalable MemoryStore resource that any agent can bind to.As with my previous posts on observability for agentic systems and autonomous always-on agentic patterns, I will use KAOS as the concrete implementation example (concretely from Part 2, hands-on in Part 4), but the goal is to provide practical intuition for the primitives (tiers, scopes, folding, degradation), so that it applies whether you use KAOS, Mem0 directly, LangGraph, CrewAI, or a memory layer you wrote yourself.This post is the first of a 4-part series, which we plan to release weekly over the next couple of weeks:Part 1 (this post): What agent memory is and what to build on. The taxonomy, the baseline implementations everyone starts with, and the engine landscape from surveying ~30 tools.Part 2: Tiers and scopes for multi-tenant agents. The three-tier design and the answer to whose memory it is (coming soon…).Part 3: Memory as infrastructure. The Kubernetes MemoryStore resource, its deployment topology, and how to integrate it in your own agent (coming soon…).Part 4: Agent memory in action. A worked example that runs end to end on a secured cluster, with real outputs (coming soon…).A Working Taxonomy of Agent Memory“Memory” is one of the most overloaded words in agentic systems, so it is worth separating it from the concepts it gets conflated with, such as:The context window, which holds working state for a single model call.Session history, which holds an auditable transcript of what was said.Prompt telemetry, which holds the specific prompts relative to events in the system.To be more precise we can look at Princeton University’s paper on Cognitive Architectures for Language Agents (CoALA) to provide a more precise definition for “Memory” in agentic systems. We can define “Memory” as the component that holds the short-, medium- and long-term information an agent carries across turns and sessions to inform its reasoning.This research paper quoted also provides a useful taxonomy for “memory types” that we will use to reason throughout the series, and especially in the tier design of part 2. This includes the memory types for episodic, semantic, procedural and temporal memory.These memory types are also mentioned in the Berkeley paper that released MemGPT, as well as how the Stanford paper on large-scale LLM simulations Generative Agents: Interactive Simulacra of Human Behavior structured their memory event stream.The formal definition of these memory types (+ a few examples) is outlined as follows:Memory typeWhat it holdsExampleShort-term (working)Verbatim recent turns of the live conversation“The user just said port 8080”EpisodicRecords of specific past events“On Tuesday the deploy failed twice”SemanticDistilled, durable facts“The user prefers blue-green deploys”ProceduralLearned skills and how-tos“Here is how we roll back this service”TemporalFacts with validity intervals“Joe was in a relationship until March, but not anymore.”In practice what I found out however is that most frameworks only implement a small number of these, namely short-term is always present, episodic and semantic are bundled (the only difference is whether time is preserved), procedural tends to be present mainly in coding agents (eg creating skills, commands, extensions), and temporal tends to be replaced with “forgetting memory” functionality instead, or embedded with episodic/semantic.These appear more informally defined as:Conversational continuity: The agent remembers what was said three turns ago; a same-session problem.Learned knowledge: The agent remembers what it figured out last week; a cross-session problem.For example, frameworks like LangGraph separate thread-scoped checkpointers from a cross-thread store. Another example is Letta, which separates always-in-context memory blocks from an archival tier.Most of the design mistakes I made early came from either trying to tackle all of these “memory-types” separately, by bundling sub-optimally, or by oversimplifying too much.But before we dive into the implementation, let’s cover the basics.Memory 101: The Version Everyone Starts WithAlmost every agent system starts with the same memory implementation:memory = []async def handle_message(user_message): memory.append({"role": "user", "content": user_message}) response = await run_agent(memory[-20:], tools) memory.append({"role": "assistant", "content": response}) return responseAnd to be honest, the original KAOS memory was exactly this. It was an in-process queue with a max length, which ensured it was replaying the last N events into the next prompt.The second version everyone builds is “just embed everything”:async def handle_message(user_message): hits = await vector_store.search(embed(user_message), top_k=5) context = "\n".join(h.text for h in hits) response = await run_agent([context, user_message], tools) await vector_store.add(embed(user_message), user_message) return responseThis is better, but this is not memory in the form that we introduced eariler, it is just a better search mechanism across the prompt history.Another tempting alternative as the next step is “context windows are huge now, just replay everything”.However this is not a great approach, and there are some benchmarks like UCLA’s Bench on Long-Term Interactive Memory, which showed that models reasoning over full ~115K-token interaction histories lose 30-60% accuracy versus the same models given oracle retrieval.If we look at it from a feature / functionality standpoint, we can summarise the gaps between the base and the production implementation as follows:Naive memoryProduction memoryLast-N turns, unbounded token growthToken-budgeted window with principled evictionVerbatim replay of everythingDistilled facts, separated from the transcript (eg long- / short-term)One user, one processMany tenants, many agents, many replicasMemory lives inside the agent podMemory survives restarts and is shared across the fleetWrites block the responseExtraction runs off the hot pathNothing is ever forgottenDecay, retention, and right-to-erasureMemory failure crashes the turnMemory failure degrades the turnIn this case we can position “production memory” a tiered, scoped, context-specific and dynamic store, as opposed to purely a vector database connected to an agent.Achieving this in a way that scales does get complex, as we need to decide who can see each memory tier, when we store facts, and how the agent behaves when memory fails.However now that we have the conceptual foundation in place, we can start looking at these functionalities relative to the frameworks available.Choosing an Engine: Build, Adopt, or WrapBefore designing anything, I surveyed the landscape thoroughly, assessing dozens of tools across three tiers, and we will cover the scope, approach and learnings in this section, starting with an overview of all the tiers as follows.Tier 1: Dedicated memory frameworks. This tier encompasses purpose-built frameworks whose whole job is agent memory. From the longer list, we reduced it to the actively maintained ones:CandidateApproachStoreStrengthWeaknessMem0extracts facts from conversations into a vector store and recalls them by similarityQdrant, pgvector, othersmost adopted, cleanest library integrationno OTel, tenant isolation only at application levelZep / Graphitibuilds a temporal knowledge graph where facts carry validity intervals and provenanceNeo4j or FalkorDBrichest memory model, time-aware fact invalidationheaviest to operate, costliest writesCogneecombines a knowledge graph with vector search, populated by an extract-and-load pipelineLanceDB by default, Postgres or Neo4j optionalmulti-tenancy and OTel built inearly stage, heavy dependencies, changing APIMemobasemaintains structured user profiles and event timelines, with no embeddings on the hot pathPostgres + Redischeapest write pathprofile-only recall, weak self-hosted multi-tenancyRedis Agent Memory Serverserves two memory tiers (working and long-term) behind one REST APIRedisthe two-tier model mirrors what agents actually needyoung project, no OTelWhat this tier taught me is that the architectural differences are really differences in recall pattern and write cost. Vector-first designs answer “what do we know about X” cheaply, graph-first designs answer “how did this fact change over time” at the price of an LLM-heavy ingestion pipeline plus a graph database. The profile-first designs answer “who is this user” with no embeddings on the hot path at all, and the two-tier designs bake in the working-versus-long-term split directly.There were also clear shared gaps, mainly at the infrastructure level; none of them enforces tenant isolation below the application level, and almost none ships OpenTelemetry, so whichever one you pick, scope enforcement and observability become your integration work. That shared gap shaped the KAOS design more than any individual feature did.It’s also worth noting that several of these libraries also offer an enterprise tier, so it was important to validate that basic features are not gated behind a paywall (similar to what we previously experienced with Google ADK and Vertex). More specifically Mem0’s own platform-versus-OSS documentation gates temporal reasoning, memory decay, webhooks, export, analytics, and auto-scaling behind the managed platform, and Zep draws the line at governed multi-tenancy and compliance, with the OSS Graphiti engine giving you a single context graph to run yourself. The pattern across vendors is that the memory algorithms are open while the operational maturity is the commercial product, which previews the exact layer a platform adopting one of these engines has to build.Tier 2. Agent frameworks with native memory. This tier encompasses the embedded memory functionality across end-to-end agentic frameworks, and included the usual suspect / popular frameworks like LangGraph’s Store and LangMem, CrewAI memory, LlamaIndex memory, Google ADK’s MemoryService, and the Microsoft Agent Framework. These were reviewed for their design choices, but adopting one for its memory means importing a second agent runtime next to your own, so they served as references and not as candidates.The learning from this tier is actually what they all had in common. Every framework independently separates session-scoped state and cross-session knowledge, such as how LangGraph has thread-scoped checkpointers versus its cross-thread Store.There was also a clear separation between “local playground” and “production grade” when it comes to memory for all frameworks.LangGraph: In-memory store is for development, but it’s recommended to use a database-backed checkpointer and store for production.Google ADK: Heavier paywall, as it only offers the InMemoryMemoryService as open source, but anything serious would need to use Vertex AI.Microsoft Agent Framework: Defaults to an in-memory context provider, with durable state via first-party Cosmos checkpoint storage, and ships a first-party Mem0ContextProvider.CrewAI: Community documented replacing its native store with Mem0 after hitting redeploy and user-isolation gaps.Also interesting learnings from agent runtimes and coding agents:OpenClaw: layers markdown memory files (MEMORY.md, dated notes, per-skill SKILL.md) and runs a “Skill Workshop” where the agent proposes new skills from successful conversations for human approval.Hermes agent: Uses skills explicitly as procedural memory, which are auto-proposed after repeated successful tool-call patterns, carry semver versions bumped on each self-improvement, and follow an anti-sediment principle where a skill should get shorter and sharper over time.Claude Code skills: Uses progressive disclosure, loading only ~100 tokens of skill metadata until a skill is triggered.Native memory is increasingly an extensible interface where the shipped default is a placeholder, which means memory is being externalized by design across the ecosystem, and the dominant production pattern is framework plus engine. This is something that we had to take into consideration as well.Tier 3. Managed and commercial services. This tier included commercial services with managed memory platforms, which provided insights on the broader design of the system and the interactions with the memory, as opposed to just the design of the memory capability itself. These included the Mem0 Platform, Zep Cloud, Letta Cloud, OpenAI memory, and Google’s Vertex Memory Bank.The learnings from this tier were also quite helpful to understand some of the architectural and feature tradeoffs that were done at the platform level. Every managed platform has the same two-layer model: namely 1) an explicit, user-curated layer (eg. OpenAI’s saved memories, Claude’s editable memory summary) that is visible/available at the surface, and that is build on top of; 2) an automatically inferred and consolidated layer (eg. OpenAI’s chat-history reference, Vertex Memory Bank’s LLM extraction with per-scope deduplication and contradiction checks) where the memory store/retrieval algorithms live.There was however a clear distinction on the scope in which memory is available across each platform: For Claude, memory scope is per project, in Vertex scope is per identity with configurable memory “topics”, and Zep scopes per subject graph. None of them defaults to one global memory per account, which makes it clear that there is a design decision required on the isolation boundary.Conclusions from Surveying the EcosystemGiven the review was done in context of KAOS, the lens / considerations through which these were reviewed included the following non-exhaustive list:Long-term capability coverageRetrieval qualityEmbeddability as a libraryPluggable storage backendsInfrastructure delta / overheadMulti-tenancy hooksObservabilityLicensingMaturityWrite-path costBased on these, the library that clearly stood out was Mem0. At least at the time of writing, Mem0 maximized the features and capabilities with the lowest integration friction. Mem0 also has the strongest ecosystem maturity, and pluggable stores.It’s however worth noting that despite Mem0 being the right choice for this context, one learning that may seem obvious in retrospect was that there is no “Perfect Candidate”. The graph-first leaders (Graphiti, Cognee) have the most features but at the highest cost. Low-delta options (Redis AMS) buy fit at maturity cost. Building it yourself directly on the raw vector or graph stores, which we also weighed as the baseline option, allows you to have all the features and fit, but at the cost of rebuilding mature extraction and retrieval that already exists under permissive licenses.This also applies to the numbers the frameworks publish about themselves. Interestingly enough, Mem0’s own research supports that extraction-based memory improves latency and cost, however it does not improve raw accuracy: in Mem0’s own evaluation on LoCoMo, a full-context baseline beats Mem0 on raw accuracy (72.9% vs 66.9%), while memory buys a 91% cut in p95 latency (1.44s vs 17.1s) and over 90% fewer tokens per conversation. At fleet scale that trade is exactly right, since you cannot ship 17-second turns and 26K-token replays, but it is a trade you should make knowingly.As part of this, despite Mem0 being the strongest choice, it became clear that adopting a memory engine means choosing which 60% of the system you do not have to build, and committing to build the remaining 40% around it.For Mem0, this meant working on the bridge to close some of the gaps, particularly at the infrastructure and interoperability layer. These included:Enabling telemetry by instrumenting every operation and ensure correlation+consistency with the broader KAOS telemetry.Introduce tenant isolation, as this is enforced at the Mem0 application level, so enforce scope through the memory service.Bundle up the kubernetes packaging to ensure high availability and scalability as a distributed service.Bridge the short- and medium-term memory with a native integration with the Pydantic AI server that we have built as part of KAOS.Each of these gaps becomes a design decision in parts 2 and 3.Based on these initial decisions we were able to proceed to the architecture of the memory system itself, which is exactly where part 2 picks up.Closing Thoughts for Part 1This first part covered the ground you need before writing any memory code with a working taxonomy that separates memory from the context window and the session transcript. We also reviewed the baseline implementations and the understood some of the limitations. This included the survey of the memory engine landscape, and the thinking process that went into selecting the memory framework to build upon.However this is only the beginning as choosing the memory framework is only 40% of the work, we still need to build the remaining 60% to ensure we can integrate it in a coherent and scalable way for our distributed agent system use-case.In part 2 we take that position and design the memory system itself. This includes the three tiers that separate “memory tiers”, and the scope model that answers the title’s question of “whose memory is it?”. Stay tuned, part 2The series:Part 1 (this post): What agent memory is and what to build on.Part 2: Tiers and scopes for multi-tenant agents (coming soon…).Part 3: Memory as infrastructure (coming soon…).Part 4: Agent memory in action (coming soon…).