From Context to Memory: Your Agents Need a Real Memory Layer — Anders Swanson, Oracle

0:01 [music] >> Hey, so I'm Anders. I'm a developer advocate for Oracle AI database and I'm talking about how we can approach agent memory as a systems

0:22 problem, how we can take context and turn it into a real memory layer that is going to help your agents work better. Um when we're talking about agent memory, we're talking about composing

0:34 memory retrieval, memory formation, and the persistent storage that's backing these memories. Um and we've probably all used different agents at this point, agents like Claude

0:46 code or Codex, and we've seen them work on a task and reread the same files over and over again. And this this is because they have no knowledge of what has happened in the past.

0:57 Context windows are are not memory. Um they're the input for the current run and not a persistent experience. If I have some agent session, let's say he's working on customer support tickets, he

1:10 reads a bunch of documents, he runs a bunch of queries, he reproduces a failure, finds root cause, and does all this like great valuable work that synthesizes a useful procedure.

1:20 What happens at the end of the session? It's tossed away. And this is because there's no persistence. Uh you know, a long long later another

1:29 another agent comes on and he works on a similar task and he has to repeat all of the same work. So he's he's searching more documents, he's running more queries, and this is more opportunity

1:38 for failure, more opportunity for hallucinations, and and ultimately uh less fidelity. And memory really cannot begin until these useful experiences

1:50 from our agents are transformed into managed state. Uh larger context windows are also not the solution. We end up throwing more and more tokens. The session grows

1:59 longer, compaction occurs. The useful signal that occurs for our agents is lost in a slew of other irrelevant information. Likewise, scratch pads are workarounds.

2:13 So, things like markdown files, we start building large hierarchies of files, uh wikis, prompts. These do help, but this is not governed recall. Uh files

2:26 do not scale in large multi-tenant environments. Uh tools like grep are extremely useful for finding data, but they also don't scale well, and it is not complex hybrid

2:37 retrieval. And I feel like if we continue to the logical conclusion of trying to build a memory layer on top of files, we're going to be building our own

2:46 database. And I don't I you could just use a database. Uh in this presentation, this is unpacking the lessons that we've learned

2:57 as a database vendor working with customers who are developing agent memory, customers who are solving real problems with agents and needing recall.

3:07 And where we're seeing modern agent memory systems converge is there's there's a pattern that's implementing memory formation, memory recall, evolution, and evaluation.

3:17 So, when we form memories, these are these are distilled experiences, structured facts. This is not a raw transcript dump. When we're recalling memories, hybrid

3:28 retrieval is emerging as a dominant method. This is This is a form of retrieval that fuses different types of search. So, vector similarity search, lexical search, entity search over

3:37 relational objects, graph traversals, and different signals over temporal data, recency, and feedback. We also need consolidation in these systems. So, duplicates can emerge. We

3:48 need to resolve conflicts. The system needs to evolve over time. Otherwise, because it becomes stale, irrelevant, and useless. Memories need element of temporality.

3:57 So, when was this created? When was it accessed? How long is it valid for? It may only be valid for a certain time window. And this all needs to come into regular

4:07 feedback and evaluation. If I'm not measuring use, I'm not sending feedback into the memory layer, it's all going to fall over. Um another

4:18 important piece is the governance controls. So, secrets, personal information, that needs to be redacted before it hits the memory layer. Rights need to be scoped,

4:27 scoped to the agent, scoped to the workflow, the team, the user. This is especially important in multi-tenant environments. We need appropriate admin approvals,

4:36 review, and ability to suppress stale or poisoned data. When we walk through the memory loop, we have an agent session. It does some work, reads documents, run queries,

4:48 synthesizes useful data. After that session or during that session, a fact is created. We record that fact through redacting, enriching, storing it in the persistence

4:58 layer. And then when another session comes along with related context, it can retrieve that fact and use it for context card reconstruction. So, it's

5:06 using that fact that came from a prior art to make the next task easier. That's less work for the next agent to do. And if that agent uses the fact

5:17 successfully, we can add feedback scores to that memory. So, we're adapting over time. We get related data and new temporality. We access it again. We We did something

5:27 with it. Now, memory, it's invoked as a function by the agent runtime. So, it needs read, write, and storage layers. Your agent runtime can invoke the memory

5:40 layer through things like hooks, MCP, skills, or other methods. And the memory layer API is exposing these the read write the feedback administrative APIs and then talking to

5:52 the storage platform or databases. The choice of database here I I think multi-model databases are especially useful because they allow you to not work work with

6:04 your rows, columns, vector embeddings, full text search, relational entities, graphs, episodes and facts all in one place. So your memory can live in a

6:13 consolidated place with one database query connection. I can do all sorts of hybrid search and return that data back. Uh what does the memory look like?

6:26 Uh it's going to be typed and this you may have seen this model at different presentations or some form of it. Um this loosely comes from human memory can be applied to agents.

6:35 So we have episodic memory. This is specific past events or slices of transcripts that the agents worked on. We have semantic memory. Um these are specific facts about our

6:45 workflows, user preferences, and other entities. We have procedural memory. This is like how we did a specific task or routine, maybe a build step. These are reusable

6:54 rules the agents are going to want to pull up again and again if they're are repeating similar work. And we have working state. Working state is like short-term memory. It's what are

7:03 we what are we current currently working on and that that temporal data may only be valid for a certain time window and we may promote it to a different type of memory. This helps us bucket into

7:12 different areas and we can use different search techniques on these types of memory. Uh what does the storage schema look like for memory?

7:20 Um so memory are typed objects plus lookup path. So our our memory object is going to have some ID like a primary key. It'll have its text content, JSON tags, document data.

7:32 It'll have a vector embedding. Uh we have a vector created over the content with a vector index for similarity search. Ideally we'll also have a text index to

7:41 facilitate full text keyword search. That combines with vector search. Uh we can have graph links to different edges. Memories can be related through an entity graph. So, this memory is

7:51 connected to this memory, which is connected to that memory. It's additional context that we can pull back during the search stage. We can also have temporal facts.

7:59 When is this memory valid to? What was the last time that we used it? And then also it's super importantly audit events. Who Who created this memory? What did they use it for?

8:10 Uh the log of of of of how it's been used and associated metadata. Um this should all be part of your storage schema when we're working with memories.

8:20 When we create memories, we want to intentionally form them. Good memories are very intentional. So, we need to decide the life cycle, the scope, the different relationships, and

8:29 associated metadata. When an agent runtime decides to create a memory, we should first start with redaction. Uh redaction is very important as the

8:37 first step because before I do embeddings, before I do enrichment, I want to strip out PII. I don't want secrets or irrelevant weird information slipping

8:46 into the memory layer. That's a recipe for disaster. So, we do redaction first. Then we can enrich the memory. What is the mem- What is the memory related to? What is the metadata of the memory? We

8:57 can apply the scope to it. Uh what is the agent that worked on this memory? What's the team? What's the organization? This helps bucket into specific areas so I'm not getting

9:07 memories in other sessions that are associated with other agents or workflows, which adds confusion. We can do the embedding so we create the vector of the content.

9:14 And then we can form what's called the episode. These are the slices of the transcript that created the memory. This is good for provenance and additional context.

9:23 Um any organization-specific policies here can also be applied. Uh whatever you need for your specific use case. And then we can save down to the storage layer.

9:33 Um when we create memory, the individual memories can be quite compact. So, we take the transcript and we distill it down to that content and metadata and

9:41 embedding and and and related stuff, but we can also we create these episodes. So, we take the agent transcripts, we filter them,

9:49 find candidates for these episodes, save the memory record, and link it back. You can do this with uh relations, just straight-up database relationships. You can do it with

9:58 graphs. I particularly like graphs because they allow a broader search here. Uh when we're talking about memory recall, that is

10:07 how do we remember something? This is a retrieval problem, so a query problem on the storage layer. Um and with with retrieval or recall, hybrid search is emerging as the

10:19 dominant method of context reconstruction. And this this form of retra retrieval can fuse different search methods. So, I can take vector similarity search, I can

10:29 take a full-text keyword search, I can take relational entities, I can do graph traversal, I can take other feedback signals like importance uh

10:39 uh and usefulness of the memory to yield more comprehensive results than one specific query branch alone. Like, if I was only doing similarity search, I only get

10:48 the most similar results according to the embedding model that I'm using. Um and in general, this has not been enough. So, we have all these different query branches,

10:57 um different types of memory, you can decide which ones you want to do. Things like facts, you might not want similarity, you might just want to do like an exact ID match on a fact.

11:07 Um we can take these results, we can fuse them, apply thresholds, do re-ranking, and then we get a context card with scored evidence. So, we can say this

11:15 memory it rate it rated very well on vector search and a little bit lower on lexical search. Um that's the provenance of the memory, where it came from, how we got it.

11:24 That's important, too. Um as we as we walk through hybrid search, uh vector distance useful for for semantically similar things like

11:34 contains operator, very good for lexical search, uh, graph table, temporal data, all this stuff. We take these branches, we apply weights to them. We can await differently as

11:45 needed. This is something that can be tuned in your your hybrid search algorithm. The The results are merged to an overall score from the different query branches,

11:54 and then we can take the top K of the overall results to say hey, here are the top three best results from our hybrid search algorithm. And that's loosely how you can implement

12:03 hybrid search in a multimodal database, a database that is capable of providing all of these different search methods from one pane of glass here. Now, memories, it's not just a

12:15 repository where we push data in and we pull data out. Memories need to evolve over time to be truly useful. We also don't want to just shove all our data in every time,

12:26 otherwise we get tons of memories. Um, and this is where we can kind of close the memory loop with consolidation and error handling. Uh, stale memory can occur when, say, my

12:37 my project uses NPM at one time and then we decide to switch to another build system. Well, what happens to all the memories that were related to NPM? If an agent recalls those, it's it's

12:47 going to be a little tricky and it may try to do the wrong thing. Uh, so the system it needs to expect stale facts over recall quiet drift. Bad scope can occur where we're not

13:00 appropriately appropriately scoping memories in the right context. This is like if you log into your ChatGPT account and you get someone else's message history. Um, if you're not

13:09 applying scope properly, you can have some really bad data leaks. So, this needs to happen on write and on read. Uh, poison memory is when a untrusted source plants a durable instruction in

13:20 your memory layer. This is kind of the worst-case scenario, like a prompt injection happened, you didn't have the appropriate guardrails, You didn't have the appropriate review, redaction,

13:29 controls on memory, and and some really nasty stuff got in there, and all your data got exfiltrated. So, uh be very sure to avoid these kind of things through the proper guard rails

13:39 and administrative actions. Uh a memory conflict is when I have two different memories. One says, "Hey, we do the task this way." And the other says, "Well, actually, we do it this

13:49 way." Because they're both about the same task, they're going to rank very highly on retrieval. So, you probably get these same conflicts in your context card. And when the agent consume that,

13:59 it's like, "Well, you know, you told me the restaurant's to the right, and it's actually to left. I don't know what to do." Um so,

14:07 detection review here is very important for conflict resolution. Uh scoping as well. Sometimes there are different ways to do the same task. Um over recall and irrelevance happen as

14:18 your memory system grows. Um if we're just shoving all of our data in there and not really thinking about what we're writing, um we can get into the situation of over recall where we have

14:28 so many different weak matches, they're crowding out the context card. This is the same problem with the very large context window where we try and smoosh in as much data as possible. It's just

14:38 confusing. Irrelevance, uh semantically close memories that rank highly on hybrid search, but it's not actually about the task. It just it rated highly. So, these

14:48 are these are problems of of management and your search algorithm tuning. Data leakage is when we get weird, unexpected data that enters memory system. This is a less serious version

14:59 of poisoned memory. Um proper enrichment and distillation of the memory records on the right path is the way to solve leakage. Uh over time also, we get silent drift.

15:09 This is where the memory system grows. Maybe it works freaking awesome at like a thousand records. But then when I have 50,000 or a million or more, it's I'm getting all these weird results, and

15:19 it's it's really working. So, it's very, very important to apply the proper countermeasures, the reduction before write, the scope checks review, life cycle, and most importantly

15:29 feedback. Feedback is what closes the loop from memory. Memory quality needs to be measurable. We need to have correct retrieval. We need to account for stale

15:38 or harmful memories, and ultimately the downstream usefulness of the system. It needs to feedback to these ranking life cycle decisions. So, if an agent retrieves a candidate memory, it'll use

15:49 it, it'll reject it. Maybe if it uses it, everything goes haywire and it crashes. Like, that feedback needs to come back. And if if the memory was harmful, the

15:59 feedback should go down. If it was useful, maybe we promote it. So, it's not just a repository. It needs to store Your memory layer needs to store whether memories were helpful,

16:09 stale, irrelevant, harmful, etc. Um and we really we want to design for measurable recall with governed memory formation built on a multi-modal storage platform. So, without memory, these

16:23 agents are more likely to restart the same tasks. They're going to recreate over and over again. They're going to read the same files. They're going to run the same queries. They're going to

16:31 waste tons of time, tons of tokens, uh be much slower, and much more much more failure prone. They're going to repeat the same mistakes. If it doesn't know about prior art, it's going to do the

16:42 same thing over again. And if that thing was wrong, it'll be wrong again and again. And with governed memory, the agents can resume where they left off. They can reuse what

16:51 came before them, and ultimately reduce repeated failures. This is This is the true goal of recording, retrieving, governing, evolving.

17:00 Um we want to improve the success rate. We want to do it faster. We want to do it cheaper. We want to do it better. Um if you are hands-on, um I recommend checking out our Python SDK for agent

17:12 memory. This code will take you to a GitHub link with a Python notebook that walks through how to do this using our SDK. The SDK is built on database primitives. Um everything that we talk

17:26 about here lives in the database. Um you We also have our developer hub. If you're an AI engineer, you're builder, you're curious like how does a modern multi-model database fit into AI

17:37 development? This is the place to go to check out our samples, um see what's possible, and maybe even inspire you to build something cool uh with a very awesome database.

17:48 Uh again, I'm Anders um Anders Swanson. If you want to chat or connect with me, I'll be over at the Oracle booth. Um and thank you so much. I hope this was useful to you guys and uh

17:59 helpful as you're you're building awesome things. Thank you so much.