CrabRAG: Why Automated Assistants Need Graph Memory, Not More Tokens — Stephen Chin, Neo4j

[music] My name's Steven Chin. I run the developer relations team here at Neo Forj. And I'm excited to talk to you about something we've all come to love, our our crustaceian friends. So, we have um um openclaw mascot. We have a bunch of other crustaceians. And we're going to we're going to focus on one member of the crustaceian family. I I I love crab. So, our little boy, Crab D. And I think in the in the journey to to figure out how to apply agents, how to do things which are more autonomous, we're all looking for ways where we can get better results, more accurate answers, and to actually capture all of this. But the tools kind of work against us. So, um, here's our our friend Crab D. He's he's a personal assistant, very happy, very eager. He wants to to help us out with our lives, maybe to help us to code, to help us to, you know, manage our email, to do different things. But he's got a problem. And our poor boy Krabby D has a very bad memory. He wakes up every day and his memory file flips and now it's a new day and he forgets everything from yesterday. Has this happened to you where you you wake up and you're using Open Claw and suddenly it's on a new set of memory files and remembers nothing that you actually did the previous day. He's got a lot of tools at his disposal. I mean, we love giving our agents tools, but sometimes he doesn't pick the right tool for the job. I don't think either of these are going to help him drink his his bowl of soup. So, that's not the tool which he was looking to to reach for. and a little bit forgetful at times. So, you know, I think I don't remember everybody I meet, but I'm pretty good at faces. Like, if I've if I've met you before, I recognize faces. It's like, pleased to meet you. Um, Crab D is not as good at that. So, very forgetful. It's like you're retaching it every day to do the same sort of tasks. And we want agents which are more helpful, which are able to do more for us. So, let's dig into how CrabD actually works. So it's basically a a memory loop, right? So we're we're prompting, we're thinking about the response, maybe calling tools, observing what happens. But the hard part is the memory. The hard part is what you put in context, what you're recalling from. And the way you have memory structured in most tools, this is an example of um how open cloth structures things is you have a sol for your agents memory. You have maybe um memory files, you have different tool files, you have daily memory files. Now, if you look at this, there's one thing which is in common with all of these. They're just markdown files. So, markdown files are great. That's easy for us to read. Like, we can we can look through it. We can quickly figure out what's not needed and compact them. Um they're intentionally small for agents because you have a limited context window and also you need to keep the right things at the top of the context. But if your whole memory is a bunch of markdown files, you're wasting a lot of tokens. So, um, my my average agents are are loading up at least 100k in tokens for each round. Um, they're doing a a lot of skills. They're adding a lot of things into the context constantly. It's very repetitive because they they basically load up everything in the hopes that something will be useful in the context. At small scale that works where you get the results you want with a high quality model. It doesn't work at large scale and I'm going to show a demo of large scale where we take open claw and we let it run loose on my home lab. So um high demo risk but a lot of fun and um a classic digital twin scenario. So I think we'll have we'll have a lot of fun here. Um anybody use Hermes agent at all? >> Okay. I'm I'm a big fan of Hermes agent. Um I think it's got a much better memory system. It kind of at the end of each task it goes and it reflects and it adds back in new skills or new things which it needs. So um it's a really powerful system and um you know again we're relying a lot on markdown files. Skills are just basically markdown files. But we can teach the agent to do a lot of things with skills and it can it can get the right skill if it gets loaded up and then good things happen. But sometimes we don't get the right skill loaded up. So our our poor boy crabd here is not going to get that clam. He just doesn't have the open clamshell skill. Lots of shrimp, no clams. Maybe you picked the wrong skill for the job and suddenly you're you're jet skiing on the on the beach, right? This this is not this is not going to get him very far. And sometimes you you might get that clam open, but then you don't have the skill to eat them. So skills, you need to have the right skills, the right chain of skills. Um actually we have an awesome project by one of the neo forj folks which is just bananas as an arvix paper which is a graph for skills. So that's an exciting way of like like figuring out what the right skills are but maybe we can do better. So um goose is a project that's part of the um agentic AI foundation is a new foundation which MCP is part of. Um anthropic is backing this. We're we're also a member of this. So, it's a it's a great automation tool for a lot of enterprise workflows. You can also use it kind of like a personal assistant. It relies heavily on MCP as the layer, over 70 MCP extensions. And what it does is it treats memory just like another MCP server. So, this is great, right? It's it's pluggable. You can call different commands on it to retrieve memories, remember memories, um, forget memories. Memories are just plain files on disks. So now you can manipulate them. So same great idea, same fundamental problem. We're storing the memory. We're storing the memory of agents as markdown files on disks. And again, you end up with what if what if you pick the wrong tool for the job, the wrong paddle? Now, in this case, if you pick the wrong paddle, you're a genius because you've invented the most the fastest rising sport in the US, which is pickle ball. Um, actually, the origin of pickle ball was was um a family wanted to create a new game and they just took what they had around the house, a bad men court, and um made up the rules along the way. So, creation can be good when you have the wrong tools. Maybe you remember everything, but it's too much. is too much weight because you can't actually solve the problem. So our poor friend Goose here is encumbered by too many notes, too many memories, or most dangerously now you have MCP tools. You're one step away from calling the forget command and just wiping out your own memory. Okay, so we want to be able to do better than this. So vector databases, right? So we can store everything. We can create embeddings for it. Now we actually have a d a database. We can store it in a vector database. So this is great. I mean you have to pick the right vector database. Um and then now you can do similarity searches. So you can pull back information which is which is relevant. So we're doing much better. We have a larger repository of knowledge. We can pull back related information. Um open claw comes with pg vector out of the box. given embedding, you can just start using this. Um, lance DB is a great option. I'm going to use both of those in my demo. But the challenge here is similar what what vectors give you, which is similarity in vector space is not the same as actual relationships. And so you get hallucinations. You get a lot of problems when you're relying solely on vector lookup as the answer. And it compounds with more complex scenarios when you're doing things like like I'm going to show you an example of a digital twin. When you're doing things which are very complex, they they just don't scale and you make silly mistakes like this obviously is not what poor Crab D wanted to munch into and it's a very expensive lunch for him. Also, it's sometimes impossible to get to the answer even though you have all the facts because those large multihop reasoning chains don't work on similarity searches. They're also very expensive on traditional relational databases. And often things look similar, but they're not exactly the same. And this is one of the problems with the responses you get from a vector database is you suffer from getting facts which are related in some way and they're not your shell and you don't you don't want to take the wrong shell out of the locker room. That's that's very unfortunate. So enter graphs. Graphs are a great way of finding the relationships, finding those identities, bu mapping out the paths, getting that full chain and they're built for this sort of connected data. So now that you have first class nodes which are the the circles, edges, those are the relationships between different objects and then you can put properties on top of graphs to store information. You can also store embeddings in your graph and that gives you a way to both use vectors and graphs together. Um architecturally the demo I'm going to show you is um both a vector search and a graph search. So it uses the vector search to get the seed nodes where it starts the traversal and then it uses a graph search pulling the the nearest neighbors and then ranking those by how related they are. And this gives you this complex multihop queries to solve more difficult more domain specific problems and to figure out where that where that reef is that we want to get to with all the the tasty um the tasty junk food across the ocean. And graphs are they're accurate so they give you very precise information. Explainable because you can look at the graph which got returned and auditable because now you can actually say these are the this is the context. This is the part of the graph which resulted in that answer. So it's very powerful and it gives you more tools as a developer where if you're not getting the right answer, you know where it's coming from. You can actually see and introspect the graph and you can change how you're doing extraction. You can reduce duplicate nodes in the graph and then you can get to and converge very quickly on a great answer. If you're not a graph expert, guess what Claude is. Claude can write cipher better than I can. Claude can extract build entity extractors and it can do pretty much everything you need to do to get started with graphs today as long as you know the the basic kind of model for what you want to accomplish. That's what I'm going to cover in the demo. So, we're going to do have Claud action into the graph as he works. We're going to follow up by traversing, not rereading it. And then in a fresh session, we will get the results we want to get out. Now, what I did for this um high stakes demo is I over the past week or two, I took my home lab as the demo environment, did a full digital twin as a graph, and I have two separate environments built off the same original markdown files. One is a vector database store, that's our our A test, and the second is a graph store, that's our B test. And the graph store is built on top of um Cognite. So I'm using Cogni which is a startup. Um they do amazing stuff in the memory space. They have a Neo Forj backend. This is my the structure. So we have a bunch of Proxmox servers in my my home lab. It's really a couple computers around my desk and I built a separate VLAN for the demo. So it's segmented off my real network. So it was trained on real network for my network. But now it's it's cut off. It can only answer from memory. it can't actually look up the hosts and get dynamic information. So, let's see how it does in a live demo. Okay. So, all right. Here we have our our crab rag cockpit. Um, and I have five different questions queued up with schematics. You can see this is the same home lab schematic that you saw earlier in the slides and um let's let's start with this one. So WR exposed end of life soft WAN exposed end of life software. So we're going to basically we're going to try to find out if there's anything on my network which is exposed to the network the the internet the WAN that's running out ofdate software which put my home lab at risk right so if if somebody can attack the home lab and um you can see here that [clears throat] there there is some servers um Tsterland which is my daughter's Minecraft server it's running oh my god dbna Jesse and let's see how the the two agents did in looking this up Okay, so we got the vector response back. Couldn't find specific details, excluded by policy for more precise information, yada yada yada, source it separately. Okay, that's that's not very helpful. Now, on the graph side, it's done a bunch of cipher queries. Here are the cipher queries. It's fired off. Um, this is the graph traversal. And the the color coding on the graph traversal is these blue guys. These are the seed nodes. So this came from a a vector lookup and a ranking. But it didn't stop there. It does the one hop traversals. Those are all the gray nodes. Some of the nodes get highlighted in green and those are the ones which which won and got into context. And you can see the answer here. So guest name tinsterland exactly as expected. Um OS version out of date and it's flagging. So so it gives us very precise actionable information. And so that's the difference between same same exact data. One is a vector store, one is a graph store. And you can see the difference where the the vector store is having a lot of trouble pulling the information out, the relevant information out. Okay, let's try another one just for fun. Um, let's see. Expose 0.0.0.0 management ports. That's that's bad. So, um, basically, you don't want your management ports on the network exposed to the, you know, the world. And there's a bunch of these. So, I have a new matrix server I set up, and also hroxy, which are exposed to the internet. That's bad. The rest of these, like my Cognney demo, my openclaw instance, those are inside the LAN. You need to get into the LAN to access them. That's that's what you want. Okay. And let's see how the two agents did in identifying this. So the memory search returns some information and it's telling me check services configuration expect PFSense rule. So it told me to go do the job for it. Um okay and then the graph memory side found an open port exposed to WAN haroxy and openVPN which are the are the two we expected. Now this you can see the shape of this graph is entirely different from the previous one. And what it did is it it actually found the node for for my router the PFSense router and it was able to follow that directly to all of the results which related to it and then give us like a very precise answer. All right. So, um, so now, so now we've seen our little boy Crab D with his certified Neo Forj developer t-shirt is able to do a lot more. Right now he's able to follow that full chain, crack, eat, do the next thing. So, he's getting his he's getting his clams. He's helping me fix all the security holes in my network. Um, oh, by the way, I I patched all those security holes after the demo. Um, so this was good for me too. It found a bunch of security holes in my home lab and then I I went and patched them later. And um, now we have an agent which actually can do interesting things. Now, if you can imagine like I have a three or four node home lab at home. If if you have a big enterprise which has a huge data center, if you're doing things in financial services where you have like a huge set of companies and customer records you're trying to do, if you're doing anything at at large scale where it doesn't fit into the 1 million context window of the modern models, you really need a better memory system than just throwing things in markdown files. our little boy Krabby knows his whole crew, all the Crustaceian friends and he's read the book. Um, so we just finished my my co-authors and I Michael Hunger and Osus Barasa finished Graph Ragg the definitive guide. The full book is out on on early release. It'll be published um in a couple months once they finish the editorial process. But super excited about this. It's got information not only on graph rag but also on building memory on on different industry vertical use cases on agents. So it's kind of the the whole umbrella if you're building on top of graph solutions how you need to build applications the technologies you need end to end. And then finally, a great free resource which everybody in this room can take advantage of is Neo Forj um's graph academy. So it's free online training um dev.ne.com-rag or the QR code below there. And um we have courses on doing agent memory, doing context graphs and everything you need to get know to get started and to do some of the amazing stuff which I showed you on stage today. So thank you so much for coming to the the kickoff talk for the graph track. [applause] You're in the right place for all of the content from graph experts. Andreas Colliger, my colleague and I crafted a great set of speakers from industry experts, people who really know about graph technology. So hang out here, find out more and then you can see me in the Neo Forj booth. Thank you. >> [music]