Beyond RAG: Build a Relational Context Engine from Scratch — Peter Werry, Unblocked

0:12 Okay. So, I'm I'm Peter. I'm from Unblocked. Um, over here is my colleague Brandon. And today, uh, well, first of all, I just want to talk a little bit about what we do. So, uh, we build a

0:23 context engine. Our company's name is Unblocked. And uh we deliver organizational context to humans and increasingly of course agents. Um so the session was originally

0:35 a couple hours. It's it's now one hour so it's quite condensed. U which means that I'm going to step through the workshop pretty quickly. If you don't get uh you know all the steps nailed

0:45 down as we go don't worry about it because the repos up online um and you can run through it at your leisure um afterwards. Um, so the thesis here is that rag and I'm just going to say that

0:58 rag in our parlance today is going to be semantic similarity search or vector retrieval. Okay, rag is only half of retrieval and today we're going to build the other half which is turning na

1:10 natural language into real database queries. So here's the again the QR code for setup um if people haven't already done so. I'm gonna now skip um on to the next

1:22 thing and I'm just going to briefly bring up hopefully that renders. No, it won't. Of course. Give me one second to switch. Whoops.

1:37 And hopefully people can see this. Uh this is the repo here. So setup steps are pretty simple. Um there are a few sort of uh dependencies up here, prerexs that that uh you'll need to run through.

1:50 Um and the the course throughout the course of the workshop, we're going to be running through a series of pull requests uh and the network is basically gone

2:04 or at least isn't running properly. Let's try this one instead. Well, the good thing is we don't need network for this talk. Um, so hopefully you got everything before we started.

2:23 Um, because this one's obviously not going to work. But at any rate, uh, we're going to be following a series of six pull requests and they stack on top of each other. So, I'll be There we go.

2:34 I'll be referencing these as we go. Um, and I'll just go back to here so that everyone can see that. And back to the keynote. Now I'm gonna have to extend it again.

2:47 Sorry. >> There we go. All right. Hopefully people can see that. Great. So um our agents need context and context is scattered across

3:10 every tool that engineering teams live in. So you're going to find it in PRs, review comments, um issues and tickets like in linear and jura docs [clears throat] and wikis, notion

3:20 confluence, and of course in conversations and places like Slack and Teams. And all of that is semantically queryable, but it's it also has

3:30 structure. It has authors, dates, uh, states, labels, relationships, etc. And you know, the the instinct I I love to say this like even two years ago is ancient history in the realm of AI, but

3:43 the instinct even two years ago was let's just dump this all in a vector store and retrieve it and rerank it and hopefully we get the right thing. Um, but that's that's not going to work. So

3:54 I'm just going to step back and and kind of relate this to the larger discussion about context. um in in the context of a context engine that serves organizational context

4:05 um this is this is made up of more than just code as I said we're connecting everything together so the scattered context uh on the left in all your code and PRs docs etc goes in and the idea is

4:18 to get you know grounded context that relates to each other out the other side that's specifically tailored to the task at hand um and in that model uh we have to care not just about the semantic

4:29 relationships, but of course the structural relationships that connect all the data. So again, rag is about embedding the query, finding docs with similar

4:39 meaning, and then we take that stuff and we stuff it into the prompt, right? Well, um, that shines for content questions like find the code, find the PR that changed it, find the Slack

4:51 thread that debated it, but um, and if you wrap it in an agent loop, you can get pretty far with that. And you can also wire this thing up to like um, you know, tools that can recurse

5:04 files and it acts as a good jump off point. you get a good semantic head start and then from there your agent can recurse over the actual file system in GitHub um and you can get pretty far

5:15 with that. But in our experience um our this is not really the way things go in practice. engineers often ask questions like this PRs that I merged last week or my team merged last week. Um they're

5:30 they're basically, you know, hunting for something in particular and they know where the starting point is, but they don't know where the destination is, but they do know the relationships.

5:42 So, as an example, PRs I merged last week, let's think about this in a rag context. you'd you'd you know embed that sentence, you'd find the nearest vectors, you'd rerank it top K. Uh you

5:53 get PRs that mention merging about a talk last week, but uh you wouldn't get the actual relationships like the author, the merge state, um and whether it was merged last week actually. So

6:08 what you could end up with is the wrong answer where you pull up PR that's semantically related but it was merged three years ago. Um and so that doesn't actually answer the question because

6:19 there's a structural there's a temporal aspect to that question. And so retrieval has two halves here semantic and again today we're going to be talking about this structured

6:31 queries. And here's the key. We're going to take uh the model doesn't actually answer the question in this paradigm. It writes the query that accesses the database.

6:45 Okay. So, um it would be naive to say that generating the query is the only thing that we have to do. Um in order to generate the query and to do it reliably um and and to take care of maybe even

7:00 some security concerns, we need to look at all four of these points here. So that first of all the LM needs to be aware of what the schema is before uh creating the query. It needs to

7:10 understand uh some of the relationships in the data. So for example, if you pass a natural language query that says show me Peter's open PRs, um then it's not going to know what to do because it'll

7:22 synthesize a query that says Peter when really the GitHub login name might be different. So it needs to be able to map that across. Um, and of course you're going to have some uh untrusted queries

7:35 coming out the other end. Um, maybe using the function operator in the in parlance which might execute JavaScript directly on the server and we probably don't want that. Um, and then

7:46 finally, if there are errors the first time around, we want to be able to self-correct those things. So here's here's what this looks like in one single shot. What we want is

7:56 something that looks like this. Um it it maps uh my colleagueu's name is Rashine. His user login name is is Rashab. Um and then we want to be able to map things like the state and get maybe uh

8:10 operators like limit and sort. So uh this is what we're going to build. We're going to start with a full UX, a stubbed engine, um, that is not capable to start with, and we're going to

8:26 gradually build this up. And then at step three, the thing will come alive. And so we'll actually be able to issue queries to it and see what it does. And uh, if you hadn't seen that before,

8:38 now is another good time to scan that and get it in. I'm here to help you. >> Yeah. >> Yes. Any repo is fine. Doesn't matter

9:05 which one. Um and if if you want to take a shortcut, you can just point an agent at the thing and let it set everything up for you. That works.

9:16 Okay, let's move. So, this is what we're going to do to start with. Um, by now you you probably should have run this ingestion thing. Don't worry if you haven't. Um, again,

9:27 like you can revisit this at the end. I'm just going to quickly show what the the kind of barebones interface looks like for this thing. So, now this thing is running. Now, if I

9:38 type anything in here, this is obviously not going to work. Um, and what you should see is something like that. Um, there's also a tab for chat. This is a fully implemented agent uh that can uh

9:49 do work for you. So, we'll come back to that in a sec. And so, this this is the contract in the code if you wanted to read. Um, we're going to don't worry about this. We're

10:03 going to skip through this. Um, but basically like that's the entry point and we're going to be building around that. So the first thing we want to talk about is schema discovery. Um during

10:15 ingestion, if you if you look at the way that the ingestion script worked, it it actually had to build out a schema in order to download the content and store it. Um but you can imagine that you may

10:26 not know the schema. So the ingestion may be happening in a service elsewhere and you want to build like a fully dynamic schema discovery engine. Um schemalis. So there's no schema to

10:37 read. um unlike you know PG or something where we could read the table definitions. Um so what we'll do is we'll sample the documents to infer the schema

10:51 and that's the whole trick. So the we're just going to reconstruct it. We're going to take um all the nodes and merge them down. So this is going to sample 300 documents, collapse it into a single

11:02 tree. uh nulls are skipped because missing fields won't be able to constrain the type. And then uh we're going to do something kind of interesting here with enum types.

11:17 And so now uh with with enum types, we can do an interesting sort of procedural trick. This is just one thing. I would suggest using procedural tricks to to do this

11:29 type of thing because you could pass it through an LLM and do extraction and that works u but it's fairly non-deterministic. So you always need some kind of structural

11:39 um harness around it that is running procedural code to either validate the things that you pull out or to actually do the extraction itself. And in this sample project um it's the procedural

11:51 code that's doing the extraction. And let's go and show that. So we'll do and then run it again. And now what we get is this thing here.

12:12 So the schema ran uh the the discovery process ran and it created a schema for us that outlines what the pull request schema looks like in GitHub. It even created some indexes for us. Now, this

12:25 is like marked down on purpose because we're going to take this schema and pass it to the LLM when we synthesize the actual uh query. Now, the second step is the thing that I

12:41 referred to a few slides ago about identity resolution. And this is because like my lo GitHub login um is similar but it's not quite the same. And so if I were to ask a question like um hey find

12:56 all the PRs by Peter um it if without doing some sort of identity mapping it would just say you know it would just match across Peter and it's not going to find anything. So we have to be able to

13:08 solve that. [snorts] And again, um, we're going to do this. At first blush, this seems like we're coupling to the schema a little bit here. Um, but you know, it is absolutely

13:22 possible to determine this dynamically. And you can have uh your LLM essentially synthesize the mappings for you. So, but in in our case, we're just going to do these simple little tricks. Um

13:36 very simple sort of fuzzy matching. Um Levenstein distance is fine. You can just like match across the names and uh there you go. Oh, actually I want to show you one

13:51 thing before we get on to that one. So let's do the same thing again. We can pop up >> Oh yeah, I'm sorry. Absolutely. Tell me when it's big. Good. Thanks.

14:08 Good call out. Okay. So, let's run our thing again. And now, um Oh, actually, I have to I have to wait till the next step. I apologize. That's

14:22 not going to be wired up just yet. So, we'll go on to the thing that will really give us the payoff here, which is step three. So this is the actual query plan step

14:39 and it's it's it's arguably sort of the simplest part. It's the LM part. Um and what we do here is we pass the schema give it a tool and force it to call that tool for name extraction and

14:52 synthesis. Um and then it will execute the plan. There's one other thing which is the temporal aspect. uh LLMs are actually pretty good about

15:04 generating temporal queries if you pass them the current date and time. So that's another important thing to do in the in the prompt. And so here this is the name extraction

15:20 part. We're going to force that tool call to happen in this case. And at this point I am going to show you the the payoff for this. So there we go. So now I can do things like

15:49 this. open PRS by Peter and bam. So what we'll see here is that it uh generated this um this match clause and uh you can't see it from here

16:08 but it it mapped Peter to my GitHub user login puery and generated the query and that's the outcome. So that's cool, but um we need to get on

16:25 to the next step because even though we are able now to query the data, we have some safety concerns. Um it could be the case that it produces untrusted output

16:39 and so you know it could use the wear or function clauses and literally run JavaScript on the server which means that this is now a prompt injection point. Um,

16:50 you could also have hidden fields in your schema like ID or metadata fields that leak internals. So, we need to sanitize um, and we don't deny list these things.

17:01 We allow list them. So, it's deny by default. You want to create basically an allow list of allowed operators and uh, you know, potentially an allow list of allowed fields. In our case, this is

17:13 dynamic schema discovery, so we're not going to do that. Um, but we could have some fields that we know about like metadata fields. And this is a a useful thing for like tenant uh tenant

17:25 isolation. So if you have a a uh collection that houses multiple tenants, I don't recommend that. you should probably separate your collections by tenant, but if you did, then you could

17:36 have a metadata field that contains the tenant ID and you would want to hide that no that um information from the LLM, but also validate the output. So if it outputs

17:50 that metadata field, you need to wipe it out and then you need to inject your own metadata field underneath. And again, like technically this is it's not actually that difficult to do. You

18:08 just have to write the code. It's all procedural. Um I I wouldn't recommend using LLMs for this part. And so again, allow list, don't deny or deny everything, I should say.

18:33 Now the last part is um we're going to this kind of dovetales into the next step. At the end of validation we're obviously going to return an error and it's going to get dumped out. And right

18:44 now that error is just it's just there. We're not going to do anything with it just yet. But I will show you now what this looks like. Um,

19:03 oops. Stay on that one and run our service. So now I'm going to write run another query that checks for authentication in the

19:16 title and we've disallowed regular expressions. So now what we can see is that it rejects the query. And finally, this is the thing that kind of ties it all together is the retry

19:40 loop. The plan can fail in a couple of different ways, right? Um, it can fail validation. So maybe the uh the LM actually outputs things that are illegal

19:56 according to our validator. Or it could actually fail during execution. Um, so for example, let's imagine that it it mixed an inclusion exclusion project. Um, then that's obviously going to fail.

20:13 And then but we can self-correct. We can take that error and pass it back into the LLM and say hey that was wrong. Um maybe you can do better up to some number of max attempts of

20:25 course. So I'm going to show now what this looks like back over to our project

20:39 so we can punch up a level. Whoops. Sorry. You have to forgive me. I actually have to read this off of the screen out here. Okay.

20:57 So now if we run that same query. Perfect. So, what we can see is that it took several attempts to get it right. Um, by the third attempt though, there's a

21:17 bit of a problem here because we said no regular expressions, no FTS. Um, so now it said we're going to just use match, but now it's an exact match. So it tries to find

21:30 authentication fails which means there's one more thing that we can do which is this. We'll add full text search.

21:49 It's pretty basic. Um we need to create an index obviously and we'll combine it with author and state filters uh in the same match. And so now

22:02 we should be able to answer a question like this. And let's see what happens. And see if it is actually going to do what I said it's going to do.

22:29 Bam. There we go. And so now the the final payoff here is if we go into the chat tab, we can ask some sophistic more sophisticated questions that um you know an agent can

22:46 actually reason about behind the scenes. So you can say things like um did we work on anything authentication related last week.

23:10 So here we can see that it generated the query. Now it's pulling the data and kind of reasoning about it. There we go.

23:22 That's it. So, just a quick summary of what we built. We're basically like reaching the end. There's going to leave lots of time for

23:42 questions. So just to recap here, um there are questions that that rag can't touch. All those structural questions around things like state or temporality. Um you don't get the right data out when

23:57 you ask those types of questions with rag, traditional rag. So the same engine behind the query playground and the chat agent, uh they're they're the same. So if you guys

24:06 want to play with that after, go ahead. Um it's it's the same backing thing. The only difference is of course that uh the chat agent actually does reasoning whereas the query interface is just for

24:18 queries. So just to to recount on this here are the reusable ideas here. Schema aware prompting dynamic schema discovery identity resolution

24:32 validation and error retries. The error loop is actually one of the most important things. And so we built something that's capable of filtering,

24:50 providing aggregations, it's time scoped, has full text search. Um, it's validated, it self-corrects, and it powers a full-blown chat agent. So you could take this into your organization

25:02 and just deploy. I I would recommend maybe working a little bit more on the validation harnesses before deploying this internally, but um yeah, it'll it'll answer all your questions about

25:12 your PRs. And that's it. So, we've left lots of time for questions now. [laughter] Did Did everyone uh was everyone able to actually set it up? Just curious. No.

25:24 Okay. If if you want uh you can come and fetch Brandon and I afterwards, we'll just kind of linger outside here. Oh, at the and at the P16 booth as well. Yeah, but that that starts later today

25:37 at around four o'clock and we'll be there all week. >> Cool. Any questions? >> Yes, >> I have a question. I think you did a

25:46 great job breaking down like basics. >> Yeah, >> a lot of my mind goes like okay so at scale how does for example you you generate the schema and you put it into

25:56 the system prompt how does that work at scale? It's like also depending on the size of your company, how are you fitting all this data? How do you choose what data and like how long does it take

26:05 to process all this? What's the cost? That's where my mind is. >> Yeah, it's a good question. So, you know, um I'm just going to repeat the question so everyone can can hear it.

26:13 The demo showed obviously a fairly contrived example where you're just focusing on pull requests. Um and so the question is how do you scale this? Like how do you do this across multiple

26:23 different collections, multiple different tables, etc. Um, so you you can add uh another LM step on top where uh this the schema collections have tiny little summaries and descriptions and

26:36 then uh depending on what the query is, you can load the right schema for that query. LMS are surprisingly good at this. They weren't a year ago, but now they're very good at it. Um, so that's

26:48 kind of the way to shave the context thing down. And that's to your question about scaling things out. Um, was there another aspect of scaling that you were interested in?

26:57 >> Getting all the data in and I guess like part of that is like cost. >> Yeah, it's livveness and stuff too. >> How long does it take to do like what is what data should I pick to put into

27:07 this? >> Yeah, it's it's a good question. So unblocked um at unblocked like we're continuously ingesting content. Uh we get web hook callbacks for all the

27:17 events for example in in PRs. And so every time there's a change, we get that web hbook event, we ingest the content, we rem um and store it in the database. Um that that's just really just like a

27:29 distributed systems question about how you scale that out from an ingestion level. Um you know, storage, same thing. So yeah, >> so maybe this will make it more

27:39 concrete. So what if you would say like these are the top three data sources like someone seems like GitHub maybe like something >> um so Slack for sure a lot of the like

27:52 why questions and and decision- making happens in Slack um people uh in the context of this sort of like structural query um people usually think about conversations that they've had in the

28:06 past um and they'll be able to query unblocked and say Oh yeah, you know, roughly three weeks ago, I was having a conversation with so- and so. And so that that's a perfect thing for

28:16 structural query. Um yeah, like I would say that um Slack is the most important for why questions. Um a lot of the why decision-m is also embedded in PRs. PRs uh have all of that like organizational

28:31 history in the comments and that's just basically black hole, right? So uh you can basically run over that data before you serve it and and kind of mine it for interesting insights. So for example at

28:45 unblocked we we create these things called best practice memories which are distilled from PR comments in the past. Does that make sense? >> Yeah. Yep. Any else? Yeah.

28:57 >> Question. directly into how this building. [snorts]

29:26 >> Yeah, I mean you you can definitely do that. So if you have a structured if you have a structured query table, you could express that through MCP. You can even do that dynamically by updating the tool

29:36 every time the schema changes behind the scenes. Um this this type of thing is showing like what happens on the other end of that MCP call. So you you could you know have an MCP interface that just

29:48 takes natural language queries for example and then translate that all on the back end or you could provide a direct interface to the database. It depends on like what the customer

29:57 interface looks like, but yeah. Yeah. >> Yeah. >> Yeah. >> Yeah. Yep.

30:22 Second part is better. Thoughts around >> uh how to mean how to combine it or let me just make sure I understand your last question.

30:50 >> Yeah. feel like this is like >> No, not at all. No, no, they're absolutely complimentary. Yeah. Yeah. I mean, you can have a full text search that does this kind of um you know, like

31:03 that's full text search, right? So, it'll identify keywords. You still need some kind of mechanism to to do like the actual semantic similarity search because there you all have ideas and

31:15 sentences that are encoded that aren't not ne necessarily textual matches. So you want you want that like conceptual matching, right? So as I said um at the beginning like there's similarity search

31:28 is actually really powerful for jump off points. So you can for example start with a vector search and then from there give it access to tools like this because then it'll actually have like

31:40 some interesting starting data and help it form queries to go and and fetch the data. So yeah that that's that's something that we do and so sorry the first part of your question was

32:03 >> so you take um you take you know in in anyways like these are all documents and they're they kind of like form into JSON so you can merge those things down and look for certain tokens

32:15 and then you you know, extract the tokens and then that's how you distill the types. So you you can look through the source code and you'll see that at least like this iteration of that

32:25 algorithm is pretty hacky, right? It'll do things like look to make sure that the character count is between two and eight for uh for type information and then for enumerations it's it's looking

32:36 for particular patterns as well. >> Yeah. >> Yeah. Yes. >> Yeah.

33:05 >> Now, that that's a very overloaded term, so I'll just make sure I understand exactly what you're talking about. Do you mean hybrid search in terms of um a provider that does semantic similarity

33:16 and then also does a lexical search and then does rank fusion? Yeah. Oh, yeah. Yeah. No. So, so we we do that. Yes. The answer is yes. Yeah. >> Yeah.

33:44 >> Yeah. >> Uh just empiric empirical trial and error. And so I mean there's there's no easy solution to this. Um be behind the scenes like at unblocks we're we're

34:05 using a lot of static schemas because we want to get the correctness absolutely nailed down. This this was like a just a technique that I wanted to show because um you know for exploration of data or

34:16 experimental um uh for experiments or either to wire up like arbitrary databases this is a good technique. So we we got this through some trial and iteration with

34:28 customers and you know we we'd ingest a whole bunch of contents from the document store and we'd get responses back from customers that said oh you missed the

34:38 thing that I was searching for and we'd dig into it and we'd figure out why and we kind of got to the sample size that made the most sense. Yeah,

34:58 >> it's absolutely true. Yeah, definitely. You're always actually gonna you're going to miss something. There's always going to be some document that has a little edgy thing. Yeah.

35:09 Anyone else? I thought I saw a question over there. Oh, yep. Sorry, can you say that first part again?

35:22 >> There's mention that you understand the schema. >> Uh yeah, what we do is we force a tool to be called for not for schema discovery because that happens at the

35:36 start. It's for name extraction. we so we have a validator that watches what the agent's doing. So like if it doesn't call the the first part which is name extraction. Um so there's there's

35:57 actually two techniques for this. One is you create a tool that does the the query and as an argument to the tool you accept a username. So you can fail right there if the username is not correct or

36:10 malformed. Um, if you want to do name extraction uh via tool call, then you have to watch what the agent does. And then if it doesn't call it, you just basically insert a user message that

36:23 says you forgot to call the extraction tool. Do so now. Yeah. >> Yeah. How do you handle the changes?

36:52 Yeah. >> Yeah. So remember like for this particular example the schema is is completely unknown and so as documents get added um that schema could update.

37:14 So you your your choice is either you know on a sampling interval you you rerun the schema generation um in our case at unblock the schema is is static and it's tied to our codebase

37:26 and we control the schema actually. So if the schema of if GitHub decides to change the schema um that of course rarely happens because that would have serious implications for their customers

37:37 but if they did then we would just adopt whatever changes are made statically and then obviously deploy that through CI. Yeah. Anyone else?

37:55 >> Awesome. I think we can uh reclaim some time in that case. Thanks. [applause] >> [music]