Graphs vs Vectors: The Real Shift Happening in RAG

Ask your AI a question that lives inside one paragraph and retrieval works exactly like the demo promised. Ask it what the recurring themes are across two years of complaints and it falls apart. Not because the model is weak, because no single chunk of text contains that answer. Similarity search finds passages that sound like your question. It cannot join them together. That gap has a name now and last month it took over the timeline. Graph engineering. The idea is small. Store what you know as entities and typed relationships, not flat text. Then let the agent walk the connections instead of guessing which paragraph looked closest. Microsoft priced this. Stanford measured it. Anthropic wrote the whole pipeline down. And the same lab that published that pipeline had already deleted its own vector database. Both of those are real. Both happened this year. And together they explain what is changing. So here is the grounded version with the numbers including the ones that argue against it. The term is about a month old. A researcher named Josh Simmons wrote that we were entering the graph engineering phase in early July. Two weeks later, Peter Steinberger asked the timeline whether we had shifted from loops to graphs yet. And inside 2 days, there were three competing definitions. The argument underneath is older. Microsoft Research wrote the serious version in April 2024. In a paper called from local to global. Their line about ordinary retrieval was blunt. It struggles to connect the dots. So they built a system that pulls an entity graph out of your documents, then summarizes every cluster of it upfront against plain vector retrieval on about a million tokens of podcast transcripts. The graph version won on comprehensiveness between 72 and 83% of the time and on diversity between 75 and 82. On a larger news corpus, the pattern held and the top level community summaries needed over 97% fewer tokens than summarizing the raw text. better answers at query time for a fraction of the tokens. Which raises the obvious question, if graphs were winning like that in 2024, why is anyone still arguing now? Because building the graph was brutal. Every document goes through a language model to pull out entities and relationships than more calls summarize every community it finds. The figure that followed graph rag around $33,000 to index one 5 GB set of legal cases in early 2024. $33,000 before one person asks one question. That is roughly a full-time engineer for a quarter spent entirely on pre-processing. For most teams, that ended the conversation in the budget meeting long before anyone reached the architecture review. Therefore, Microsoft went after the price instead of the pitch. In November 2024, three of the same researchers shipped lazy graph rag, and the trick is in the name. It builds nothing upfront. Indexing costs in their own words are identical to vector rag. 0.1% of full graph rag. The summarizing moves to query time where it only ever touches the slice of the graph your question actually needs. On their benchmark of 5 a half thousand news articles that worked out to more than 700 times lower query cost than the global search it replaced. 2 years on that objection is gone. A practitioner review published this week puts graph indexing on a working corpus at around $50 with a current model. different data set. So, it is not a like for-like against the 33,000, but the order of magnitude is the whole point. And the project is not a museum piece. Microsoft's repository shipped version 3.11 on the 18th of July, sits at 35,000 stars, and had commits landing yesterday. So, the cost objection died, and the argument moved somewhere more interesting. Once it was cheap, people could finally test whether it was better. And the paper that tested it most carefully opens with a sentence you will not find in the marketing. Recent studies they write report that graph rag frequently underperforms vanilla rag on many real world tasks. That is graph rag bench accepted at ICLR 2026 and it splits questions by what they demand of you. On simple fact retrieval text chunk score 60.9 and graph score 60.1. That is a tie. You paid for structure and got nothing back. But on complex reasoning, graphs score 53.4 against 42.9, 10 points. And on summarizing context across a whole corpus, 64.4 against 51.3, 13 points. The harder the question, the more the structure earns. A separate evaluation out of Michigan State and Meta updated again in March, pushes the other way. On single hop questions, plane retrieval scores 64.8 and the best graph method scores 63. On the easy questions, a graph makes your system slightly worse. Flip to multihop and it inverts. 70.3 against 67 on multihop rag. And on retrieval recall, the graph guided version pulls 87.8% where the naive one pulls 73.4, 14 points from the same document. On the sets built specifically to need two or three facts from different places, the gaps get large. 31 points on music, 28 on two wiki multihop. Those are the questions similarity search was not shaped to answer. The efficiency case is not theoretical either. Hipparag out of Ohio State in 2024 wires a knowledge graph to a personalized page rank walk. It reports up to 20% better multihop accuracy running 10 to 30 times cheaper and 6 to 13 times faster than iterative retrieval. One caution before you trust any table in this field. The review that gathered these numbers audited the language models being used as judges and found position bias worth more than 30 points. One method read as winning 67% of its comparisons after correction 39. So a graph is not an upgrade you switch on. It is a targeted instrument and the entire skill is knowing which questions deserve one. Which is why the interesting work this year is not the argument on the timeline. It is what three labs actually shipped. Stanford went after the part the framework skipped. whether the graph you extracted is any good. A team from the stair lab with Toronto and Far AI built an extractor called KGEN and a benchmark called mine to score how much of an article survives being turned into a graph. On that test, KGEN recovers 66% of the facts. Microsoft's graph rag extractor recovers 47.8 and classical open information extraction manages 29.8. About 18 points separate the two graph systems working from identical text. The difference is entity resolution. If one person appears four times under four spellings, you get four lonely nodes and no path between them. So the traversal you paid for does not happen. KGEN clusters them into one. But read their second test and they are careful. On downstream question answering the two came out comparable. Anthropic published the third piece in March as a working knowledge graph pipeline in their own cookbook. Their explanation of why is the clearest sentence anybody has written about this. No single document contains the answer, they write. Rag retrieval won't chain the facts for you. The pipeline is four stages and deliberately unglamorous. Haiku does the high volume extraction, one call per document. Pulling typed entities and subject predicate object triples. Sonnet handles resolution and summarizing because deciding that two spellings are one person needs judgment, not string matching. Six Wikipedia articles produced 36 raw entities and 34 relations resolved down to 22 nodes in a single connected component and they published a failure alongside it. Precision of 1.0 recall of 0.55. Everything it pulled out was correct. It missed nearly half. This is spreading past retrieval into memory. Gravity, the temporal knowledge graph engine behind Zep, sits at 29,000 stars with a release out last week and its paper reports up to 18 and a half% better accuracy on long conversation evaluation at 90% lower latency. Now, the part that complicates all of it. In February, the engineer who built Claude code posted that early versions used Rag and a local vector database and that they dropped it because agentic search generally works better. Simpler, he said, without the same problems around security, privacy, staleness, and reliability. So, one lab tore out its vector index and published a knowledge graph pipeline about 7 weeks apart. Several other coding tools have reportedly followed the first move that reads like a company arguing with itself right up until you look at what each system is actually searching. Code already has a graph, files, imports, call sites, a commit history. The structure sits on disk and a well- aimed GP walks it for free. Pros has no such structure. So if you want to traverse it, somebody builds the edges first. That is the rule under both decisions. Index what you cannot traverse, which is what the word replacing actually covers here. Retrieval is not going anywhere. What is being replaced is the default. The assumption that the first move with any pile of documents is to chop it up, embed it, and hope the nearest neighbor is holding your answer. And structure has a cost the pitch decks leave out. Entity resolution is not error-free, and the errors compound. At 85% accuracy per hop, a five hop chain is right about 44% of the time. Long traversals need verification, not faith, which is why the deployment numbers stay humbling. Industry surveys put fewer than 15% of enterprise knowledge graph projects past the pilot stage and the reason people give for abandoning them is usually not the bill. It is a shortage of people who can model one. When it does land, it lands hard. LinkedIn rebuilt customer support retrieval on a graph of past issues and reported retrieval accuracy up 77.6%. Median resolution time dropped from 7 hours to 5 and the slowest tenth of cases from 87 hours to 47. that was published in 2024. So practically for whatever you are building this week. If your questions are lookups, your embeddings are fine and a graph is a tax. If they are about how things connect, who touched what, what changed when, which findings contradict, then chunks will not get you there. And that structure now cost about a thousandth of what it did. The teams winning with this are not the ones graphing everything. They are the ones who can tell which questions deserve a graph. That is the actual skill the phrase graph engineering is pointing at underneath all the noise on the timeline.