Knowledge Graphs and RAG: When Structure Beats Pure Vector Search

If you build retrieval systems, the choice of retrieval mechanism determines which questions the system can answer, and defaulting to vector search everywhere means the relationship-based questions fail without the failure being obvious. Vector search is the right tool for questions answered by semantically similar content, and it is the wrong tool for questions answered by traversing relationships between entities, which is a genuinely different kind of retrieval. Matching the retrieval mechanism to the question, using vector search where similarity suffices and knowledge graphs where structure is needed, is what produces a retrieval system that handles the full range of questions rather than only the ones vector search happens to fit.

Where vector search fails

Vector search works by embedding text into vectors and retrieving the chunks whose vectors are most similar to the query’s, which finds semantically similar content and is excellent for questions answered by such content: find the passage that discusses this topic, retrieve the text relevant to this question. Its limitation appears on questions whose answers depend on relationships between entities rather than on semantic similarity, because the relationship is not something semantic similarity captures. A question that requires knowing how entities are connected, following a chain of relationships, or reasoning over the structure connecting pieces of information is a question vector search struggles with, because the relevant information is in the relationships, and vector search retrieves by similarity, not by relationship.

This failure is quiet, which makes it dangerous. Vector search on a relationship-based question returns the semantically similar chunks it can find, which may be individually relevant and collectively insufficient to answer a question that required following the structure, and the system generates an answer from that insufficient context, producing a confident, incomplete result. This is the same silent insufficiency that single-shot retrieval produces, and it is especially acute for relationship-based questions, because vector search does not just retrieve incompletely, it retrieves along the wrong dimension entirely, since the answer was in the structure and vector search only sees similarity. A retrieval system that uses vector search for relationship-based questions fails them in a way that looks like an answer, which is the failure mode that matters most.

How knowledge graphs retrieve structure

A knowledge graph represents information as entities and the explicit relationships between them, which makes the structure that vector search cannot see into something retrieval can traverse directly. Where vector search finds similar text, graph retrieval follows relationships: given an entity, it can traverse to related entities, follow chains of relationships, and reason over the structure connecting information, which is exactly what relationship-based questions require. The knowledge graph makes the relationships explicit and traversable, so a question that depends on how entities are connected can be answered by following those connections rather than by hoping semantically similar text contains the answer.

This is a genuinely different retrieval mechanism suited to a genuinely different kind of question, and the strongest retrieval systems frequently combine both rather than choosing one. Vector search handles the questions answered by semantic similarity, and graph retrieval handles the questions answered by structure, and a system that has both can route each question to the mechanism that fits it, using similarity where similarity suffices and structure where structure is needed. Building and maintaining a knowledge graph is substantial work, because the entities and relationships have to be extracted and represented and kept current, which is more effort than embedding text for vector search, and that effort is the cost of being able to answer the relationship-based questions that vector search fails. The combination gives the retrieval system reach across both kinds of question, at the cost of building and maintaining the graph alongside the vector index.

The trade-off, and when structure is worth it

The knowledge graph’s power comes with a real cost, and naming it honestly is what keeps the choice deliberate rather than reflexive in either direction. Building a knowledge graph requires extracting entities and relationships from information and representing them as a graph, which is substantial engineering, and maintaining the graph as the underlying information changes is ongoing work, both far more effort than the relatively simple embedding that vector search requires. This cost is justified when the questions the system must answer genuinely depend on structure, and it is not justified for a system whose questions are answered by semantic similarity, where the graph would be expensive infrastructure for questions vector search already handles well.

The judgment is matching the retrieval mechanism to the questions the system actually faces, which requires understanding whether those questions are similarity-based or relationship-based, and many are a mix, which is why the combined approach is often right. A system facing mostly similarity-based questions with occasional relationship-based ones might use vector search as the primary mechanism and a knowledge graph for the relationship-based subset, and a system whose core questions are relationship-based needs the graph as a central component. Building a knowledge graph for a system that does not need it is over-investment in infrastructure the questions do not require, and using pure vector search for a system whose questions depend on structure is under-serving the relationship-based questions that vector search fails. Matching the mechanism to the questions, and validating through evaluation which questions each mechanism actually handles well, is the discipline that gets the retrieval architecture right.

Where it almost broke

We operated a retrieval system built on vector search, which handled most questions well, and a category of questions kept producing incomplete answers that looked authoritative. Examining them showed the pattern: these questions depended on relationships between entities, on how pieces of information were connected, and vector search was retrieving semantically similar chunks that did not capture the relationships the questions required, producing answers that were confident and incomplete because the relevant information was in the structure vector search could not see. The system was failing the relationship-based questions silently, returning plausible answers assembled from similar text that missed the structural connections the questions actually needed.

The correction added a knowledge graph for the relationship-based questions. We extracted the entities and relationships into a graph that made the structure traversable, and routed relationship-based questions to graph retrieval while keeping vector search for the similarity-based questions it handled well. The relationship-based questions started getting answered correctly, because the retrieval could now follow the structure the questions depended on rather than hoping similar text contained the answer. The combined system handled both kinds of question, at the cost of building and maintaining the graph alongside the vector index. The lesson was that vector search fails relationship-based questions silently, and knowledge graphs retrieve the structure that similarity cannot see, which is what those questions require.

Retrieve by the dimension the question lives in

The reframe that completes a retrieval system is matching the retrieval mechanism to the dimension the question actually lives in, similarity or structure, rather than defaulting to vector search for everything. Knowledge graphs and RAG together handle the questions vector search alone fails, because relationship-based questions require following the structure between entities, which semantic similarity cannot see, and the strongest systems combine both mechanisms and route each question to the one that fits. Pure vector search fails relationship-based questions silently, returning plausible answers that miss the structure.

The teams that build retrieval systems handling the full range of questions are the ones that recognised where vector search fails, added knowledge graphs for the relationship-based questions that depend on structure, and combined the two mechanisms so each question is answered by the retrieval that fits it. Vector search is right for similarity and wrong for structure, and defaulting to it everywhere under-serves the questions whose answers live in relationships. Retrieving by the dimension the question actually lives in, similarity where similarity suffices and structure where structure is needed, is what turns a retrieval system that handles some questions into one that handles them all.