When Vectors Aren't Enough - Hybrid Search, Knowledge Graphs and Choosing

The first three parts assumed the answer was tuning a vector pipeline — chunk better, enrich chunks, rewrite queries, rerank results. This part is about the cases where the substrate itself is wrong, and about how to decide what to build.

Two things vectors are genuinely bad at. One of them has a fix so cheap it should probably be your default.

Embeddings can’t do exact tokens

This is the failure I see teams tune around for weeks without naming.

Semantic search works because embeddings put similar meanings near each other. “Can I send this back?” finds a document about refunds with no shared vocabulary. That’s the whole pitch, and it’s real.

Now search for ORA-01555. Or SKU 4417-B, or a customer named Okonkwo, or the config key spring.ai.model.chat.

These have no meaning to embed. An error code is an arbitrary string; the model has no useful representation of it, so it lands somewhere near other things that look like error codes. You will confidently retrieve ORA-01552. Semantic similarity is working exactly as designed and giving you precisely the wrong answer.

Keyword search — BM25 — has the opposite profile. Useless for “send this back”, perfect for ORA-01555, because it matches the token.

Hybrid search runs both and fuses the results. It isn’t exotic and it isn’t new; it’s what search engines did for twenty years before embeddings, now put back alongside them. The usual fusion is reciprocal rank fusion: score each document by its rank in each list, sum, re-sort. Documents that both methods like win; documents only one method found still surface.

The evidence is right there in the numbers from part two. Anthropic’s contextual embeddings alone cut retrieval failures 35%. Adding contextual BM25 took it to 49% — keyword search contributed nearly as much as the contextual enrichment did. That’s a striking result for a technique most “advanced RAG” posts don’t mention at all.

Spring AI has no HybridDocumentRetriever, but you don’t need one — DocumentRetriever is an interface, and you already have a full-text index sitting in Postgres:

public class HybridRetriever implements DocumentRetriever {

    private final DocumentRetriever vectorRetriever;
    private final KeywordRetriever keywordRetriever;   // Postgres FTS, Elasticsearch, Lucene

    @Override
    public List<Document> retrieve(Query query) {
        List<Document> dense = vectorRetriever.retrieve(query);
        List<Document> sparse = keywordRetriever.retrieve(query);
        return reciprocalRankFusion(dense, sparse);
    }
}

Then hand it to the advisor and rerank the fused list, exactly as in part three. Some vector stores do this for you — pgvector alongside Postgres full-text search is the combination I’d reach for first, since it’s one database you already run.

If your corpus contains identifiers, part numbers, error codes, version strings or proper nouns — and most technical corpora do — this is not an advanced technique. It’s a missing default.

Knowledge graphs: relationships vectors can’t see

The second thing vectors can’t do is answer questions about connections.

“Which of our customers are affected by the outage in the Frankfurt region?” No single chunk contains that. The answer is assembled by traversing relationships: customers → subscriptions → services → region. Chunk your documentation however you like and no similarity search will produce it, because the answer isn’t written down anywhere — it’s implied by structure.

Knowledge graphs store entities and the relationships between them, and retrieval becomes graph traversal, usually alongside vector and keyword search over node content.

Be clear-eyed about the cost. You need entity extraction at ingestion, which is LLM work and imperfect. You need a graph database — Neo4j is the common choice, and it is available as a Spring AI vector store, though the vector store interface gives you similarity search over nodes, not traversal. You need a schema, and schemas need maintenance as your domain shifts. Query latency is higher. In the source catalogue this series started from, knowledge graphs carry the highest cost rating of any strategy.

So: worth it when relationships are the product — supply chains, org structures, dependency graphs, fraud rings, anything where “what connects to what” is the question people actually ask. Not worth it because your documents mention entities. Most RAG applications answer questions that live in a single passage, and for those a graph is expensive scaffolding around a problem you don’t have.

If you’re unsure, that’s the test: write down ten real user questions and count how many need two or more hops. If it’s none, you don’t need a graph.

Choosing: symptom to strategy

Here’s the table I’d actually keep. Start from the failure you can observe, not from the technique you read about.

The three in bold are the ones I’d reach for before anything else — contextual retrieval, reranking, hybrid search. Between them they address the majority of what actually goes wrong, and none is exotic.

What I’d build, in order

If I were starting a RAG system today:

1. Baseline. Structure-aware chunking, a sensible similarity threshold, metadata filtering for tenancy. This is the basic RAG post.
2. A golden dataset. Thirty real questions and their expected sources, and a recall@k number in CI.
3. Hybrid search, if the corpus has identifiers. Usually it does.
4. Reranking. Retrieve twenty, rerank to five.
5. Contextual retrieval, when the ingestion cost is justified — with prompt caching.
6. Then, and only against measurements, the rest.

Notice what isn’t in the first five: agents, graphs, fine-tuning, late chunking. Those are answers to problems you should be able to demonstrate you have.

The thing that took me longest to accept

Most “advanced RAG” content — including the catalogue this series started from — presents its techniques as a menu of roughly equivalent options, each with a tidy row and some stars. That framing is the actual problem.

The techniques differ by an order of magnitude in payoff. Contextual retrieval: 35% fewer retrieval failures, rising to 67% with reranking, all from published measurements. Late chunking: about 3.6% relative in its own paper. Fine-tuned embeddings: 5–10%, plus a permanent re-embedding treadmill. Hybrid search: nearly as much as contextual embeddings, and it’s a well-understood technique from before any of this existed.

They also differ in kind. Some fix precision, some fix recall, some fix a failure that only occurs with certain data. Adopting all of them gets you a slow, expensive pipeline that is worse than a careful baseline plus two well-chosen additions — and you won’t know which change hurt, because you made them together.

The discipline is dull and it’s the whole game: name the failure, pick the one strategy that addresses it, measure, keep it or throw it away.

Conclusion

Vectors are a good default and a bad universal. They can’t match exact tokens, which is why hybrid search deserves to be a default rather than an advanced technique, and they can’t traverse relationships, which is why graphs exist and why most applications don’t need one.

Across the series the shape is consistent: retrieval failures are diagnosable, the fixes are specific to the diagnosis, and the published numbers are public and uneven. Contextual retrieval, reranking, hybrid search. Start there, measure, and add nothing you can’t justify with a number.

More info: docs.spring.io/spring-ai/reference/api/retrieval-augmented-generation