Combining keyword and vector search in Trove

Trove is a search engine for documents and code, built to give AI coding tools the right context. Its searches have to cope with two kinds of question. Some are about meaning: “how do I authenticate” should find a section called “Signing requests”. Others are about an exact string, such as a function name, an error code or “Appendix C”. Vector search is good at the first kind and unreliable at the second, and keyword search is the other way round. So Trove runs both, merges them, and lets a third model decide the final order. This is how each step works.

Two searches at once

The two searches run at the same time, in a pool of two threads. If one fails, the other’s results are used on their own.

The keyword side is BM25, from the rank-bm25 library. Text is lowercased, split into words of two or more characters, and stripped of common English words. Stemming is optional, with NLTK’s Porter or Snowball stemmer, so that “countries” and “country” match. Each client can also list synonyms, which are added to the query but not to the index:

tokenization:
  stemmer: "porter"
  synonyms:
    kyc: ["know your customer"]

The vector side embeds the query with a sentence-transformers model, all-MiniLM-L6-v2 by default, and asks ChromaDB for the nearest chunks by cosine distance. A chunk’s score is one minus its distance.

Two scores on one scale

The two scores don’t measure the same thing. A BM25 score has no upper limit and depends on the query and the collection, while a cosine similarity is at most 1. Adding them as they are would let BM25 drown out the vectors.

By default, Trove divides each list’s scores by the best score in that list, so the top result on each side scores 1, and then takes a weighted sum, half and half unless the client sets other weights. A chunk that only one search found gets nothing from the other. Z-scores are an alternative way to normalise.

The other option ignores the scores and uses rank, with reciprocal rank fusion:

for rank, (chunk, score) in enumerate(bm25_results):
    chunk_id = chunk["id"]
    rrf_scores[chunk_id] = rrf_scores.get(chunk_id, 0) + 1.0 / (k + rank + 1)

The same loop runs over the vector results. With k at 60, first place in a list is worth 1/61 and tenth place 1/70, so a chunk that sits fairly high in both lists beats one that tops one list and is missing from the other. Rank fusion copes better when the two searches score on very different curves. evaluate.py --compare "weighted,rrf" measures both against a client’s own test questions, so the choice can rest on numbers.

Letting a cross-encoder decide

The embedding model turns the query and each chunk into vectors separately, without either seeing the other. A cross-encoder reads the query and a chunk together and scores how well one answers the other. It’s far more accurate and far slower, so it only sees the best 50 candidates from the merge. Trove runs cross-encoder/ms-marco-MiniLM-L-6-v2 locally, or calls a Cohere-compatible reranking API and falls back to the local model if the API fails.

The screenshot on the Trove page shows why this step matters. It’s a search of Trove’s own documentation for “write a custom reranker plugin”. The plugin guide came first, with a reranking score of 4.74, although a chunk about source connectors had a higher combined score, 0.840 against 0.811. The chunk with the best keyword score of all came third, with a reranking score below zero. It was a block of example settings that happens to contain the words “custom reranker”, and it doesn’t answer the question.

When the top merged score is above 0.9, both searches agree strongly, so Trove re-ranks only the top 15 candidates instead of 50. Rank fusion scores are tiny, at most 2/61, so this never happens with rank fusion, and a comment in the code says that’s intended.

Optional extras

Maximal marginal relevance, when it’s switched on, stops five near-identical chunks filling the results. It picks results one at a time, scoring each candidate as λ times its relevance minus (1 − λ) times its similarity to the closest result already picked.

With a language model set up, two more steps are available. HyDE asks the model to write a short answer to the question, and searches with that answer’s embedding as well as the query’s. Multi-query search asks for several rewordings of the query, runs each one, and fuses the results by rank. Both are off by default, because each adds a model call to every search.

The MCP server gives these combinations names, so an AI tool can choose one per search. precise weights keywords at 0.7 and returns ten results, for names and IDs. broad weights vectors at 0.7 and returns thirty. exhaustive uses rank fusion and re-ranks 100 candidates. diverse switches on maximal marginal relevance, and code_first returns code rather than documents.

Keeping it fast

Building the BM25 index means tokenising every chunk, so the index is cached on disk as JSON. The cache is keyed by a hash of each chunk’s ID and first 100 characters (a sample of them, for very large collections), the tokenizer settings and the BM25 parameters, so changing the stemmer or a synonym rebuilds it. It used to be stored in a format that can run code when it’s loaded, so a tampered cache file was a risk. JSON can’t do that. Search results can be cached too, for a set time, keyed by the query and every filter.

Over the 53 chunks of Trove’s documentation, the search in the screenshot took about a third of a second once the models were loaded. Two thirds of that was the two searches, and most of the rest was the reranking.

Measuring it

Changing a weight or a model is guesswork without numbers. A client can keep a file of test questions, each with the passages it should find. evaluate.py reports mean reciprocal rank, hit rate, precision, nDCG and recall over them, and can ask a language model to judge the answers too. benchmark_embeddings.py runs the same questions against different embedding models.

Trove is archived now. The code is on GitHub under the Business Source License 1.1.