--- title: Trove canonical: 'https://www.dealmobile.co.uk/projects/trove' url: 'https://www.dealmobile.co.uk/projects/trove' type: Project summary: 'A Python search engine for your own documents and code. It runs keyword and vector search together, re-ranks the results with a cross-encoder, and answers AI coding tools through an MCP server with 38 tools.' --- # Trove A local search engine for documents, code and web pages, by keyword and by meaning, that serves its results to AI coding tools. ![A Trove search in a terminal for “write a custom reranker plugin”, over an index of Trove’s own documentation used as sample data. Each of the three results shows its re-ranking, keyword, vector and combined scores, and the plugin guide comes first although another chunk had a higher combined score.](https://www.dealmobile.co.uk/sites/default/files/styles/almanac_shot_md/public/content/images/products/trove-light.png.webp?itok=t_QjcyJz) - **Kind:** Tool - **Status:** Archived - **Started:** 2026 - **Licence:** BSL 1.1 - **Stack:** Python, ChromaDB, sentence-transformers, tree-sitter, SQLite, Model Context Protocol, pytest, Docker, GitHub Actions - **Links:** [Source on GitHub](https://github.com/tomgould/trove) Trove indexes a collection of documents and code on your own machine and searches it in a way that suits AI coding tools. A question in plain words finds the right section even when the section uses different words, and an exact name, such as a function or “Appendix C”, still finds that exact thing. Each collection, called a client, has its own folder, settings and index. Claude Code, and other tools that speak the Model Context Protocol (MCP), can search it while they work. Trove is archived. In April 2026 I stopped work on it in favour of [Mullet](https://www.dealmobile.co.uk/projects/mullet.md), whose knowledge engine grew out of it. By then its plugin system was written but not yet loaded at startup, and the rest of its roadmap (a REST API, a web interface and packaging) hadn’t been started. The public code is as it stood then, with fixes to its CI. ## What it does - Reads PDF, Word, Excel, CSV, HTML, Markdown, plain text and MBOX email archives with Unstructured, keeps their headings and sections, and splits them into chunks by section, by meaning or by size. - Indexes code with tree-sitter, as functions, classes and methods, in Python, TypeScript, JavaScript, PHP, HTML, CSS, YAML, JSON, Gherkin and Dart. - Fetches its own sources. A GitHub connector clones and updates the repositories of a user or organisation, and a web crawler saves a documentation site as Markdown, fetching only the pages that have changed on later runs. - Runs keyword (BM25) and vector search together, merges them by weighted scores or by reciprocal rank fusion, and re-ranks the best candidates with a cross-encoder. Optional steps add variety to the results, search with a hypothetical answer or several rewordings of the query, and search again when the first results are weak. - Has the start of reference following, so a result that says “see Appendix C” could bring Appendix C with it, but that code was never wired into search before the project was archived. - Builds an optional knowledge graph of the people, organisations, places and ideas in a client, extracted by a language model, and keeps memories and multi-step workflows in SQLite. - With a language model, in LM Studio or Claude, it can answer questions from the results and cite its sources. - Measures search quality against a client’s own test questions, with mean reciprocal rank, hit rate, precision, nDCG and recall, and can compare two merge strategies side by side. The MCP server has 38 tools. They search one client or several, find a function by name, add a file or a URL to an index, crawl a site, build and query the knowledge graph, and store memories. ## How it’s built Trove needs Python 3.10 or later. ChromaDB stores the vectors, and sentence-transformers makes the embeddings (all-MiniLM-L6-v2 by default, with BGE and E5 models as options) and runs the cross-encoder that re-ranks results. rank-bm25 does the keyword search, with optional stemming from NLTK, and NetworkX holds the knowledge graph. A search runs the keyword and vector searches in parallel, puts their scores on one scale and merges them, then lets the cross-encoder read the query and each of the top 50 candidates together to decide the final order. [Combining keyword and vector search in Trove](https://www.dealmobile.co.uk/writing/combining-keyword-and-vector-search.md) explains each step. Indexing is a producer and consumer pipeline. Files are parsed in parallel while their chunks are embedded in batches, and each batch is written to the index as soon as it’s ready. Stopping it part way loses at most one batch, and a re-run with `--resume` skips the files already indexed. GitHub tokens are read from environment variables, never from the config files. The keyword index is cached as JSON, so a tampered cache file can’t run code, and client names and file paths from MCP calls are checked before they’re used. There are 724 tests, and GitHub Actions runs them on Python 3.10, 3.11 and 3.12, with ruff checking for errors. A Dockerfile runs the MCP server. Trove is source-available under the Business Source License 1.1. You can use it inside your own organisation, but not offer it to others as a hosted service or pass it on. On 1 January 2050 the licence changes to MIT. ## How to use it ```bash git clone https://github.com/tomgould/trove.git cd trove python -m venv venv && source venv/bin/activate pip install -r requirements.txt python scripts/create_client.py ``` The wizard creates a client folder. Put documents in its `documents/` folder or connect a source, then index and search: ```bash python scripts/sources.py crawl --client myproject --url https://docs.example.com python scripts/ingest.py --client myproject python scripts/search.py --client myproject "how do I authenticate" ``` To search it from Claude Code, add the server to your MCP settings: ```json { "mcpServers": { "trove": { "command": "python", "args": ["/path/to/trove/scripts/mcp_server.py"], "env": { "TROVE_DEFAULT_CLIENT": "myproject" } } } } ``` --- Index of this site: [llms.txt](https://www.dealmobile.co.uk/llms.txt)