Indexing code by function, class and method

Trove indexes code as well as documents, so an AI coding tool can ask where something happens and get the code back. Splitting code by size, as you might a document, works badly: a chunk that starts halfway through one function and ends halfway through the next is a poor answer to any question. So Trove parses each source file with tree-sitter and makes one chunk for each function, class or method, labelled with its name, file and lines. This is how that works, and where it falls short. Trove is archived, so this is the public code as it stood.

One extractor per language

core/code/parser.py picks an extractor by file extension, from a map in core/code/extractors/__init__.py. There are ten: Python, PHP, JavaScript, TypeScript, HTML, CSS, YAML, JSON, Gherkin and Dart. Some extensions share one: .vue, .svelte and .mdx go to the HTML extractor, and .scss and .less to the CSS one.

The grammars come ready-built in the tree-sitter-languages package, and each extractor loads its parser the first time it’s needed:

@property
def parser(self):
    """Lazy-load the tree-sitter parser."""
    if self._parser is None:
        import tree_sitter_languages
        self._parser = tree_sitter_languages.get_parser(self.tree_sitter_language)
    return self._parser

The rest is each extractor walking the syntax tree for the node types it knows. A file that can’t be read or parsed gives no chunks and a warning, and the run carries on.

What a chunk holds

Each extractor returns CodeChunk objects, defined in extractors/base.py. The content is the source text of the node, and the metadata says what it is: the language, the kind of symbol, its name and lines, and where they exist, its signature, docstring, parent class and visibility. Chunks from a git repository also get its name, remote and branch. Every code chunk is marked chunk_type: code, which is what the code searches filter on.

The Python extractor shows the approach. A top-level function becomes one chunk holding the whole function, with its parameters and return type as the signature and its docstring pulled out. Each method becomes a chunk of its own, with its class recorded as the parent. Visibility comes from the name: one leading underscore is protected and two is private. Module-level assignments are kept too, when the name is in capitals or starts with an underscore.

A class is the interesting case. Indexing the whole class would repeat every method a second time, and a long class would become one big chunk that matches everything a little. So the class chunk is a summary: the definition line, the docstring and a list of the methods.

class_content = class_def_line
if docstring:
    class_content += f'\n    """{docstring}"""'
if method_names:
    class_content += f"\n\n    # Methods: {', '.join(method_names)}"

A question about what a class is for finds the summary, and a question about how it does something finds the method. The summary still records the class’s full line range, so a tool that wants the whole class can read it from the file.

JavaScript works the same way, with more shapes to find: function declarations, arrow functions assigned to a variable, such as const load = async (id) => ..., and exports. It remembers the byte range of everything it has taken, so a function inside a class or an export isn’t indexed twice. The TypeScript extractor runs the JavaScript one and then adds interfaces, type aliases and enums. .tsx files get the TSX grammar, because, as a comment in the code says, the TypeScript grammar hangs on JSX.

Re-indexing only what changed

Code changes all day, and embedding a whole repository again for one edited file is slow. So core/code/incremental.py keeps a state file with each file’s modification time, a SHA-256 hash of its contents and the IDs of the chunks it produced. On the next run, a file whose modification time hasn’t changed is skipped without being read. A file whose time has changed is hashed, and only a different hash sends it back through the parser.

The chunks of every changed or deleted file are removed from ChromaDB by ID before the new ones go in. The IDs are stable, built from the file, the kind and name of the symbol, its first line and a hash of its content:

key = f"{source}:{symbol_type}:{symbol}:{line_start}:{content_hash}"
return hashlib.sha256(key.encode()).hexdigest()[:12]

Inside a repository, the file list comes from git ls-files --cached --others --exclude-standard, which reads git’s index instead of walking the disk and leaves out whatever .gitignore excludes. Files over 500 KB are skipped by default. With more than ten files, parsing runs in separate processes, each building its own parser, because tree-sitter’s language objects can’t be passed between processes.

Searching it

The MCP server has two tools just for code. trove_search_code runs the usual keyword and vector search, keeps only code chunks, and can narrow them by language and kind of symbol. trove_find_symbol searches for a name, then prefers an exact match:

exact_matches = [
    r for r in results
    if r.get("metadata", {}).get("symbol_name") == symbol_name
]

If nothing matches exactly, it says so and shows the five closest results instead. Names survive the keyword index well. The tokenizer lowercases text and keeps runs of letters, digits and underscores, so fetch_user stays one word and fetchUser becomes fetchuser, and a search for the exact name matches it by keyword.

A client can also switch on cross-referencing in its search settings. For each of the first five document chunks in the results, Trove pulls likely code names out of the text (CamelCase words, snake_case words and anything in backticks) and runs a quick search for matching code. The idea is that documentation goes out of date and the code doesn’t.

What it misses

Running the extractors over a few short sample files for this post turned up two gaps.

The Python extractor looks only for function_definition and class_definition nodes directly inside the module or the class body. tree-sitter wraps anything with a decorator in a decorated_definition node, so decorated code is skipped: a @property, a @staticmethod, a @dataclass class, a function under @lru_cache. It’s missing from the class summary’s list of methods too. There’s code to record a function’s decorators, but since decorated functions are never reached, no chunk ever gets any. Given a plain function, a decorated one, a class with a plain method, a @property and a @staticmethod, and a @dataclass, it found the plain function, the class and one method.

The TypeScript extractor gets its classes from the JavaScript one, which looks for the class name as an identifier. The TypeScript grammar calls it a type_identifier, so TypeScript classes, and every method in them, aren’t indexed at all. Functions, arrow functions, interfaces, types and enums are.

There are smaller problems around search. The code_first search mode filters the results to code, adds 0.5 to every code chunk’s score and switches on cross-referencing. The filter runs first, so cross-referencing finds no document chunks to start from, and adding the same amount to every score changes no order: in practice it’s a code-only search. The cross-referencing’s pattern for file names has a capturing group around the extension, and re.findall returns only the group, so src/api/client.ts comes back as ts and is dropped for being too short. And the setup wizard writes the file extensions you choose to code.extensions, which the indexer never reads. It reads code.languages, and without that, it indexes every extension it supports.

Trove is archived, so these are written down here rather than fixed.

Using it

Code indexing is switched on for each client in its client.yaml, with the extensions listed under languages, which is where the indexer reads them:

code:
  enabled: true
  paths: ["src/"]
  languages:
    python: {extensions: [".py"]}
    typescript: {extensions: [".ts", ".tsx"]}
  exclude_patterns: ["**/node_modules/**", "**/dist/**"]

Then index the code on its own:

python scripts/ingest.py --client myproject --code-only

Later runs re-parse only the files that changed, and --rebuild-code starts from scratch. From Claude Code, trove_find_symbol finds a function or class by name, and trove_search_code answers a question such as “where are retries handled” with the functions themselves.

The code is on GitHub under the Business Source License 1.1.