All work04 / 07

Own product · Code search for AI agents

TypeTrace

A code-search tool that cut a coding agent’s token use by 23% against graphify — in a pre-registered benchmark anyone can re-run.

recall@10 on three repos; nearest tool 0.10–0.31
0.33–0.69
fewer tokens per task than graphify
23%
agent runs, scored with no LLM judge
216
tokens, the most an answer may cost
800
From the repo

Benchmarked · open source

Role
Solo — research and engineering
For
Independent research
When
Nov 2026
Stack
TypeScript · Node 22 · TypeScript compiler API · MCP SDK · Transformers.js · tree-sitter · ripgrep · Vitest · Python
Start a project

Building something like this? Tell me about it.

Does it find the code that matters?Recall@10 — the share of the functions a bug or question is really about that appear in the top ten results. Same tasks, snapshots and budget for every tool; no model calls.
  • nestjs-boilerplate6.3k lines · 20 tasks

    • TypeTrace0.69
    • code-review-graph + embeddings0.31
    • graphify0.12
  • Immich server63k lines · 26 tasks

    • TypeTrace0.48
    • code-review-graph + embeddings0.26
    • graphify0.04
  • Twenty server763k lines · 18 tasks

    • TypeTrace0.33
    • code-review-graph + embeddings0.10
    • graphify0.01

Paired over tasks; 16 of 18 comparisons have p ≤ 0.001. Both exceptions are on Immich against code-review-graph + embeddings. A good part of the lead is the embedding model rather than the graph — the project’s README says so.

The brief

Code search for coding agents, measured rather than claimed.

TypeTrace indexes a TypeScript codebase with the TypeScript compiler and gives an AI coding agent one tool that answers in at most 800 tokens: where the relevant code is, and what calls it. It is benchmarked head to head against two popular graph tools.

Tools that promise to make coding agents smarter mostly publish their own numbers on their own tasks. I wanted to know whether a typed code graph actually helps an agent — so the hypotheses, the tasks and the statistical tests were written down before the runs, the answers were frozen, and the competitors ran at their latest release in their strongest documented setup.

What I built

What I built

  • 01

    The compiler builds the graph

    Typed calls, dependency injection, decorators, GraphQL resolvers, event and job dispatch and database queries, resolved by the TypeScript compiler rather than guessed from text. No LLM builds or queries the index, and nothing leaves the machine.

  • 02

    One tool, a hard budget

    A single MCP tool returns file-and-line pointers with each function’s callers and callees, capped at 800 tokens. Keyword search and a local embedding model are fused with the graph.

  • 03

    Pre-registered and frozen

    Hypotheses and tests were fixed before any run; ground truth is frozen and independent of every tool; scoring is automatic. Comparisons are paired by task and corrected for multiple tests.

  • 04

    Fair to the competition

    A harness bug that broke a competitor’s first run was fixed before any result was read, and an adapter that would have let a competitor index the already-fixed code was corrected — both changes help the other side.

What it looks like

Does it make an agent cheaper?Tokens per run for a coding agent on 18 held-out tasks in a 763k-line codebase — three repeats per task and arm, scored automatically against frozen answers.
  • Tokens per runtask score 0.896 · 0.837 · 0.867 · 0.851

    • TypeTrace453k
    • grep and read only513k
    • graphify521k
    • code-review-graph + embeddings779k

Pre-registered, Holm-corrected: 23% fewer tokens per task than graphify (p = 0.003) and half as many as code-review-graph (p < 0.001). The differences in task score are not statistically significant.

The hypothesis that didn’t hold

It began as a single-repository benchmark, where the hybrid found more of the code a bug touches than embeddings did — recall 0.67 against 0.47 on fourteen held-out tasks. I expected the graph to make agents more accurate too. It didn’t, measurably: the same tool with and without the graph differed by 0.02, and TypeTrace’s lead in task score over the other tools is within run-to-run noise at 18 tasks. A good part of the retrieval lead turned out to be the embedding model, not the graph.

So the claim shrank to what replicated. Agents using TypeTrace spent 23% fewer tokens per task than with graphify and half as many as with code-review-graph, on 13 and 17 of 18 tasks. Cheaper, not smarter — and the write-up says so before it says anything else.

Limits

TypeScript only — it cannot run on Python or Go, so it says nothing about them. Three repositories, 18 to 26 tasks each, one model family. It has not been shown to beat the other tools on their own benchmarks.

Have something that needs to be right?

Open to select projects and roles. I read every message myself and reply within one working day.

Next case study

Unbound

A travel-readiness app that says “Verify” instead of guessing, with 3,409 tests behind the verdict.

Read the case study