Benchmarks
How well repo-sensei follows a feature from the screen to the data, measured on code it was never tuned on.
Last updated
CrossLayerBench
Most code-search benchmarks ask whether a tool finds a relevant file. CrossLayerBench asks whether it finds the path. Each flow starts where a user acts: a form, a page, a button. It ends at the table the request reads or writes. Every hop in between was labelled by hand at a pinned commit, through the state, the API client, the route handler, its services and background jobs:
- 101 published flows over 5 open-source repositories: FastAPI, Flask, Next.js with Prisma, and Spring Boot with JPA, under React front ends.
- 20 held-out flows labelled the same way and kept private. A published benchmark can end up in training data; these check that a gain is real.
- Renamed copies of the Python and TypeScript repositories. Every project-defined name is replaced, to measure tracing that doesn't lean on names.
The flows, their format and the labelling conventions are at github.com/K-sau07/crosslayerbench, under CC BY 4.0.
Results
These are repo-sensei's tracer on the published flows (engine ff84fa2e68b4). It builds a call graph across languages from a checkout and makes no model calls. Hop recall is the share of a flow's labelled hops the best path reaches. Exact is a best path that is the labelled path in order, and exact@3 counts any of the three paths it offers.
| Repository | Flows | Hop recall | Hop precision | Exact | Exact@3 | Layers hit |
|---|---|---|---|---|---|---|
| fastapi-template* (tuned on) | 8 | 0.95 | 0.96 | 0.75 | 1.00 | 0.75 |
| fastapi-template (renamed) | 8 | 0.70 | 0.79 | 0.38 | 0.50 | 0.38 |
| langflow* (tuned on) | 17 | 0.70 | 0.65 | 0.29 | 0.41 | 0.41 |
| langflow (renamed) | 17 | 0.23 | 0.49 | 0.00 | 0.00 | 0.00 |
| redash | 25 | 0.23 | 0.65 | 0.00 | 0.00 | 0.00 |
| redash (renamed) | 25 | 0.21 | 0.80 | 0.00 | 0.00 | 0.00 |
| umami | 26 | 0.33 | 0.57 | 0.00 | 0.00 | 0.00 |
| umami (renamed) | 26 | 0.18 | 0.79 | 0.00 | 0.00 | 0.00 |
| jhipster-react | 25 | 0.28 | 0.65 | 0.00 | 0.00 | 0.00 |
* The tracer was developed against these two repositories.
The table says plainly where the tracer stands:
- Repositories it was tuned on: it finds their paths nearly whole.
- Repositories it has never seen (redash, umami, jhipster): it reaches the first hop or two, then loses the path. Typical breaks are:
- an API helper that builds URLs at run time;
- Next.js handlers found by their file's place in the tree;
- Flask-RESTful resources registered as plain statements;
- Redux thunks.
- Renamed copies: the scores drop, which shows how much of what it finds rests on names.
That gap is the work ahead, and these flows are how it will be measured. Meanwhile, every hop in a trace says how it was found: verified from the code itself, inferred by a rule, or unresolved, where the graph has no next hop.
Reproduce
The scorer and the tracer are open source in repo-sensei (packages/evals). A run makes no model or embedding calls, so it costs nothing:
- Check out each repository at the commit in the dataset's repository list.
- Put the bench file at
evals/private/crosslayerbench/v1.yamlin a repo-sensei checkout. - Run
sensei-eval run --suite crosslayerbench-v1 --retriever trace --repo umami --checkout <path>. It writes a report with every flow's hits and misses.
FixBench
FixBench is 30 bugs planted in a small full-stack app, with fixes that ship as pull requests. The bugs include off-by-ones, timezone and DST windows, N+1 queries, a lost update under concurrency, a wrong SQL filter, and an authorization bypass. A fix counts only when:
- a hidden test written for the bug passes;
- the app's own tests still pass;
- nothing outside the bug changed.
The harness itself is checked with known-good patches: all 30 are judged fixed, and none unsafe. The model's results will be published here from its first full run. Until then, there is no number to show.