What the tests found
Three questions, answered with real past changes from open-source projects.
Does it find the right files?
Yes. Jev put 78% of the files each change needed in its top 10. Keyword search put 69%. When the task was written the way a developer would ask, it was 91% against 75%.
Does that hold on big projects?
Yes, though everything finds less. On vscode and airflow, with 8,000 to 11,000 source files, Jev put 60% in its top 10 and keyword search 47%. A search took about 5 seconds and a few cents.
Does it make Claude Code faster?
Not on small changes. Claude Code (Opus) made 40 changes with and without the file list. Both ways took about 10 steps, 20 cents and 40 seconds. Opus found the files quickly on its own.
How the test works
Each test is a real past change. Its commit message is the task, and the files it changed are the answer. The tool sees the project as it was just before the change, so it can't cheat. The score is how many of the needed files it put in its top 10. Each project has 30 tests, run once, after decisions were made on 10 others.
The details
Each project on its own, including the two large ones and two small ones that Jev didn't run on.
The difference between two tools, and the range it plausibly sits in. A line that crosses zero could be luck.
A shorter list shows a bigger gap. In the top 5, Jev found 69% and keyword search 56%.
Time and cost per search. Jev bills its input at $0.042 per million tokens. Keyword search and Laya run on your computer and cost nothing.
The Claude Code test
Each of 40 changes ran three times in a clean copy of the project: on its own, with Jev's file list, and with the file list plus the most relevant lines of code.
| Average per change | On its own | With file list | With file list and code |
|---|---|---|---|
| Steps | 10.4 | 10.3 | 9.9 |
| Searches | 4.5 | 4.3 | 3.8 |
| Needed files it changed | 77% | 80% | 80% |
| Cost | $0.21 | $0.20 | $0.20 |
| Time | 41 s | 43 s | 40 s |
Limits
- Thirty tests per project is enough to see a clear gap, not a small one. On airflow alone, Jev's lead could be luck.
- The realistic requests were written by a language model that saw each change. It wasn't allowed to name files or code from the change, but its wording may still be a little more specific than a real person's.
- Finding the right files isn't finishing the task. The Claude Code test measured steps, cost, time and which files changed, not whether each change was correct. It used one strong model on small changes.
- Two things didn't work. Summaries tailored to each language found no more files and cost more, so they didn't ship. Laya, the local model, found fewer files than keyword search.
- The Jev credits ran out before the large projects were tested with realistic requests, and before the test with weaker models finished.
- All Jev use together cost $7.08.
Everything can be rerun from the repository with bench/mine.ts, bench/run.ts, bench/report.ts and bench/agent.ts. Jev's answers are cached, so a rerun costs nothing.