IntelliJev context engine eval
Can Jev, a model that answers typed questions but never writes text, find the files a coding task needs better than keyword search, and does handing those files to an agent help it? Measured on real JetBrains/koog commits.
70 held-out tasks
same tasks
[+0.10, +0.23]
2,206 files, about $0.03
The headline
Each task is a real Koog commit subject. The right answer is the .kt files that commit modified, and
every file is read as it was before the commit so the answer can't leak in. The way BM25 and Jev are combined
was chosen on 40 dev tasks, then measured once on 70 test tasks nobody had looked at.
| 70 held-out tasks | recall@5 | recall@10 | recall@20 |
|---|---|---|---|
| BM25 keyword search | 0.41 | 0.53 | 0.63 |
| Jev re-rank alone | 0.51 | 0.63 | 0.74 |
| Jev + BM25 (passes 1 and 2) | 0.54 | 0.69 | 0.80 |
| IntelliJev context engine as shipped (+ stage 3) | 0.57 | 0.69 | 0.80 |
Improvement over BM25, paired bootstrap over tasks. Neither interval touches zero.
How it works
- Sketch every source file into ~300 tokens: path, package, declarations and first doc lines.
- Pass 1. Jev reads 60 sketches per call and gives, for each, P("implementing the task requires reading or editing this file"). BM25 ranks the full text alongside.
- Pool the top 60 of each, then pass 2: Jev asks the same question over the pool's full source, 6 files per call.
- Fuse:
score = jev + 1 / (1 + bm25_rank / 10), chosen on dev. - Stage 3: one Jev
choiceover the top 10, "which file must be edited?", adds2 × pto each. Passes 1 and 2 judge each file alone; this one compares them.
How we got here
The number above is the end of a path with two wrong turns. Both are kept here, because they're why the final number is believable.
- API limits. 100 sketches fit in one call (28k tokens, 0.8 s); 150 fail with
max_tokens_exceeded. A full sweep of Koog is 18-37 calls. - Jev alone loses to keyword search. On sketches only, recall@10 was 0.37 against BM25's 0.46 (10 tasks). Sketches hide what many tasks are about.
- Re-ranking on full source wins, on the tuning set. Pooling BM25 and Jev, then re-reading the pool in full, reached 0.64 against 0.47 on 20 tasks.
- It didn't hold on new tasks. On 30 held-out tasks with those settings: 0.51 against BM25's 0.56. Two causes: commits that mostly add files were in the set (nothing can find a file that doesn't exist yet), and the BM25 weight was tuned on too few tasks.
- Fixed properly. We dropped add-dominated commits, ran the pipeline once over all 136 usable tasks saving every score, chose the fusion on dev only, and reported test once. The credits ran out at task 111; the 26 affected tasks are excluded, and none were in dev.
- Held-out result: 0.69 against 0.53, interval [+0.10, +0.23].
Choosing the fusion on dev
| variant | dev recall@5 | dev recall@10 |
|---|---|---|
| blend_w1.0 ← chosen | 0.54 | 0.71 |
| rrf_k30 | 0.53 | 0.70 |
| rrf_k10 | 0.52 | 0.69 |
| blend_w0.5 | 0.49 | 0.69 |
| rrf_k60 | 0.50 | 0.68 |
| rrf_k5 | 0.51 | 0.65 |
| bm25 | 0.42 | 0.56 |
The pool holds 90% of the right files on test, so there's headroom left in the re-ranking.
A cheaper variant: skip pass 1
Pass 1 is two-thirds of the Jev calls. Scoring only BM25's top files with pass 2 costs little recall:
| test, 70 tasks | recall@5 | recall@10 | Jev calls per pack |
|---|---|---|---|
| Full pipeline | 0.54 | 0.69 | ~56 |
| BM25 top 60 → pass 2 | 0.53 | 0.66 | 10 |
| BM25 top 40 → pass 2 | 0.52 | 0.65 | 7 |
| BM25 alone | 0.41 | 0.53 | 0 |
Against an LLM re-ranker
The obvious question: why not have an LLM re-rank BM25's shortlist? Same 70 test tasks, same pool (BM25's top 30), same full source per file. z-ai/glm-5.3-flash read all 30 files in one call and listed the ones the task needs; Jev's scores for the same 30 files were already saved.
| BM25 top 30, re-ranked by | recall@5 | recall@10 | time per task | cost per task |
|---|---|---|---|---|
| nothing (BM25 order) | 0.41 | 0.53 | 0 | 0 |
| Jev + BM25 | 0.52 | 0.65 | ~1 s | ~$0.002 |
| LLM (z-ai/glm-5.3-flash) | 0.61 | 0.67 | 29 s median | $0.0074 |
The LLM picks the top five better; at ten they tie. Jev + BM25 minus LLM, paired bootstrap: recall@5 -0.09 [-0.15, -0.03], recall@10 -0.02 [-0.07, +0.03]. Jev's advantage is not judgement but economics: about 30× faster and 3.5× cheaper, which is what lets an agent call it before every task. The shipped pipeline also pools Jev's own picks with BM25's, which this 30-file comparison leaves out.
Stage 3: a comparative question
The LLM's lead is at the very top. Passes 1 and 2 ask about each file on its own, while an LLM ranking a list
compares them. Jev's choice type returns a probability per option, so one extra call can ask which of the
top files is the one to edit. Two shapes were tried on dev, the best was fixed, then test was run once.
| dev recall@5 | test recall@5 | recall@10 | |
|---|---|---|---|
| Passes 1 and 2 only | 0.54 | 0.54 | 0.69 |
| Stage 3, top 15 at 4,000 chars (add1) | 0.58 | not run | - |
| Stage 3, top 10 at 6,000 chars (add2), shipped | 0.58 | 0.57 | 0.69 |
On test, stage 3 adds +0.033 to recall@5, 95% interval [+0.006, +0.068]: small, but it clears zero. It costs one Jev call and about 0.4 s. It narrows the LLM's top-five lead rather than closing it.
A second repository
Everything above is Koog. To check it isn't tuned to one codebase, the shipped pipeline ran frozen, with no setting changed, on 40 commits from JetBrains/Exposed (Kotlin ORM, about 746 files per snapshot), once.
| JetBrains/Exposed, 40 tasks | recall@5 | recall@10 | recall@20 |
|---|---|---|---|
| BM25 | 0.43 | 0.56 | 0.68 |
| IntelliJev context engine | 0.54 | 0.62 | 0.77 |
recall@5 +0.10 [+0.01, +0.20]: the gain carries over. recall@10 +0.07 [-0.02, +0.16]: same direction, not significant on 40 tasks. 2.8 s per task, 0 failed calls.
Claude Code, and why a tool isn't enough
Given pack_context as an MCP tool, headless Claude Code (Sonnet) never called it, in
5 runs on the same task, even with an instruction in the system prompt naming the tool. The server was
connected every time, and the Jev ledger didn't move. Claude trusted its own search instead:
| setup | tools Claude used | recall | time |
|---|---|---|---|
| tool connected, no instruction | Grep ×2, Read ×2 | 0.17 | 19 s |
| same, init checked | Grep ×1, Glob ×1, Read ×2 | 0.33 | 18 s |
| + system prompt: call pack_context first | Grep ×2, Read ×3 | 0.83 | 32 s |
| + ToolSearch allowed | Grep ×2, Glob ×1 | 1.00 | 10 s |
| + exact tool name in the instruction | Grep ×1, Read ×3 | 1.00 | 16 s |
So the plugin also ships a Claude Code UserPromptSubmit hook (agent/pack_hook.py). It runs on
every request before Claude sees it, asks the IDE's IntelliJev context engine, and hands Claude the ranked files as context.
The agent never has to decide to use anything; the IDE panel shows what it was given.
Claude Code with the hook
The same Claude Code (Sonnet), prompt and read-only tools, on 8 held-out Koog tasks (vague wording) at their parent commits. The only difference is the hook, which fired on every run. Tokens count everything Claude processed, including cached context.
| 8 tasks | no hook | with hook |
|---|---|---|
| Final recall | 0.54 | 0.50 |
| Wall time, median | 26 s | 26 s |
| Turns, mean | 13.0 | 9.8 |
| Grep + Glob calls, mean | 7.9 | 4.9 |
| Tokens processed, mean | 551k | 467k |
| Cost, 8 tasks | $2.21 | $1.88 |
With the hook, Claude Code needed 25% fewer turns (-3.2 per task, 95% interval [-5.9, -0.5]) and 38% fewer searches, for about the same answers: recall was equal on 7 of 8 tasks. It was not faster, and eight tasks can't settle the token and cost savings.
Per task
| task | turns, no hook | turns, hook | tokens, no hook | tokens, hook | recall, no hook | recall, hook |
|---|---|---|---|---|---|---|
| Update the Koog example that demonstrates a trip planning ag | 6 | 9 | 136k | 280k | 0.50 | 0.17 |
| Add support for OpenAI's GPT-5.5 and GPT-5.5 Pro models. | 7 | 2 | 354k | 78k | 0.33 | 0.33 |
| Fix the agent error being populated incorrectly during refle | 38 | 33 | 2224k | 2335k | 0.50 | 0.50 |
| When creating tools from callables, make sure any custom nam | 21 | 12 | 706k | 423k | 0.50 | 0.50 |
| Return tool calls before the assistant message when handling | 6 | 4 | 212k | 119k | 1.00 | 1.00 |
| Also accept non-streaming Ollama responses that come back wi | 6 | 6 | 179k | 196k | 0.50 | 0.50 |
| Support processing embeddings in batches when using Ollama. | 16 | 8 | 436k | 184k | 0.50 | 0.50 |
| Set the trace attributes on every Langfuse span instead of o | 4 | 4 | 125k | 98k | 0.50 | 0.50 |
In the IDE
Timings from the plugin's own log on Koog, 2,206 candidate files. The first version built sketches from the IDE's Structure View, which runs Kotlin analysis at about 19 ms a file. The shipped version uses the regex sketcher the eval measured (a parity test checks the Kotlin port against it).
| sketch | pass 1 + BM25 | pass 2 | total | |
|---|---|---|---|---|
| Fresh IDE, Structure View sketches | 42.7 s | 2.9 s | 2.2 s | 47.7 s |
| Fresh IDE, regex sketches (shipped) | 1.4 s | 4.2 s | 1.0 s | 6.6 s |
| Warm, sketches cached | 0.1 s | 3.4 s | 0.9 s | 4.4 s |
| After warm-up on project open | 0.0 s | 3.1 s | 0.8 s | 4.1 s |
| Shipped: + stage 3, passes overlapped | 0.1 s | 2.2 s | 2.0 s | 4.4 s |
Every pack: 53-56 Jev calls, 0 failed, about 0.65M input tokens ($0.03). Since the warm-up pass, the plugin sketches every file in the background when a project opens (11 s alongside indexing, no Jev calls).
Why TypeSafe's API and not the free gateway
Vercel's AI Gateway serves the same Jev model for free, but under load it mostly refused:
| probe | result |
|---|---|
| 1 call, 60 sketches | 0 of 1 ok (503) |
| 4 concurrent | 1 of 4 ok (3 × 503) |
| 16 concurrent | 2 of 16 ok (9 × 503, 5 × 429) |
| Sequential, 0.5 s apart, 2-60 files | about half 429 at every size; 503 at 60 |
| Simulated 10-call pack | 28-62 s, ~70% of attempts failed before a retry landed |
TypeSafe's own API ran more than 5,000 calls at 16 concurrent with no rate limiting. The plugin uses it by default and keeps the gateway only as a fallback.
Agent A/B
The same agent (z-ai/glm-5.3-flash), prompt and tools (list_dir, grep,
read_file) on a real checkout of Koog at each task's parent commit, 10 held-out tasks. The only difference
is whether a pack_context tool exists. Every run ends when the agent submits its file list.
| 10 tasks | without pack_context | with pack_context |
|---|---|---|
| Final recall | 0.97 | 0.97 |
| Wall time, median | 87 s | 68 s |
| LLM calls, mean | 16.9 | 13.2 |
| Tokens, mean | 242k | 165k |
| LLM cost, 10 tasks | $0.169 | $0.117 |
| Time to first right file, median | 4.2 s | 12.7 s |
This result is mixed. With the packer the agent used fewer calls and tokens for the same recall. It was
not faster to the first right file: these commit subjects name identifiers, so the explorer's first
grep often hits at once, while the harness's Python pack_context took a median of 9.2 s (the plugin
takes 4.4 s). The agent also kept exploring after it had the pack. Ten tasks is too few to call any of these differences
significant.
Rerun with vaguer wording
Commit subjects often name the exact class or field, which hands grep the answer. So GLM rewrote the
same 10 tasks without any code identifiers, keeping product names a person would say (Ollama, Langfuse). For example,
"AIAgentError populated incorrectly for reflective…" became "Fix the agent error being populated incorrectly during reflective mode.".
| 10 tasks, vague wording | without pack_context | with pack_context |
|---|---|---|
| Final recall | 0.72 | 0.72 |
| Wall time, median | 51 s | 61 s |
| LLM calls, mean | 15.7 | 13.1 |
| Tokens, mean | 190k | 176k |
| Time to first right file, median | 4.3 s | 7.9 s |
Vaguer wording cost both agents recall equally (0.97 to 0.72) and did not
favour the packer: product names still give grep a way in. Running BM25 alongside pass 1 cut the harness's
pack_context from 9.2 s to 3.5 s, yet a grep-first
agent still reaches a right file sooner. Across both runs: the packer does not make this agent faster.
It saves some calls and tokens, and ten tasks can't separate that from noise.
Per task
| task | wall, explore | wall, pack | calls, explore | calls, pack | first right file, explore | first right file, pack | recall, explore | recall, pack |
|---|---|---|---|---|---|---|---|---|
| Update Koog trip planning agent example | 96 s | 38 s | 19 | 4 | 1.9 s | 13.7 s | 1.00 | 1.00 |
| Add OpenAI GPT-5.5 and GPT-5.5 Pro models | 76 s | 76 s | 15 | 16 | 2.3 s | 15.5 s | 0.67 | 0.67 |
| AIAgentError populated incorrectly for reflective… | 113 s | 97 s | 19 | 19 | 28.3 s | 28.3 s | 1.00 | 1.00 |
| Consider tool's customName in tools from callable | 47 s | 69 s | 10 | 12 | 4.3 s | 14.7 s | 1.00 | 1.00 |
| Return tool calls ahead of assistant message in OllamaClient | 93 s | 45 s | 18 | 11 | 1.6 s | 19.9 s | 1.00 | 1.00 |
| Accept text/plain Content-Type on Ollama non-streaming responses | 60 s | 68 s | 19 | 13 | 19.9 s | 11.5 s | 1.00 | 1.00 |
| Process Ollama batch embeddings | 82 s | 69 s | 19 | 19 | 9.0 s | 7.9 s | 1.00 | 1.00 |
| Add trace attributes to all Langfuse spans, not j… | 31 s | 17 s | 12 | 4 | 1.1 s | 11.7 s | 1.00 | 1.00 |
| Stop additionalProperties leaking as additional_properties to Op | 101 s | 187 s | 19 | 19 | 22.9 s | 9.5 s | 1.00 | 1.00 |
| Refactor LLAMA model definitions for clarity | 102 s | 66 s | 19 | 15 | 4.2 s | 9.0 s | 1.00 | 1.00 |
What it cost
| input tokens | cost | note | |
|---|---|---|---|
| Spike and eval, first TypeSafe account | 126M | $5.30 | Ran dry at task 111 of the 136-task run. |
| Agent A/B, Jev (ledger) | 5.3M | $0.22 | Capped at 11M tokens by the ledger. |
| In-IDE checks, Jev | 2.6M | $0.11 | Four packs across three IDE restarts (idea.log). |
| Agent A/B rerun, Jev (ledger) | 5.3M | $0.22 | Vague wording, same cap. |
| Agent A/Bs, GLM on OpenRouter | - | $0.53 | Both arms, both runs, 10 tasks each. |
| LLM re-ranker baseline, GLM | 2.3M | $0.52 | 70 tasks, under a $0.80 cap. |
| Stage 3, Jev (ledger) | 2.2M | $0.09 | Two dev shapes, one test run. |
| Exposed eval, Jev (ledger) | 13.5M | $0.57 | 40 tasks, frozen pipeline, one run. |
| Claude Code runs, Claude plan | - | $4.68 | Five adoption probes and the 16-run hook A/B. |
| Claude Code A/B, Jev (hook) | 4.7M | $0.20 | One pack per hooked run. |
Jev lists at $0.042 per million input tokens, output free. After the first account ran dry, every eval run checks a persistent ledger and stops at a token cap, and the plugin caps each IDE session at 20M tokens.
Caveats
- One repository (Koog), one language (Kotlin). The sketcher handles other languages, but they aren't evaluated.
- Commit subjects stand in for tasks. Real requests are often vaguer, which should favour Jev over keywords, but that isn't measured.
- Recall counts only files the commit modified. A pick that's useful to read but wasn't edited counts as a miss.
- The agent A/B is 10 tasks, with one model, and that model keeps exploring after it has the pack.
- The LLM baseline is one model (GLM-5.3 Flash) on a 30-file pool. A stronger model would likely widen its lead at the top and its latency gap.