IntelliJev jury Q&A

A model from TypeSafe that can't generate text. You give it some data and typed questions; it returns calibrated probabilities. That makes it fast (about 1 s a call) and cheap ($0.042 per million input tokens, output free). We ask it, for every file, "does this change need this file?"

We tested exactly that. An LLM re-ranking the same shortlist ties us at the top 10 and is a bit better at the top 5, but takes about 29 s per request against ~1 s, at 3.5× the cost. For a step that runs before every agent task, speed is the product.

"An LLM can do this judgement. Jev does it thirty times faster, which is what makes it usable on every request."

The standard keyword-search formula: a file scores higher when it uses the task's words often, when those words are rare in the project, and it isn't just a long file. It's the honest baseline, since "just grep it, done well" is exactly this. It's also half of our shortlist, because it's genuinely good when a task names an identifier.

Because Jev reading only a summary of each file lost to keyword search. The fix is standard in search: cast a cheap, wide net first (keyword search plus Jev on summaries, best 60 of each), then look carefully at just those ~100 files, with Jev reading their full source. The careful look is where Jev is strong.

Three safeguards. (1) No leakage: each test is a real commit, and we read the code as it was before the change, so the answer isn't sitting in the files. (2) Held out: every setting was chosen on 40 dev tasks, then we measured 70 other tasks once. (3) Intervals: the +16-point gain has a 95% interval of +10 to +23.

And we publish the failure: our first held-out attempt lost to keyword search. The eval report explains why and what we fixed.

No. We ran the pipeline, frozen with no changes, on a second JetBrains repo, Exposed. It clearly beat keyword search in its top 5 (0.54 against 0.43). At top 10 it points the same way but isn't significant on 40 tasks. Both repos are Kotlin; other languages run but aren't measured.

"It makes them do less work, not finish sooner."

With the hook, Claude Code needed 25% fewer turns (interval −5.9 to −0.5 per task), 38% fewer searches and 15% less money for the same answers. Wall time was the same. Good agents are already fast at grepping for words the task names. The saving shows up in steps and spend.

No, and that's a finding. Offered as an MCP tool, Claude Code never called it in five tries, even when its system prompt said to. It trusts its own search. So a Claude Code hook runs before every request and hands over the ranking. The agent doesn't have to decide anything; it just starts with the files.

Different mechanism. Embeddings compare vectors for similarity and need an index kept in sync. Here there's no index to build: Jev reads the actual code and answers "does this change need this file?" as a probability, fresh on every request, combined with keyword search. We didn't benchmark an embedding baseline; the baselines we did run are keyword search and an LLM re-ranker.

IntelliJev is one IntelliJ plugin with Find context and Review changes in the same tool window. The context engine scans eligible project files, skips generated and excluded code, reads unsaved editor text, and builds summaries in the background. It also registers pack_context on IntelliJ's built-in MCP server. Selected context can be handed to the reviewed-edit workspace, where the developer inspects a proposed diff before applying a change.

The measured Jev ranking cost about $0.03 on a 1,700-file project. The plugin checks a 20M reported-token session threshold before starting another Jev pack. An in-flight pack can exceed it, and missing usage makes spend uncertain; this is not a hard billing cap. Typing previews and Fast keywords make no model calls. Local Laya has no API fee, excluding hardware and electricity. Ask LLM is a separate cloud cost.

For Jev: to TypeSafe's API, a short summary of every candidate file and the first 6,000 characters of about 100 shortlisted files per ranking. Local Laya sends bounded excerpts to your local server; Fast keywords stays in the IDE. Ask LLM sends the picked source to the configured cloud provider. The plugin keeps sketches in memory and saves provider preferences; credentials use environment variables or the IDE password store.

If Jev answers nothing at all, the plugin says so rather than quietly showing keyword results. Partial failures (a slow call, a failed later stage) only degrade the ranking, with incomplete scoring disclosed. Calls retry with backoff. The live hook defaults to local keywords, has a 25 s deadline, and exits silently on failure. Explicit Laya hook use needs a longer deadline on this CPU; explicit Jev use may incur API charges.

Broader language and repository evaluations, faster ranking, and more task-aware decisions inside the same IntelliJev workflow.