IntelliJev context engine wiki

Everything you need to explain how it works and why each design choice was made. The eval report has the full numbers; this page explains them.

In one minute

Given a task in plain English, IntelliJev's Jev context engine ranks every eligible source file in the project by how likely the task is to need it. It combines two very different signals: BM25, a classic keyword-matching formula, and Jev, a model that answers yes/no questions with calibrated probabilities. Jev first judges a short summary (a sketch) of every file; the best candidates from both signals are then re-read by Jev in full, the two scores are combined, and a final comparative question reorders the top 10. It runs inside IntelliJ, and serves people (a tool window) and agents (an MCP tool, and a Claude Code hook that injects the ranking before the agent starts).

Architecture

One pack on Koog (1,705 Kotlin files): about 50 Jev calls, 0.6M input tokens, $0.03, about 4 s end to end. Sketches are built in the background when the project opens and cached by file modification stamp and sketch mode. A pack still reads missing or changed sketches; warm-up reduces that work.

Jev

Jev is TypeSafe's first "System One" model. It cannot generate text. You send it a state (any JSON) and a set of typed questions; it returns calibrated probabilities. That's the whole interface. We use two question types:

• noul returns P(statement is true). Our statement: "Implementing the change described in `task` requires reading or editing the file in `f07`." One question per file.
• choice picks among up to 255 labelled options and returns a probability for each. Stage 3 and the role labels use it.

Batching is the key trick: the state is an object with one key per file (f000, f001, …), and each question names its key, so 60 files share a single request. The limit is about 32k tokens of state per call: 100 sketches fit, 150 don't. Price: $0.042 per million input tokens, output free. Latency: 0.5-2 s a call, and we run up to 48 at once.

BM25

BM25 is the standard keyword-ranking formula behind most search engines. For each word in the task, a file scores higher when:

• the word appears in it often (term frequency), with diminishing returns so 50 mentions isn't 50× better than one;
• the word is rare across the project (inverse document frequency): "OpenAI" is informative, "the" isn't;
• the file isn't just long (length normalisation), so a huge file doesn't win by containing every word.

We split identifiers, so cachedContentTokenCount matches the words "token count". BM25 reads full source, costs nothing and takes milliseconds.

Sketches

A sketch is a ~300-token summary of a file: its path, its package, and its declarations (classes, functions, properties) two levels deep, each with the first line of its doc comment. For example:

path: prompt/.../google/GoogleLLMClient.kt
package ai.koog.prompt.executor.clients.google
- public class GoogleClientSettings(  // Configuration settings for the Google AI client.
  - public val baseUrl: String
- public open class GoogleLLMClient @JvmOverloads constructor(  // Implementation of [LLMClient] for Google's Gemini API.

Sketches exist so that 60 files fit into one Jev call. Cost was never the reason: reading every file in full would cost about $0.11. The plugin uses the same regular-expression sketcher the eval measured, checked by a parity test on 60 real files. Sketches built from IntelliJ's Structure View read better but take about 19 ms a file cold and were never measured, so they're behind a flag.

Why re-rank

This is the core design decision, and it came from a failure. Jev scoring sketches alone lost to BM25 (recall@10 0.37 against 0.46). A sketch shows a file's outline, but many tasks are about something inside a function body.

The fix is a classic in search engineering: retrieve, then re-rank. Cast a cheap, wide net first; then look carefully at only what you caught.

1. Wide net. Take BM25's top 60 and Jev-on-sketches' top 60. They catch different files: BM25 finds files that use the task's words, Jev finds files that fit its meaning. Together that's about 100 files, and the right answer is in this pool about 90% of the time.
2. Careful look. Jev reads those ~100 files in full (6,000 characters each, 6 files per call). The pool is small, so full source is affordable, and full source is where Jev is strong.

Fusion

The final score combines both signals:

score = jev_probability + 1 / (1 + bm25_rank / 10)

Jev's full-source probability is 0 to 1. The BM25 term is a bonus that decays with keyword rank: 1.0 for BM25's top file, 0.5 at rank 10, 0.1 at rank 90. So a file BM25 ranks highly needs only moderate Jev support, and a file BM25 missed needs strong Jev support to climb. The equal weighting was chosen on the dev tasks from a grid of weights and a second method (reciprocal rank fusion). Equal weight won, and then we measured test once.

Stage 3: a comparative question

Passes 1 and 2 judge each file on its own. An LLM ranking a list compares files, and that's where it beat us (top-5 recall). So one more Jev call asks a single choice over the fused top 10: "Which file must be edited to implement the change described in `task`?" Each file's probability, times 2, is added to its score. Only the order inside the top 10 changes. It lifted held-out recall@5 from 0.54 to 0.57, for one call and about 0.4 s.

Role labels

A separate Jev call, run in parallel with stage 3, asks per top file which part it plays: edit, test, example (a pattern to copy), dependency (an API used but not changed) or unrelated. A label shows when Jev is at least 50% sure. It's a separate call on purpose, so it can't change the measured ranking. Labels are for display only and weren't evaluated.

How we measured

Results

Shipped vs BM25: recall@10 +0.16 [+0.10, +0.23]. Stage 3 vs fused: recall@5 +0.03 [+0.006, +0.068]. Stage 3 only reorders the top 10, so it can't change recall@10 or @20.

Wrong turns

1. Jev alone lost to BM25. This led to re-ranking on full source.
2. The first "win" didn't hold. Tuned on 20 tasks, fused recall@10 looked like 0.64 against 0.47. On 30 unseen tasks it lost (0.51 against 0.56). The causes were unfair new-file commits in the set and a fusion weight tuned on too few tasks. Fixed with the filter and a proper dev/test split.
3. The free gateway couldn't serve it. Vercel's AI Gateway answered about 30% of calls under load, so the plugin uses TypeSafe's own API.
4. Agents ignored the tool. Claude Code never called pack_context, even when told to. So it gets the ranking through a hook instead.
5. A code review caught a mismatch. The plugin's sketcher differed from the eval's on 2% of files. Fixed, with a stronger parity test.

IntelliJ internals

• Collecting files: ProjectFileIndex.iterateContent, skipping excluded, ignored, library and generated sources, binaries, docs/config and files over 100,000 bytes on disk or 100,000 unsaved editor characters. Project containment also applies to manual pins and source export. CONTEXT_PACKER_EXTENSIONS=kt restricts discovery to the language used in the evaluation.
• Threading: file text and sketches are read in many short parallel read actions, never one long one, and never with a network call inside, to limit interference with editing. Network runs on Dispatchers.IO, UI updates on the EDT, and cancellation always propagates.
• Caching and warm-up: sketches are cached by modification stamp and built when the project opens (a ProjectActivity, no model calls). Fast keywords skips sketch warm-up. Switching Jev/Laya sketch formats invalidates that cache. Unsaved editor text is used when a file is open, within the source-size limit.
• Tool window: task field, search-as-you-type preview (after 700 ms: full-source local BM25, zero model calls, labelled preview), explicit provider choice and Pack, Jev role chips, Add open file / Drop, Copy prompt, Ask LLM.
• MCP: pack_context(task, limit, provider) is registered on the IDE's built-in MCP server through the com.intellij.mcpServer.mcpToolset extension point. Agent requests are replayed in the tool window.
• Keys: environment variables win, otherwise the IDE password store. TYPESAFE_API_KEY first, Vercel gateway as fallback; a changed key takes effect on the next pack.
• Resilience: retries on 408, 429 and 5xx with jittered backoff, honouring retry-after (capped at 5 s). If every pass-1 call fails, the pack errors rather than pretend keyword results are Jev's. Later failures only degrade the ranking.

Agents: tool vs hook

An MCP tool depends on the agent choosing to call it. In five headless runs Claude Code never did, even with a system prompt naming the tool; it trusted its own Grep and Glob. The weaker GLM agent did call it, every time.

So agent/pack_hook.py is a Claude Code UserPromptSubmit hook. It runs on every request before Claude sees it, asks the IDE's pack_context, and adds the ranked files to Claude's context. The installed live hook defaults to keywords, with a 25 s deadline and silent failure. Set CONTEXT_PACKER_HOOK_PROVIDER explicitly to jev, laya, or configured to use a model. Laya needs a longer deadline on this CPU. The published Claude experiment used Jev: a warm hook added about 4 s, and the first MCP call after an IDE restart took about 10 s. Those measurements do not apply to the live keyword default.

Cost and safety

• Per Jev pack: about 0.6M input tokens, $0.03 in the reported sample. Current typing previews use local keywords and make zero model calls. Local Laya has no API fee; hardware and electricity are excluded.
• Budget threshold: the IDE checks 20M reported Jev tokens before starting another pack. The in-flight pack can exceed it, and missing usage makes cost uncertain. It is not a hard billing cap. Eval scripts separately share a persistent ledger and check JEV_TOKEN_BUDGET.
• Data: sketches of every candidate file, and the full source (first 6,000 characters) of about 100 shortlisted files, are sent to TypeSafe's API on each Jev pack. Laya uses a local endpoint; Fast keywords stays in the IDE. Ask LLM separately sends the picked source to OpenRouter.

Limits

• Measured on Kotlin, in two JetBrains repositories. Other languages run but are unmeasured.
• Tasks are commit subjects; real requests are often vaguer. We tried identifier-free rewrites in one agent test; both arms got worse equally.
• Recall counts only files that were edited. A useful file to read that wasn't edited counts as a miss.
• Agent results are small samples (8 and 10 tasks) with one model each.

Code map

Glossary

BM25

Keyword ranking: frequent in the file, rare in the project, adjusted for length.

Jev / System One model

A model that returns typed decisions with probabilities, and never text.

noul / choice

Jev question types: a probability that a statement is true; a probability per labelled option.

Sketch

A ~300-token outline of a file, so 60 fit in one Jev call.

Pool

The shortlist: BM25's top 60 plus Jev-on-sketches' top 60.

Re-rank

Score a shortlist again, more carefully (here: Jev on full source).

Fusion

Combining two rankings into one score.

Stage 3

One comparative Jev choice over the top 10 that reorders them.

Recall@k

The fraction of the truly needed files that appear in the top k.

Held-out / dev / test

Tune on dev, report once on test, which was never looked at while tuning.

Bootstrap interval

The range the improvement would plausibly take with a different sample of tasks.

Leakage

The answer showing up in the input. Avoided by reading code as it was before each change.

MCP

Model Context Protocol: how agents discover and call tools. IntelliJ ships an MCP server.

Hook

A command Claude Code runs on an event; ours runs on every prompt and adds context.

Read action / EDT

IntelliJ's rules: read code under a read lock, update UI on the event dispatch thread.