IntelliJev · local Laya evaluation · 23 September 2026
Laya runs locally. Keyword search currently retrieves more.
We tested a frozen two-pass Laya configuration on 30 held-out Koog tasks. Full-source keyword search found more of the files developers actually changed. The plugin provides both options, so a local workflow can use the stronger measured baseline.
Different provider, separate experiment. The team's Jev study uses different tasks, model settings and hardware. Its speed and quality figures cannot be compared directly with these local results.
Retrieval quality
Recall@10 is the fraction of a task's modified files present in the first ten results, averaged across tasks. The task is a real commit subject; candidate code comes from its parent, before the change.
| Ranking | Next 25 · @10 | All 30 · @10 | All 30 · @5 |
|---|---|---|---|
| Full-source BM25 | 0.555 | 0.553 | 0.410 |
| BM25 recomputed within top 60 | 0.427 | 0.428 | 0.242 |
| Fixed two-pass Laya + BM25 | 0.382 | 0.391 | 0.265 |
| Laya second pass alone | 0.103 | 0.114 | 0.047 |
The original five held-out tasks and next 25 were kept separate, then combined. A secondary paired bootstrap of the unchanged 30-task result estimated a difference of −0.162, with a 95% interval [−0.269, −0.070]. Laya improved 0 tasks, tied 20 and worsened 10. This interval describes the sample; it does not establish performance on other repositories.
What became faster
The optional exact-response cache reuses an identical task and file excerpt without new inference. It preserves the score. A changed input requires a new decision.
| Same 60 development requests | First pass | Exact repeat |
|---|---|---|
| Median round-trip | 1,303.2 ms | 2.8 ms |
| 95th percentile | 1,431.7 ms | 24.5 ms |
| New inference tokens | 21,438 | 0 |
| Cache hits | 0 / 60 | 60 / 60 |
All 120 scores matched saved uncached scores exactly, with zero errors. This is an exact-repeat latency diagnostic, not a quality improvement. The retrieval benchmark had caching disabled.
CPU cost and server reliability
The 25-task extension averaged 117.2 seconds per task for two Laya passes. Per-call median / 95th-percentile round-trip was 1,315 / 1,560 ms, without concurrent IDE or Gradle load.
A separate sequential replay of 60 saved requests compared server implementations. The persistent worker kept 33 threads and grew from 3,028 to 3,040 MiB of private memory. The original playground grew from 35 to 450 threads and from 2,969 to 3,970 MiB. Both returned identical scores without errors in this replay.
An earlier playground attempt exited after 96 calls with 47 errors during concurrent IDE and memory pressure. The failed attempt is preserved separately; retries used the same frozen protocol. The resource replay implicates thread growth, but cannot prove a single crash cause or long-term stability.
2,240 requests with cache eviction
A separate, predeclared stability run completed 2,240 local requests in 57.9 minutes with zero request errors and exact score parity. Its 150 distinct development requests exceeded the 128-entry cache: 14 cycles produced 2,100 misses and 140 expected hot-repeat hits.
Sampled private memory was 3,015→3,065 MiB, peaking at 3,088 MiB. Threads moved from 25 to 31, with a sampled peak of 33; C: free space stayed above 7.86 GiB. No IDE build overlapped the run. It finished at 04:01:41 Dubai on 23 September 2026.
The request order and 5 GiB private-memory / 3 GiB free-space stops were frozen in advance. The saved result records duration and resources; offline verification checks the frozen inputs. The run does not establish indefinite operation or improve retrieval quality. See the stability evidence and offline verification and replay instructions.
Checked inside IntelliJ
A headless IntelliJ Community 2025.2.6.2 session indexed a module-backed Koog fixture with
1,705 Kotlin files. Its real pack_context MCP tool returned ten ranked paths through
both local providers, rejected an unknown provider and limits outside 1–20, and preserved the ranked result on an exact repeat.
| Actual MCP operation | Elapsed | Local model requests | Cache hits |
|---|---|---|---|
| Full-source keywords | 1.234 s | 0 | 0 |
| Laya, 60-file keyword shortlist | 138.552 s | 93 | 0 |
| Laya, identical task and source | 1.302 s | 93 | 93 |
Cache hits are HTTP requests that skip inference. This check reused one development task; it is functional evidence, not another held-out quality result or a controlled comparison with the Python benchmark. The live keyword hook script also returned eight source paths without model calls; Claude itself was not launched.
The fixture needed an IntelliJ module/content root and read access, and startup required a second project-open handoff. Native desktop visual interaction and cloud code generation were not verified. The sanitized MCP evidence records the exact production revision, responses and limitations.
What you can use
- Fast keywords: full-source local ranking with no model or API key. Typing previews and the installed live hook use it by default.
- Laya: explicit local inference from the tool window or MCP, optional exact caching, and separate new / reused token counters.
- Source handoff: inspect results, add or drop files, and copy a bounded prompt. Cloud answer generation is a separate explicit action.
Limits and reproducibility
All held-out tasks come from one Kotlin repository and period. Short commit subjects are imperfect user requests, and modified files are imperfect relevance labels. Laya's English model has a 512-token window; this adapter uses the first 1,000 characters per file, while BM25 reads full source. The 60-file prefilter and later pool also limit possible recall.
The experiment used Laya 0.3.5, pinned English checkpoint 1c5edc17a7acd8701df6fc341c0d179f1c62c982,
PyTorch 2.14.0 on Windows CPU, and three development tasks before the frozen five + 25 held-out split.
Read the full methods, incidents and audit links. Raw per-task scores, timings, token counts, fixed manifests and SHA-256 digests are committed alongside that report.