Merge pull request #1376 from oraios/new_evals

New evals

CHANGELOG.md

@@ -14,6 +14,8 @@ Status of the `main` branch. Changes prior to the next official version change w
 * Dependencies:
   - `pywebview`: Switch back to official release (new version 6.2) #1253
 
+* Evaluations:
+  - Added new evaluations for Junie Plugin with Opus 4.6 and GLM 5.1 in Claude Code.
 
 * Language Servers:
   - Fix: clangd capability checks now tolerate valid initialize response shape differences and invalidate cached C++ document symbols when clangd/compile commands context changes #1359                                                                                                                                                                                                            

docs/04-evaluation/000_evaluation-intro.md

@@ -16,7 +16,7 @@ medium-sized Java project in the case of Codex).
 Different agents in different settings independently converge on the same core finding: Serena's strongest
 contribution is collapsing multi-file, semantically-aware operations into single atomic
 calls, while built-in tools remain the better choice for small local edits, text search,
-config files, and shell work.
+config files, and shell work. See the full [results](030_results/000_evaluation-results.md) for the scenarios mentioned above and several other scenarios.
 
 ## Approach
 
@@ -32,3 +32,16 @@ The agent evaluates itself. This is deliberate: the agent is the actual user of
 so it can judge workflow improvements from direct experience rather than through proxy
 metrics. And because the prompt defines task *categories* rather than fixed tasks, anyone
 can rerun the evaluation on their own project with their own agent.
+
+
+## Serena in JetBrains Junie
+
+A particularly notable evaluation scenario is Serena in JetBrains' Junie plugin. The latter also has access 
+to some of JetBrains' refactoring tools. At the time of writing, the only overlapping capability between Serena's tool
+and Junie's native tools is the renaming feature. Opus correctly noticed this during the evaluation and marked the
+renaming capability as equivalent. However, many symbolic and refactoring tools offered by Serena have no inbuilt equivalents,
+leading to the following summary result for Junie:
+
+> **Junie Plugin (Opus 4.6)**: Serena gives me what my built-in tools can't — the ability to move a function between modules with all imports updated atomically, trace a class hierarchy into dependencies, and safely delete symbols with usage guards — and I'd ask my owner to add it for the move-refactoring and semantic navigation capabilities alone.
+
+See the full evaluation [here](030_results/050_junie_plugin_on_tianshou.md).

docs/04-evaluation/030_results/000_evaluation-results.md

@@ -13,6 +13,8 @@ evaluation can easily be repeated with the LSP-based backend to assess its subse
 - [Claude Code (Opus 4.6) on a large Python codebase](010_cc_on_tianshou)
 - [Codex (GPT 5.4) on a Java codebase](020_codex_on_jbplugin)
 - [Copilot CLI (GPT 5.4) on a large, multi-language monorepo](030_copilot_cli_on_ente.md)
+- [GLM 5.1 in Claude Code](040_glm_on_tianshou)
+- [JetBrains Junie with Opus 4.6](050_junie_plugin_on_tianshou.md)
 
 You can run your own evaluation on a project of your choice by reusing our
 [evaluation prompt](../020_prompts/010_evaluation-prompt.md).

docs/04-evaluation/030_results/010_cc_on_tianshou.md

@@ -2,6 +2,7 @@
 :class: note
 **Generated by**: Claude Opus 4.6 (coding AI agent in Claude Code CLI)  
 **Codebase:** [Tianshou](https://github.com/thu-ml/tianshou) — a Python reinforcement learning library (~26K lines, 43 source files)  
+**Date**: 2026-04-13
 :::
 
 # Claude Code (Opus 4.6, medium)

docs/04-evaluation/030_results/020_codex_on_jbplugin.md

@@ -2,6 +2,7 @@
 :class: note
 **Generated by:** GPT-5.4 (high) in Codex  
 **Codebase:** Serena JetBrains Plugin (Java)  
+**Date**: 2026-04-13
 :::
 
 # Codex (GPT-5.4, high)

docs/04-evaluation/030_results/030_copilot_cli_on_ente.md

@@ -2,6 +2,7 @@
 :class: note
 **Generated by:** GPT-5.4 (medium) in Copilot Cli  
 **Codebase:** [ente](https://github.com/ente-io/ente) - a large monorepo in Dart, TypeScript, Go, Rust, and other languages.
+**Date**: 2026-04-14
 :::
 
 # Copilot CLI (GPT-5.4, medium)

docs/04-evaluation/030_results/040_glm_on_tianshou.md

@@ -0,0 +1,446 @@
+:::{admonition} Evaluation Result
+:class: note
+**Generated by**: GLM 5.1 (coding AI agent in Claude Code CLI)  
+**Codebase:** [Tianshou](https://github.com/thu-ml/tianshou) — a Python reinforcement learning library (~26K lines, 43 source files)  
+**Date**: 2026-04-14
+:::
+
+# Claude Code (GLM 5.1)
+
+>Serena's cross-file refactoring turned an 8-step, error-prone rename chain into a single atomic call — and after
+measuring the delta across 20 tasks, I would ask any developer I work with to install it, not because it replaces my
+built-in tools, but because it makes the hardest part of my job (changing code safely across files) suddenly feel
+reliable.
+
+---
+
+## 1. Headline: What Serena Changes
+
+**(a) Tasks where Serena adds capability.** Cross-file refactorings (rename, move) collapse from N separate read-find-edit-verify cycles into a single atomic call. Semantic queries (type hierarchy, find implementations, find references) return structured, symbol-accurate results that text search cannot produce without manual filtering. Symbol-level addressing (name paths) is stable across edits, eliminating stale-line-number rework in multi-step sessions.
+
+**(b) Tasks where Serena applies but offers no improvement.** Single-file, small-bore edits (changing an error message, renaming a local variable with `replace_all`, inserting a function when you already know the insertion context) are equally fast with both toolsets. For a 1-line tweak in a 22-line method, Edit sends ~200 chars; Serena's `replace_symbol_body` sends ~800 chars (the entire method body). The overhead reverses for full-body rewrites of large methods, where Serena sends only the new body while Edit sends old+new.
+
+**(c) Tasks outside Serena's scope.** Reading non-code files (config, TOML, docs), free-text pattern search (`FIXME`, magic constants, log strings), shell operations, and git workflows are all built-in territory. These are not Serena shortcomings — they are outside its design scope.
+
+**Verdict:** Serena adds substantial, measurable capability in two areas — cross-file refactoring (1 call vs 5–8) and semantic code queries (structured, symbol-accurate results vs flat text matches) — while providing no meaningful delta for small single-file edits or non-code tasks.
+
+---
+
+## 2. Added Value and Differences by Area
+
+### 2.1 Cross-file renaming: 1 call replaces 5–8
+
+- **What changes:** Renaming `CollectStatsBase` to `BaseCollectStats` across 4 files (10 occurrences) required 1 Serena call vs 1 Grep + 3 Reads + 4 Edits = 8 built-in calls.
+- **Frequency:** Medium. Any non-trivial rename touches 3–10 files.
+- **Value per hit:** Saves 4–7 calls and eliminates the partial-update risk of a mid-chain failure.
+- **Atomicity:** Serena's rename is all-or-nothing. Built-in chain is not — if Edit 3 of 4 fails, 2 files are updated and 2 are not.
+
+### 2.2 Symbol moving: 1 call replaces 5+
+
+- **What changes:** Moving `_nullable_slice` to another module required 1 Serena call (moved definition + updated imports in both source and target). Built-in equivalent: read function → write to target → edit source (remove + add import) → edit target (add dependency import) = 5+ calls.
+- **Frequency:** Low. Module reorganization happens infrequently.
+- **Value per hit:** Saves 4+ calls and automates the most error-prone step (getting imports right).
+- **Caveat:** The move tool created a circular import (source imports from target, target imports from source). The tool does not detect or prevent this.
+
+### 2.3 Reference finding: Symbol-accurate vs text-matched
+
+- **What changes:** Finding references to `CollectStats` with Serena returned 10 semantically categorized results (IMPORT_ELEMENT, REFERENCE_EXPRESSION, NAMED_PARAMETER, etc.) with no false positives from `LoggedCollectStats`. Grep returned 70+ lines including false positives from `LoggedCollectStats` and noise from docstrings/comments.
+- **Frequency:** High. "Who uses this?" is one of the most common codebase questions.
+- **Value per hit:** Eliminates manual false-positive filtering. However, Serena's output was 64KB (with context snippets) vs Grep's ~5KB — a tradeoff of precision vs verbosity.
+
+### 2.4 Type hierarchy: 1 call vs 2+ grep-and-parse cycles
+
+- **What changes:** Getting the full type hierarchy of `BaseCollector` (supertypes: ABC → object; subtypes: Collector → AsyncCollector) required 1 Serena call. Built-in: parse the class definition line for bases + grep for inheritors + recurse = 2–3 calls, and still no access to external library types.
+- **Frequency:** Medium. Common when navigating unfamiliar code.
+- **Value per hit:** Saves 1–2 calls and returns transitive chains that built-ins can't produce without iteration.
+
+### 2.5 Structural overview: Hierarchical vs flat
+
+- **What changes:** `get_symbols_overview(depth=1)` on a 1551-line file returned a structured hierarchy (classes → methods + attributes) in one call. Grep for `^(class |def |    def )` returned a flat 60-line list of definitions with line numbers but no attribute information or nesting.
+- **Frequency:** High. Opening any unfamiliar file.
+- **Value per hit:** Serena shows Protocol fields, class attributes, and method grouping. Grep shows line numbers for navigation but requires more work to understand structure.
+
+### 2.6 Method body retrieval: Targeted read vs range read
+
+- **What changes:** Reading `Collector._collect` (330+ lines) required 1 Serena `find_symbol` call using the name path. Built-in: need to know the line range (773–1103) from a prior Grep, then `Read(offset=773, limit=330)`.
+- **Frequency:** High. "Show me this method" is routine.
+- **Value per hit:** Serena uses stable addressing (name path). Built-in uses ephemeral line numbers that go stale after edits. The practical difference is small when you just read once, but compounds in edit-then-read-again workflows.
+
+### 2.7 Single-file rename: No meaningful difference
+
+- **What changes:** Renaming `_nullable_slice` (4 occurrences in one file): Serena rename = 1 call. Edit `replace_all` = 1 call. Identical results.
+- **Frequency:** High.
+- **Value per hit:** Zero. Both tools handle this equally well.
+
+### 2.8 Small edits (1–3 lines): Edit is more token-efficient
+
+- **What changes:** Changing one error message in a 22-line method: Edit sends ~200 chars (old + new string). Serena `replace_symbol_body` sends ~800 chars (entire method body).
+- **Frequency:** Very high.
+- **Value per hit:** Edit saves ~600 chars of payload per small edit. This reverses for full-body rewrites of 50+ line methods.
+
+### 2.9 Insertion: Stable address vs text anchor
+
+- **What changes:** Inserting a new method after `refresh_all_sequence_stats`: Serena used 1 `insert_after_symbol` call with a name path (no line number needed). Edit used 1 Read + 1 Edit with a text anchor (surrounding context for uniqueness).
+- **Frequency:** Medium.
+- **Value per hit:** Saves 1 Read call. Both produce identical results.
+
+**Verdict:** Serena's value concentrates in cross-file operations and semantic queries. For single-file text edits, the built-ins are equally capable and often more token-efficient.
+
+---
+
+## 3. Detailed Evidence, Grouped by Capability
+
+### 3.1 Codebase Understanding
+
+#### Task 1: Repo overview
+Both toolsets use the same approach (`ls`, `find`, directory listing). No Serena advantage here.
+
+#### Task 2: Structural overview of a large file (`collector.py`, 1551 lines)
+
+| Step | Serena | Built-in |
+|------|--------|----------|
+| Call | `get_symbols_overview(depth=1)` | `Grep "^(class \|def \|    def )"` |
+| Result | Hierarchical: 14 classes, their methods, attributes, and module-level functions | Flat list of 60 class/function definitions with line numbers |
+| Output size | ~1.5KB structured JSON | ~3KB text |
+| Next step | `find_symbol("Collector/_collect", include_body=True)` — direct | `Read(offset=773, limit=330)` — needs prior knowledge of line range |
+
+**Serena advantage:** Shows attributes (e.g., `CollectStats.collect_time`, `CollectStats.returns`) that Grep cannot see. Hierarchical nesting makes the file's architecture immediately clear.
+
+**Built-in advantage:** Line numbers enable direct `Read` calls. Flat output is compact.
+
+**Verdict:** Serena provides strictly more structural information in one call. The gap widens for files with deeply nested classes or dataclass fields.
+
+#### Task 3: Retrieve a specific method body
+
+| Step | Serena | Built-in |
+|------|--------|----------|
+| Prerequisite | Name path known (`Collector/_collect`) | Line number known from prior Grep (773) |
+| Call | `find_symbol(name_path="Collector/_collect", include_body=True)` | `Read(offset=773, limit=330)` |
+| Payload sent | ~50 chars (name + path) | ~30 chars (offset + limit) |
+| Payload received | Exact method body (~330 lines) | Lines 773–1102 (~330 lines) |
+| Correctness | Always exact | Must know/guess the correct limit |
+
+**Verdict:** Functionally equivalent when line numbers are known. Serena's name-path addressing degrades gracefully across edits; line numbers do not.
+
+#### Task 4: Find all references to `CollectStats`
+
+| Metric | Serena `find_referencing_symbols` | Built-in `Grep` |
+|--------|----------------------------------|-----------------|
+| Calls | 1 | 1 |
+| Output size | 64KB (with context snippets) | ~5KB (70 lines) |
+| False positives | 0 (excludes `LoggedCollectStats`) | 5+ lines from `LoggedCollectStats` |
+| Noise (comments/docs) | Some (docstrings categorized) | Significant (docstrings, comments matched) |
+| Semantic categories | Yes (IMPORT, PARAMETER, DECLARATION, REFERENCE) | No |
+
+**Serena advantage:** Zero false positives. Semantic categorization. Shows import paths and parameter usage separately from code references.
+
+**Built-in advantage:** 10x smaller output. Faster to scan visually.
+
+**Verdict:** Serena is more precise but more verbose. For "who uses this in code?" both work; for "rename this safely" Serena's precision is necessary.
+
+#### Task 5: Type hierarchy of `BaseCollector`
+
+| Metric | Serena `type_hierarchy` | Built-in (Grep chain) |
+|--------|------------------------|----------------------|
+| Calls | 1 | 2–3 (grep subclasses, parse superclass, recurse) |
+| Result | `ABC → object` (super), `Collector → AsyncCollector` (sub) | Partial — direct sub/supertypes only, no transitive chain |
+| External deps | Shows `ABC` from `abc.pyi` | Cannot access |
+
+**Verdict:** Serena returns complete, transitive hierarchy including external library types in one call. Built-in approach requires iteration and cannot inspect external deps.
+
+#### Task 6: External dependency symbol lookup
+
+Serena can read external dependency symbols IF you have the path from a prior tool result (e.g., `<ext:abc.pyi|16198efc>` from `type_hierarchy`). Direct search with `search_deps=True` returned empty results for `numpy.array` and `torch.Tensor`. The JetBrains IDE indexing is limited to what the language server has resolved.
+
+Built-in: Can `Read` site-packages files if you know the path, but discovery is manual.
+
+**Verdict:** Minor Serena advantage — external symbols are accessible through tool chains but not through direct search. Neither toolset makes this easy.
+
+### 3.2 Single-File Edits
+
+#### Task 7a: Small tweak (1-line change in 22-line method)
+
+| Metric | Edit | Serena `replace_symbol_body` |
+|--------|------|------------------------------|
+| Prerequisite | 1 Read (6 lines of context) | 1 `find_symbol` (gets full body) |
+| Payload sent | ~200 chars (old + new string) | ~800 chars (full method body) |
+| Payload received | Success message | "OK" |
+| Total payload | ~200 chars edit + ~300 chars read = ~500 | ~800 chars edit + ~800 chars read = ~1600 |
+
+**Verdict:** Edit is 3x more token-efficient for small tweaks inside methods.
+
+#### Task 7b: Medium rewrite (~6 line changes in 20-line method)
+
+| Metric | Edit | Serena `replace_symbol_body` |
+|--------|------|------------------------------|
+| Payload sent | ~700 chars (old + new, full method) | ~500 chars (new body only) |
+| Prerequisite read | ~500 chars | ~500 chars (from prior `find_symbol`) |
+
+**Verdict:** Roughly equal. For medium rewrites, payloads converge.
+
+#### Task 7c: Large rewrite (full body of 55+ line method)
+
+For a full-body rewrite of a 55-line method:
+- Edit: old (~55 lines) + new (~55 lines) = ~110 lines sent
+- Serena: new body only (~55 lines) sent
+
+**Verdict:** Serena is ~2x more token-efficient for full-body rewrites. The advantage grows linearly with method size.
+
+#### Task 8: Insert a new function after an existing one
+
+| Step | Serena | Built-in |
+|------|--------|----------|
+| 1 | `find_symbol("refresh_all_sequence_stats")` to confirm target | `Read(offset=250, limit=10)` to find insertion point |
+| 2 | `insert_after_symbol(name_path, body)` | `Edit(old_string=anchor, new_string=anchor+new_fn)` |
+| Total calls | 2 | 2 |
+| Payload sent | New function body only (~300 chars) | Anchor context + new function (~400 chars) |
+
+**Verdict:** No meaningful difference. Both require 2 calls and produce identical results.
+
+#### Task 9: Rename a private helper (single-file, 4 occurrences)
+
+| Metric | Serena `rename` | Edit `replace_all` |
+|--------|-----------------|-------------------|
+| Calls | 1 | 1 |
+| Prerequisites | None | Must have Read the file first |
+| Result | All 4 occurrences renamed | All 4 occurrences renamed |
+
+**Verdict:** Functionally identical. Both are 1 call. Edit's Read-first requirement is usually satisfied from prior exploration.
+
+### 3.3 Multi-File Changes
+
+#### Task 10: Cross-file rename (`CollectStatsBase` → `BaseCollectStats`, 4 files, 10 occurrences)
+
+| Step | Serena | Built-in |
+|------|--------|----------|
+| Find references | Automatic | 1 Grep call |
+| Read files | Not needed | 3 Read calls (Edit's prerequisite) |
+| Apply edits | 1 rename call (atomic) | 4 Edit calls (one per file) |
+| Verify | Return: "Success" | Manual (4 success messages) |
+| **Total calls** | **1** | **8** (1 grep + 3 reads + 4 edits) |
+
+**Verdict:** Serena converts an 8-call manual pipeline into 1 atomic operation. This is the single largest efficiency gain observed.
+
+#### Task 11: Move symbol to another module
+
+Serena's `move` tool:
+1. Moved `_nullable_slice` from `collector.py` to `converter.py`
+2. Added import in source file: `from tianshou.data.utils.converter import _nullable_slice`
+3. Added dependency import in target: `from tianshou.data.collector import _TArrLike`
+4. Removed definition from source
+
+Built-in equivalent: Read function body → Write to target → Edit source (remove definition + add import) → Edit target (add dependency import) = 5+ calls.
+
+**Issue:** The move created a circular import (source ↔ target). Serena does not detect or prevent this.
+
+**Verdict:** Serena automates the most tedious part (import management) but doesn't guard against circular dependencies. Saves 4+ calls at the cost of needing manual circular-import review.
+
+#### Task 12: Move file (`segtree.py` to parent directory)
+
+Serena's `move` tool:
+1. Moved the file
+2. Updated the one direct import in `__init__.py` (`tianshou.data.utils.segtree` → `tianshou.data.segtree`)
+3. Other files (`prio.py`, tests) imported via re-export and needed no changes
+
+Built-in equivalent: `git mv` + grep for old import path + edit each file = 3+ calls.
+
+**Verdict:** Serena saves 1–2 calls and automatically discovers which imports need updating.
+
+#### Task 12 (safe delete)
+
+Serena's `safe_delete` correctly refused to delete `_HACKY_create_info_batch` because it has a usage at line 730. The `propagate=true` mode (delete symbol + all call sites) failed for all tested symbols in this codebase.
+
+**Verdict:** The usage-check is valuable (saves you from deleting a used symbol). The propagation feature was non-functional for the tested Python symbols.
+
+#### Task 13: Inline
+
+Serena's `inline_symbol` failed for all tested symbols (`_nullable_slice`, `BaseCollector/env_num`). The tool appears to have limited Python support for inlining.
+
+**Verdict:** No successful inline demonstrated. Built-in manual inlining remains the only option.
+
+### 3.4 Reliability and Correctness
+
+#### Task 14: Scope precision
+
+Serena distinguishes `BaseCollector/_collect`, `Collector/_collect`, and `AsyncCollector/_collect` by name path. Grep for `def _collect` matches all three — manual filtering by class is required.
+
+**Verdict:** Serena's name-path addressing eliminates ambiguity that text search cannot resolve.
+
+#### Task 15: Atomicity
+
+Serena's cross-file rename is atomic: 4 files updated in 1 call, all-or-nothing. Built-in: 4 separate Edit calls — if call 3 fails, 2 files are updated and 2 are not.
+
+**Verdict:** Serena provides atomicity for cross-file operations. Built-in chains are inherently non-atomic.
+
+#### Task 16: Success signals
+
+Both return clear success/failure indicators. No meaningful difference.
+
+**Verdict:** Equal.
+
+### 3.5 Workflow Effects
+
+#### Task 17: Chain three edits in one file
+
+Edit with text matching: 3 sequential calls, no re-reads needed between them. Text anchors are immune to line-number shifts from prior edits.
+
+Serena `replace_symbol_body`: 3 sequential calls, no re-reads needed. Name-path addressing is also immune to line-number shifts.
+
+**Verdict:** No meaningful difference for chained single-file edits.
+
+#### Task 18: Multi-step exploration across edits
+
+Serena's name-path results from exploration remain valid after edits. Built-in line numbers go stale, but Edit uses text matching (not line numbers), so the practical impact is limited to `Read` calls that need updated offsets.
+
+**Verdict:** Minor Serena advantage. Name-path stability eliminates the need to re-scan after edits.
+
+### 3.6 Non-Interesting Tasks
+
+#### Task 19: Read non-code file
+
+Serena tools don't apply. `Read` is the correct tool.
+
+#### Task 20: Free-text pattern search
+
+Searching for `FIXME|HACK|TODO` across the codebase is a text search. Serena's semantic tools don't target this. `Grep` is the correct tool.
+
+**Verdict:** These tasks are firmly built-in territory. They represent an estimated 20–30% of daily coding work (reading configs, searching for strings, shell operations, git workflows).
+
+---
+
+## 4. Token-Efficiency Analysis
+
+### Payload differences across edit sizes
+
+| Edit type | Edit payload | Serena payload | Winner |
+|-----------|-------------|---------------|--------|
+| 1-line tweak in 22-line method | ~200 chars | ~800 chars | Edit (4x) |
+| 6-line change in 20-line method | ~700 chars | ~500 chars | Roughly equal |
+| Full rewrite of 55-line method | ~2200 chars | ~1100 chars | Serena (2x) |
+| Full rewrite of 330-line method | ~13,000 chars | ~6,500 chars | Serena (2x) |
+
+### Forced reads
+
+- Edit requires reading a file before editing it. This adds ~300–2000 chars per file.
+- Serena does not require reading before editing (name-path addressing).
+- For single-file edits where you already read the file, this is neutral.
+- For cross-file operations on files you haven't read, Serena saves 3–4 forced reads.
+
+### Stable vs ephemeral addressing
+
+- Serena: name paths (`Collector/_collect`) are stable across edits. Results from exploration remain valid.
+- Built-in: line numbers are ephemeral. `Read` results go stale after edits. `Edit` uses text matching, which is stable.
+- Practical impact: Low for one-shot edits, medium for edit-then-read-again workflows.
+
+**Verdict:** Edit wins for small tweaks (4x more token-efficient). Serena wins for full-body rewrites (2x more efficient) and cross-file operations (eliminates forced reads). The crossover point is approximately 50% of the method body changing — below that, Edit is more efficient; above that, Serena is.
+
+---
+
+## 5. Reliability and Correctness (Under Correct Use)
+
+### Precision of matching
+
+- Serena: Symbol-accurate. `find_referencing_symbols(CollectStats)` excludes `LoggedCollectStats`. No false positives observed.
+- Grep: Text-matched. `CollectStats` matches `LoggedCollectStats`, `CollectStatsBase`, and docstring references. Requires manual filtering.
+- Edit: Text-matched. `replace_all` replaces exact string matches. For unique symbol names, this is reliable. For common strings, it can over-match.
+
+### Scope disambiguation
+
+- Serena: `Collector/_collect` vs `AsyncCollector/_collect` — correctly distinguished by class-scoped name path.
+- Built-in: `def _collect` matches all implementations. Must manually verify class context.
+
+### Atomicity
+
+- Serena cross-file operations: Atomic. Single call, all-or-nothing.
+- Built-in multi-file chains: Non-atomic. Partial state possible if one call fails.
+
+### External dependency lookup
+
+- Serena: Can read external stubs (e.g., `abc.pyi`) through paths returned by other tools. Direct search (`search_deps=True`) returned empty for `torch.Tensor` and `numpy.array`. Limited to what JetBrains has indexed.
+- Built-in: Can `Read` site-packages files if path is known. No semantic indexing.
+
+**Verdict:** Serena provides strictly more precise semantic matching and atomic cross-file operations. External dependency lookup is limited in both toolsets.
+
+---
+
+## 6. Workflow Effects Across a Session
+
+### Where advantages compound
+
+1. **Explore → edit → re-explore cycle:** Serena's name-path results survive edits. In a long session making multiple changes, this saves re-scanning after each edit. The built-in's text matching also survives edits (Edit uses text, not line numbers), so the practical gap is smaller than it appears.
+
+2. **Cross-file refactoring chains:** Rename a class, then move it, then update all references — each Serena call is atomic and builds on the previous result. With built-ins, each step requires finding all sites, reading files, and editing — the manual equivalent of what Serena automates.
+
+### Where advantages diminish
+
+1. **Repeated small edits in one file:** Edit's text matching is equally stable and more token-efficient for small changes. No Serena advantage.
+
+2. **Exploration without editing:** Both toolsets provide usable results. Serena's are more structured but more verbose.
+
+3. **Non-Python files:** Serena's JetBrains backend provides no value for config files, shell scripts, markdown, or notebooks.
+
+**Verdict:** Serena's advantages compound in multi-step cross-file refactoring sessions. They do not compound for single-file iterative editing or non-code work.
+
+---
+
+## 7. Unique Capabilities
+
+1. **Atomic cross-file rename/move** — No built-in equivalent. The closest manual process is a grep-find-edit chain that is non-atomic and error-prone. Frequency: medium. Impact: high (eliminates partial-update risk).
+
+2. **Semantic reference finding with categorization** — `find_referencing_symbols` returns zero-false-positive results categorized by usage type (import, parameter, declaration, reference). Built-in Grep cannot distinguish these. Frequency: high. Impact: medium (saves manual filtering).
+
+3. **Type hierarchy traversal** — Returns transitive super/subtype chains including external library types in one call. Built-in requires iteration and cannot reach external deps. Frequency: medium. Impact: medium.
+
+4. **Symbol-scoped body retrieval** — Read a specific method by name path without reading the surrounding file. Built-in `Read` requires line-range knowledge. Frequency: high. Impact: low (both require 1 call, difference is stable vs ephemeral addressing).
+
+**Verdict:** Four unique capabilities, all semantic-code operations. The most impactful is atomic cross-file refactoring. None of these have practical built-in equivalents.
+
+---
+
+## 8. Tasks Outside Serena's Scope (Built-In Only)
+
+| Task | Tool | Frequency | Share of daily work |
+|------|------|-----------|-------------------|
+| Read config/TOML/yaml files | `Read` | High | |
+| Free-text search (log strings, TODOs, URLs) | `Grep` | High | |
+| File discovery by name pattern | `Glob` | Medium | |
+| Shell commands (git, pip, pytest) | `Bash` | High | |
+| Write new files from scratch | `Write` | Medium | |
+| Read images, notebooks, PDFs | `Read` | Low | |
+
+Estimated share of daily work covered by built-in-only tasks: **20–30%**. The remaining 70–80% involves reading, editing, and navigating code where Serena's semantic tools are applicable.
+
+**Verdict:** Serena covers the code-editing and code-navigation portions of a session. Config reading, text search, and shell operations remain built-in territory.
+
+---
+
+## 9. Practical Usage Rule
+
+**Use Serena for:** Any cross-file refactoring (rename, move), any "who uses this?" query, any type-hierarchy navigation, any full-method-body replacement, and any situation where you need symbol-accurate results without false positives.
+
+**Use built-ins for:** Small edits (1–3 lines) inside methods, free-text search, reading non-code files, file discovery, and shell operations.
+
+**Hybrid pattern (most efficient):** Use Serena to explore (overview, find symbol, find references) and Edit for small targeted changes. Use Serena for any cross-file refactoring. Use `Read`/`Grep`/`Glob` for non-code tasks and text search. This combination captures the strengths of both toolsets.
+
+**Verdict:** The optimal workflow uses Serena's semantic tools for code navigation and cross-file refactoring, and built-in Edit for small single-file changes. The two toolsets are complementary — Serena handles the structured code operations, built-ins handle the text and system operations.
+
+---
+
+## Appendix: Call Count Summary
+
+| Task | Serena calls | Built-in calls | Delta |
+|------|-------------|----------------|-------|
+| Structural overview (1 file) | 1 | 1 | 0 |
+| Method body retrieval | 1 | 1 | 0 |
+| Find references (1 symbol) | 1 | 1 | 0 (but Serena has 0 false positives vs Grep's 5+) |
+| Type hierarchy | 1 | 2–3 | −1 to −2 |
+| Small edit (1 line in 22-line method) | 1 (+1 read) | 1 (+1 read) | 0 |
+| Medium edit (6 lines in 20-line method) | 1 | 1 | 0 |
+| Insert new method | 1 (+1 confirm) | 1 (+1 read) | 0 |
+| Single-file rename (4 occurrences) | 1 | 1 | 0 |
+| **Cross-file rename (4 files, 10 occurrences)** | **1** | **8** | **−7** |
+| **Move symbol to another module** | **1** | **5+** | **−4+** |
+| **Move file + update imports** | **1** | **3+** | **−2+** |
+| Safe delete (usage check) | 1 | 1 grep | 0 |
+
+Largest single-task delta: cross-file rename saves 7 calls and provides atomicity.

docs/04-evaluation/030_results/050_junie_plugin_on_tianshou.md

@@ -0,0 +1,345 @@
+:::{admonition} Evaluation Result
+:class: note
+**Generated by**: Claude Opus 4.6 (coding AI agent in JetBrains Junie Plugin)  
+**Codebase:** [Tianshou](https://github.com/thu-ml/tianshou) — a Python reinforcement learning library (~26K lines, 43 source files)  
+**Date**: 2026-04-17
+:::
+
+# JetBrains Junie Plugin (Opus 4.6)
+
+> Serena gives me what my built-in tools can't — the ability to move a function between modules with all imports updated atomically, trace a class hierarchy into dependencies, and safely delete symbols with usage guards — and I'd ask my owner to add it for the move-refactoring and semantic navigation capabilities alone.
+
+---
+
+## 1. Headline: What Serena Changes
+
+Serena adds three categories of capability on top of built-in tools:
+
+**(a) Tasks where Serena adds capability:**
+- **Move symbol between modules** (with automatic import updates) — no built-in equivalent.
+- **Move file/package** (with automatic import updates) — no built-in equivalent.
+- **Type hierarchy traversal** — transitive supertype/subtype chains in 1 call, including into external dependencies. No built-in equivalent.
+- **External dependency symbol lookup** — search and inspect symbols in installed packages (site-packages) without manual path discovery. Built-ins cannot do this.
+- **Safe delete with usage checking** — refuses deletion if usages exist, reports exact locations. Built-ins require a manual search-then-decide workflow.
+- **Propagated delete** — delete a symbol and automatically remove all call sites. No built-in equivalent.
+- **Inline symbol** — replace all call sites with the function body. No built-in equivalent.
+- **Scope-precise symbol targeting** — name paths like `AsyncCollector/_collect` disambiguate overrides that text search cannot distinguish.
+
+**(b) Tasks where Serena applies but offers no meaningful improvement:**
+- **Cross-file rename** — both Serena (`rename`) and built-in (`rename_element`) perform semantic, cross-file renames in 1 call. Functionally equivalent.
+- **Single-file rename** — same as above; both are 1-call semantic renames.
+- **Structural overview of a single file** — both toolsets produce comparable results in 1 call.
+- **Small edits (1-3 lines)** — Serena's `replace_symbol_body` must send the entire method body; built-in `search_replace` sends only the changed line. Built-in is more token-efficient here.
+
+**(c) Tasks outside Serena's scope (built-in only):**
+- Reading non-code files (configs, docs, changelogs)
+- Free-text search across the repo (log strings, URLs, magic constants)
+- Terminal commands, test execution, git operations
+- Creating new files from scratch
+- Repository-level directory listing
+
+**Verdict:** 
+Serena's primary contribution is move-refactoring (symbol and file moves with import updates), semantic code navigation (type hierarchy, dependency lookup), and safe/propagated delete — capabilities with no built-in equivalent.
+
+---
+
+## 2. Added Value and Differences by Area
+
+- **Move symbol/file with import updates (positive).** Frequency: occasional in refactoring work. Value per hit: saves ~7 calls and eliminates manual import tracking. Moving `get_stddev_from_dist` between modules: 1 Serena call vs. ~7 built-in calls. No built-in equivalent.
+
+- **Type hierarchy and reference finding (positive).** Frequency: a few times per exploration session. Value per hit: saves 2-4 search-and-read cycles and provides information (transitive supertypes, external dependency chains) that built-ins cannot produce at all. `BaseCollector`'s full hierarchy (ABC→object up, Collector→AsyncCollector down) returned in 1 call.
+
+- **Targeted symbol retrieval by name path (positive).** Frequency: many times per session. Value per hit: saves 1 prerequisite read (no line-number lookup needed). `find_symbol` with `include_body=true` returns exact method body in 1 call vs. 2 calls (structure + open).
+
+- **Safe delete with usage guard (positive).** Frequency: occasional. Value per hit: eliminates risk of orphaned references. Serena refused to delete `_dict_of_arr_to_arr_of_dicts` and reported 2 exact usages. Built-in: requires manual search first.
+
+- **Cross-file rename (neutral).** Frequency: several times per session. Value per hit: zero — both Serena's `rename` and built-in `rename_element` perform semantic cross-file renames in 1 call with equivalent results.
+
+- **Single-file edits (neutral to slightly negative for small edits).** Frequency: very high. Value per hit: for small edits, Serena's `replace_symbol_body` sends more tokens (entire method body) than `search_replace` (just the changed lines). For full method rewrites, comparable. For inserts, Serena's `insert_after_symbol` saves 1 prerequisite read.
+
+- **Structural overview (neutral).** Frequency: a few times per session. Value per hit: both toolsets return comparable information in 1 call.
+
+**Verdict:** Serena's value concentrates in move-refactoring and semantic navigation; rename is matched by the built-in `rename_element`.
+
+---
+
+## 3. Detailed Evidence, Grouped by Capability
+
+### 3.1 Structural Overview (Task 2)
+
+**Attempted:** Get structural overview of `collector.py` (1552 lines, 15+ classes).
+
+| Axis | Serena (`get_symbols_overview`) | Built-in (`get_file_structure`) |
+|------|------|------|
+| Calls | 1 | 1 |
+| Output | JSON tree: class names, method names, field names | Flat list: class/method names with line ranges and signatures |
+| Unique info | Field/attribute names | Line numbers, full parameter signatures |
+| Follow-up to read a method | `find_symbol` by name path (1 call) | `open` at line number (1 call) |
+
+**Verdict:** Functionally equivalent; each includes information the other omits. No meaningful delta.
+
+### 3.2 Targeted Symbol Retrieval (Task 3)
+
+**Attempted:** Retrieve body of `CollectStats/refresh_all_sequence_stats` without reading surrounding file.
+
+| Axis | Serena | Built-in |
+|------|------|------|
+| Calls | 1 (`find_symbol` with `include_body`) | 2 (`get_file_structure` + `open`) |
+| Prerequisite | None (name path is stable) | Must know line number |
+| Output payload | 4 lines (exact method body) | 100 lines (open window) |
+
+**Verdict:** Serena saves 1 call and returns a smaller, more precise payload. Minor but consistent advantage.
+
+### 3.3 Reference Finding (Task 4)
+
+**Attempted:** Find all references to `CollectStats` across the codebase.
+
+| Axis | Serena (`find_referencing_symbols`) | Built-in (`search_project`) |
+|------|------|------|
+| Calls | 1 | 1 |
+| Output | Summarized narrative: grouped by file, annotated with context (imports, instantiations, type hints) | Raw list: 100+ text matches, includes docs, comments, strings |
+| Precision | Code references only | All text mentions |
+| Recall | Code references only | Everything including non-code |
+
+**Verdict:** Serena provides higher precision (code-only references with semantic context). Built-in provides higher recall (includes docs, comments). Different tools for different questions.
+
+### 3.4 Type Hierarchy (Task 5)
+
+**Attempted:** Full type hierarchy of `BaseCollector` — supertypes and subtypes, transitively.
+
+| Axis | Serena (`type_hierarchy`) | Built-in |
+|------|------|------|
+| Calls | 1 | 3+ (grep for class declarations, read each, trace manually) |
+| Result | `BaseCollector → ABC → object` (up), `→ Collector → AsyncCollector` (down) | Partial: can find direct subclasses by grep, but supertypes of supertypes require reading external files |
+| External deps | Included (ABC from `abc` module, object from `builtins`) | Not accessible |
+
+**Verdict:** Unique capability. No practical built-in equivalent for transitive hierarchy, especially into dependencies.
+
+### 3.5 External Dependency Lookup (Task 6)
+
+**Attempted:** Find `Distribution` class from `torch.distributions`.
+
+| Axis | Serena (`find_symbol` with `search_deps`) | Built-in |
+|------|------|------|
+| Calls | 1 | Not possible without manual site-packages navigation |
+| Result | Found 41 `Distribution` classes across all installed packages with ext-identifiers | N/A |
+| Follow-up | Can use ext-identifier to read body/info | Would need to find virtualenv path, navigate to package, read file |
+
+**Verdict:** Unique capability. Built-ins have no access to dependency source code without manual path discovery.
+
+### 3.6 Small Edit — 1 Line Change (Task 7a)
+
+**Attempted:** Change error message in `_validate_buffer` (21-line method).
+
+| Axis | Serena | Built-in |
+|------|------|------|
+| Calls | 2 (`find_symbol` + `replace_symbol_body`) | 1 (`search_replace`) |
+| Input payload | ~21 lines (full method body) | ~1 line (search) + ~1 line (replace) |
+| Prerequisite reads | 1 (find_symbol to get current body) | 0 (if search string is known) |
+
+**Verdict:** Built-in is more efficient for small, targeted edits. Serena's method-granularity addressing forces sending the entire body.
+
+### 3.7 Medium Rewrite — ~20 Lines (Task 7b)
+
+**Attempted:** Rewrite `update_at_step_batch` (20 lines), changing variable names and adding logging.
+
+| Axis | Serena | Built-in |
+|------|------|------|
+| Calls | 2 (`find_symbol` + `replace_symbol_body`) | 1 (`search_replace` with old body → new body) |
+| Input payload | ~20 lines (new body) | ~40 lines (old + new body) |
+| Prerequisite reads | 1 (find_symbol) | 1 (open to see current code) |
+
+**Verdict:** Comparable. Serena sends less payload (new body only vs. old+new), but requires a prerequisite find_symbol call.
+
+### 3.8 Insert New Method (Task 8)
+
+**Attempted:** Insert `summary_string` method after `refresh_all_sequence_stats` in `CollectStats`.
+
+| Axis | Serena (`insert_after_symbol`) | Built-in (`search_replace`) |
+|------|------|------|
+| Calls | 1 | 1-2 (need to find anchor text, then insert) |
+| Addressing | By name path (stable) | By text anchor or line number (fragile) |
+| Prerequisite | None | May need `open` to find insertion point |
+
+**Verdict:** Serena's stable addressing is a minor advantage — eliminates the need to find an anchor.
+
+### 3.9 Single-File Rename (Task 9)
+
+**Attempted:** Rename `_nullable_slice` → `_slice_if_not_none` (5 occurrences in 1 file).
+
+| Axis | Serena (`rename`) | Built-in (`rename_element`) |
+|------|------|------|
+| Calls | 1 | 1 |
+| Scope | Semantic (only code references) | Semantic (only code references) |
+
+**Verdict:** Functionally equivalent. Both perform semantic renames in 1 call.
+
+### 3.10 Cross-File Rename (Task 10)
+
+**Attempted:** Rename `CollectStatsBase` → `CollectStatsFoundation` (used in 4 files, 10 occurrences including imports and `__all__`).
+
+| Axis | Serena (`rename`) | Built-in (`rename_element`) |
+|------|------|------|
+| Calls | 1 | 1 |
+| Atomicity | All-or-nothing | All-or-nothing |
+| Import handling | Automatic | Automatic |
+| Result | 4 files, 10 replacements | 4 files, 10 replacements |
+
+**Verdict:** No delta. Both toolsets provide atomic, semantic cross-file rename in a single call.
+
+### 3.11 Move Symbol Between Modules (Task 11)
+
+**Attempted:** Move `get_stddev_from_dist` from `collector.py` to `stats.py`.
+
+| Axis | Serena (`move`) | Built-in |
+|------|------|------|
+| Calls | 1 | ~7 (read source, copy to target, delete from source, find imports, update each import, verify) |
+| Files modified | 3 (source, target, test file) | Same 3, but manually |
+| Import updates | Automatic | Manual |
+| Atomicity | Atomic | Non-atomic |
+
+**Verdict:** Unique capability at this level of automation. No practical built-in equivalent at comparable effort. This is now the highest-value delta in the evaluation.
+
+### 3.12 Move File (Task 12)
+
+**Attempted:** Move `segtree.py` from `tianshou/data/utils/` to `tianshou/utils/`.
+
+| Axis | Serena (`move`) | Built-in |
+|------|------|------|
+| Calls | 1 | 3+ (bash mv, find imports, update each) |
+| Import updates | Automatic (updated `__init__.py`) | Manual |
+
+**Verdict:** Serena advantage — single call with automatic import updates.
+
+### 3.13 Safe Delete (Task 13)
+
+**Attempted:** Delete `_dict_of_arr_to_arr_of_dicts` (has 2 usages).
+
+| Axis | Serena (`safe_delete`) | Built-in |
+|------|------|------|
+| Calls | 1 (refused, reported usages) | 2 (search_project + manual decision) |
+| Safety | Refuses deletion, reports exact usages with file:line and enclosing function | Must manually verify search results |
+
+**Verdict:** Serena provides a safety guard with precise diagnostics. Minor but useful advantage.
+
+### 3.14 Scope Precision (Task 14)
+
+**Attempted:** Target `AsyncCollector/_collect` specifically (3 classes define `_collect`).
+
+Serena's name path `AsyncCollector/_collect` unambiguously selects the override. A text search for `_collect` matches all three definitions plus dozens of call sites. Built-in `get_file_structure` can disambiguate by line number, but that requires a prerequisite read and the line number goes stale after edits.
+
+**Verdict:** Serena's name-path addressing provides persistent, unambiguous symbol targeting. Meaningful advantage when multiple overrides exist.
+
+### 3.15 Chained Edits (Task 17)
+
+**Attempted:** Three sequential `replace_symbol_body` calls on different methods in the same file.
+
+All three succeeded without re-reading the file. Name paths remained stable across edits. Built-in `search_replace` also chains without re-reading (text anchors remain valid if edits are in different methods), but line numbers from `open` go stale after each edit.
+
+**Verdict:** Both toolsets chain well. Serena's name-path stability is a minor advantage over line-number-based addressing.
+
+---
+
+## 4. Token-Efficiency Analysis
+
+| Edit size | Serena payload | Built-in payload | Winner |
+|-----------|---------------|-----------------|--------|
+| Small (1-3 lines in 20-line method) | ~20 lines (full body) | ~2 lines (search+replace) | Built-in (10x less) |
+| Medium (rewrite ~20 lines) | ~20 lines (new body) + prerequisite find | ~40 lines (old+new) | Comparable |
+| Large (rewrite 50+ lines) | ~50+ lines (new body) | ~100+ lines (old+new) | Serena (2x less, no old body needed) |
+| Insert | ~N lines (new code only) | ~N lines + anchor context | Comparable |
+| Cross-file rename | ~1 line (name + new name) | ~1 line (name + new name) | Tie (both semantic) |
+| Move symbol | ~1 line (source + target) | ~7 calls with full bodies | Serena (>>10x less) |
+
+**Forced reads:** Serena's `replace_symbol_body` requires a `find_symbol` call to get the current body before editing. Built-in `search_replace` needs no prerequisite if the search text is known, but often requires `open` to discover it.
+
+**Stable vs. ephemeral addressing:** Serena's name paths survive edits; built-in line numbers do not. This matters in multi-edit sessions — Serena never needs to re-read for addressing purposes, while built-in may need to re-open after edits that shift line numbers.
+
+**Verdict:** For single-file small edits, built-ins are more token-efficient. For move-refactoring, Serena is substantially more efficient. For rename, both are equivalent. The crossover point for edits is roughly at method-level rewrites.
+
+---
+
+## 5. Reliability & Correctness (Under Correct Use)
+
+- **Precision of matching:** Serena matches by semantic identity (name path in the symbol tree). Built-in `search_replace` matches by text (can over-match). However, built-in `rename_element` also matches semantically. For rename operations, precision is equivalent.
+
+- **Scope disambiguation:** Serena's `AsyncCollector/_collect` is unambiguous. Built-in text search for `def _collect` returns 3 matches in collector.py alone. Disambiguation requires line numbers or surrounding context. Built-in `rename_element` also requires a file path and line number to disambiguate.
+
+- **Atomicity:** Both Serena's and built-in's rename operations are atomic. Serena's move operations are also atomic; built-ins have no equivalent move, so the comparison is moot. For sequential text edits (`search_replace`), partial failure is possible.
+
+- **Semantic queries vs. text search:** Serena's `find_referencing_symbols` returns only code references with semantic context. Built-in `search_project` returns all text matches. Neither is strictly better — they answer different questions.
+
+- **External dependency lookup:** Serena can search and inspect symbols in installed packages via `search_deps=true`. Built-ins have no access to dependency source without manual virtualenv/site-packages navigation. This depends on the IDE having indexed the project's interpreter.
+
+**Verdict:** For rename, correctness guarantees are equivalent between toolsets. Serena's unique correctness advantage is in move-refactoring atomicity and external dependency access.
+
+---
+
+## 6. Workflow Effects Across a Session
+
+- **Multi-edit stability:** Serena's name-path addressing remains valid across edits within a session. Built-in line numbers go stale after any edit that changes line counts. In a session with 10+ edits to the same file, Serena eliminates ~5-10 re-read calls that built-ins would need to refresh line numbers.
+
+- **Exploration → edit transition:** After exploring with Serena (find_symbol, type_hierarchy), the same name paths can be used directly for edits (replace_symbol_body, insert_after_symbol). Built-in exploration (get_file_structure) produces line numbers that must be used immediately before they go stale.
+
+- **Cross-file refactoring chains:** A rename is equivalent on both sides (1 call each). A move followed by a delete is 2 Serena calls vs. ~10+ built-in calls. The advantage compounds for move-heavy refactoring but not for rename-heavy refactoring.
+
+- **Session overhead:** Serena requires project activation (1 call at session start). The project can deactivate unexpectedly (observed once during this evaluation), requiring reactivation. This is a minor but real friction cost.
+
+**Verdict:** Serena's advantages compound for move-refactoring and exploration-heavy sessions; for rename-heavy sessions, both toolsets are equivalent.
+
+---
+
+## 7. Unique Capabilities (No Built-In Equivalent)
+
+| Capability | Frequency | Impact per use |
+|-----------|-----------|---------------|
+| Move symbol between modules (with import updates) | Occasional | Saves ~7 calls, eliminates manual import tracking |
+| Move file with import updates | Occasional | Saves ~3 calls |
+| Transitive type hierarchy | A few times per exploration session | Provides information unavailable to built-ins |
+| External dependency symbol lookup | Occasional | Provides information unavailable to built-ins |
+| Safe delete with usage guard | Occasional | Prevents accidental breakage |
+| Propagated delete (delete + remove call sites) | Rare | Saves multiple manual edits |
+| Inline symbol | Rare | Saves multiple manual edits |
+| Scope-precise symbol targeting by name path | Continuous (every symbol interaction) | Eliminates disambiguation overhead |
+
+**Verdict:** Serena provides 5-6 capabilities with no practical built-in equivalent, concentrated in move-refactoring, semantic navigation, and safe/propagated delete.
+
+---
+
+## 8. Tasks Outside Serena's Scope (Built-In Only)
+
+| Task | Share of typical session |
+|------|------------------------|
+| Reading/editing non-code files (configs, docs, changelogs, notebooks) | ~15-20% |
+| Free-text search (log strings, URLs, magic constants) | ~10% |
+| Terminal commands (git, test runners, build tools) | ~15-20% |
+| Creating new files from scratch | ~5-10% |
+| Directory listing and repo navigation | ~5% |
+| Running and debugging applications | ~10-15% |
+
+Estimated share of session where Serena's augmentation applies: **~35-50%** (code navigation, symbol editing, refactoring). The remainder is built-in-only territory.
+
+**Verdict:** Serena augments roughly a third to half of typical session work; the rest is inherently outside its scope and handled by built-ins.
+
+---
+
+## 9. Practical Usage Rule
+
+| Task type | Use |
+|-----------|-----|
+| Cross-file rename | Either (both provide semantic rename in 1 call) |
+| Move symbol/file with import updates | Serena (unique capability) |
+| Safe/propagated delete | Serena (unique capability) |
+| Inline symbol | Serena (unique capability) |
+| Type hierarchy, dependency lookup | Serena (unique capability) |
+| Find code references to a symbol | Serena (`find_referencing_symbols`) |
+| Find text mentions anywhere | Built-in (`search_project` / Grep) |
+| Small edit (1-3 lines) | Built-in (`search_replace`) |
+| Full method rewrite | Either (comparable) |
+| Insert at structural location | Serena (`insert_after_symbol`) if name path known; built-in otherwise |
+| Read non-code files | Built-in (`open` / Read) |
+| Terminal operations | Built-in (bash) |
+| New file creation | Built-in (`create`) |
+| Chained edits in one file | Either; Serena if line-number staleness is a concern |
+
+**Verdict:** Use Serena for move-refactoring, semantic navigation, and safe delete; use either toolset for rename; use built-ins for text-level edits, non-code files, and terminal operations.

src/serena/resources/config/contexts/junie.yml

@@ -12,6 +12,10 @@ excluded_tools:
   - create_text_file
   - read_file
   - execute_shell_command
+  - replace_content
+  - find_file
+  - list_dir
+  - search_for_pattern
 
 tool_description_overrides: {}