diff --git a/CHANGELOG.md b/CHANGELOG.md index f3274c1f..987c96f2 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -13,6 +13,9 @@ Status of the `main` branch. Changes prior to the next official version change w * Dependencies: - `pywebview`: Switch back to official release (new version 6.2) #1253 +* Evaluations: + - Added new evaluations for Junie Plugin with Opus 4.6 and GLM 5.1 in Claude Code. + # v1.1.2 (2026-04-14) * General: diff --git a/docs/04-evaluation/000_evaluation-intro.md b/docs/04-evaluation/000_evaluation-intro.md index 64efedee..596685a5 100644 --- a/docs/04-evaluation/000_evaluation-intro.md +++ b/docs/04-evaluation/000_evaluation-intro.md @@ -16,7 +16,7 @@ medium-sized Java project in the case of Codex). Different agents in different settings independently converge on the same core finding: Serena's strongest contribution is collapsing multi-file, semantically-aware operations into single atomic calls, while built-in tools remain the better choice for small local edits, text search, -config files, and shell work. +config files, and shell work. See the full [results](030_results/000_evaluation-results.md) for the scenarios mentioned above and several other scenarios. ## Approach @@ -32,3 +32,16 @@ The agent evaluates itself. This is deliberate: the agent is the actual user of so it can judge workflow improvements from direct experience rather than through proxy metrics. And because the prompt defines task *categories* rather than fixed tasks, anyone can rerun the evaluation on their own project with their own agent. + + +## Serena in JetBrains Junie + +A particularly notable evaluation scenario is Serena in JetBrains' Junie plugin. The latter also has access +to some of JetBrains' refactoring tools. At the time of writing, the only overlapping capability between Serena's tool +and Junie's native tools is the renaming feature. Opus correctly noticed this during the evaluation and marked the +renaming capability as equivalent. However, many symbolic and refactoring tools offered by Serena have no inbuilt equivalents, +leading to the following summary result for Junie: + +> **Junie Plugin (Opus 4.6)**: Serena gives me what my built-in tools can't — the ability to move a function between modules with all imports updated atomically, trace a class hierarchy into dependencies, and safely delete symbols with usage guards — and I'd ask my owner to add it for the move-refactoring and semantic navigation capabilities alone. + +See the full evaluation [here](030_results/050_junie_plugin_on_tianshou.md). diff --git a/docs/04-evaluation/030_results/000_evaluation-results.md b/docs/04-evaluation/030_results/000_evaluation-results.md index eecd29e8..20fecab7 100644 --- a/docs/04-evaluation/030_results/000_evaluation-results.md +++ b/docs/04-evaluation/030_results/000_evaluation-results.md @@ -13,6 +13,8 @@ evaluation can easily be repeated with the LSP-based backend to assess its subse - [Claude Code (Opus 4.6) on a large Python codebase](010_cc_on_tianshou) - [Codex (GPT 5.4) on a Java codebase](020_codex_on_jbplugin) - [Copilot CLI (GPT 5.4) on a large, multi-language monorepo](030_copilot_cli_on_ente.md) +- [GLM 5.1 in Claude Code](040_glm_on_tianshou) +- [JetBrains Junie with Opus 4.6](050_junie_plugin_on_tianshou.md) You can run your own evaluation on a project of your choice by reusing our [evaluation prompt](../020_prompts/010_evaluation-prompt.md). diff --git a/docs/04-evaluation/030_results/010_cc_on_tianshou.md b/docs/04-evaluation/030_results/010_cc_on_tianshou.md index 763f5f6f..f18a9d7a 100644 --- a/docs/04-evaluation/030_results/010_cc_on_tianshou.md +++ b/docs/04-evaluation/030_results/010_cc_on_tianshou.md @@ -2,6 +2,7 @@ :class: note **Generated by**: Claude Opus 4.6 (coding AI agent in Claude Code CLI) **Codebase:** [Tianshou](https://github.com/thu-ml/tianshou) — a Python reinforcement learning library (~26K lines, 43 source files) +**Date**: 2026-04-13 ::: # Claude Code (Opus 4.6, medium) diff --git a/docs/04-evaluation/030_results/020_codex_on_jbplugin.md b/docs/04-evaluation/030_results/020_codex_on_jbplugin.md index ee0f78d8..d9cb58e1 100644 --- a/docs/04-evaluation/030_results/020_codex_on_jbplugin.md +++ b/docs/04-evaluation/030_results/020_codex_on_jbplugin.md @@ -2,6 +2,7 @@ :class: note **Generated by:** GPT-5.4 (high) in Codex **Codebase:** Serena JetBrains Plugin (Java) +**Date**: 2026-04-13 ::: # Codex (GPT-5.4, high) diff --git a/docs/04-evaluation/030_results/030_copilot_cli_on_ente.md b/docs/04-evaluation/030_results/030_copilot_cli_on_ente.md index bef518e2..a3ba420d 100644 --- a/docs/04-evaluation/030_results/030_copilot_cli_on_ente.md +++ b/docs/04-evaluation/030_results/030_copilot_cli_on_ente.md @@ -2,6 +2,7 @@ :class: note **Generated by:** GPT-5.4 (medium) in Copilot Cli **Codebase:** [ente](https://github.com/ente-io/ente) - a large monorepo in Dart, TypeScript, Go, Rust, and other languages. +**Date**: 2026-04-14 ::: # Copilot CLI (GPT-5.4, medium) diff --git a/docs/04-evaluation/030_results/040_glm_on_tianshou.md b/docs/04-evaluation/030_results/040_glm_on_tianshou.md new file mode 100644 index 00000000..437ce905 --- /dev/null +++ b/docs/04-evaluation/030_results/040_glm_on_tianshou.md @@ -0,0 +1,446 @@ +:::{admonition} Evaluation Result +:class: note +**Generated by**: GLM 5.1 (coding AI agent in Claude Code CLI) +**Codebase:** [Tianshou](https://github.com/thu-ml/tianshou) — a Python reinforcement learning library (~26K lines, 43 source files) +**Date**: 2026-04-14 +::: + +# Claude Code (GLM 5.1) + +>Serena's cross-file refactoring turned an 8-step, error-prone rename chain into a single atomic call — and after +measuring the delta across 20 tasks, I would ask any developer I work with to install it, not because it replaces my +built-in tools, but because it makes the hardest part of my job (changing code safely across files) suddenly feel +reliable. + +--- + +## 1. Headline: What Serena Changes + +**(a) Tasks where Serena adds capability.** Cross-file refactorings (rename, move) collapse from N separate read-find-edit-verify cycles into a single atomic call. Semantic queries (type hierarchy, find implementations, find references) return structured, symbol-accurate results that text search cannot produce without manual filtering. Symbol-level addressing (name paths) is stable across edits, eliminating stale-line-number rework in multi-step sessions. + +**(b) Tasks where Serena applies but offers no improvement.** Single-file, small-bore edits (changing an error message, renaming a local variable with `replace_all`, inserting a function when you already know the insertion context) are equally fast with both toolsets. For a 1-line tweak in a 22-line method, Edit sends ~200 chars; Serena's `replace_symbol_body` sends ~800 chars (the entire method body). The overhead reverses for full-body rewrites of large methods, where Serena sends only the new body while Edit sends old+new. + +**(c) Tasks outside Serena's scope.** Reading non-code files (config, TOML, docs), free-text pattern search (`FIXME`, magic constants, log strings), shell operations, and git workflows are all built-in territory. These are not Serena shortcomings — they are outside its design scope. + +**Verdict:** Serena adds substantial, measurable capability in two areas — cross-file refactoring (1 call vs 5–8) and semantic code queries (structured, symbol-accurate results vs flat text matches) — while providing no meaningful delta for small single-file edits or non-code tasks. + +--- + +## 2. Added Value and Differences by Area + +### 2.1 Cross-file renaming: 1 call replaces 5–8 + +- **What changes:** Renaming `CollectStatsBase` to `BaseCollectStats` across 4 files (10 occurrences) required 1 Serena call vs 1 Grep + 3 Reads + 4 Edits = 8 built-in calls. +- **Frequency:** Medium. Any non-trivial rename touches 3–10 files. +- **Value per hit:** Saves 4–7 calls and eliminates the partial-update risk of a mid-chain failure. +- **Atomicity:** Serena's rename is all-or-nothing. Built-in chain is not — if Edit 3 of 4 fails, 2 files are updated and 2 are not. + +### 2.2 Symbol moving: 1 call replaces 5+ + +- **What changes:** Moving `_nullable_slice` to another module required 1 Serena call (moved definition + updated imports in both source and target). Built-in equivalent: read function → write to target → edit source (remove + add import) → edit target (add dependency import) = 5+ calls. +- **Frequency:** Low. Module reorganization happens infrequently. +- **Value per hit:** Saves 4+ calls and automates the most error-prone step (getting imports right). +- **Caveat:** The move tool created a circular import (source imports from target, target imports from source). The tool does not detect or prevent this. + +### 2.3 Reference finding: Symbol-accurate vs text-matched + +- **What changes:** Finding references to `CollectStats` with Serena returned 10 semantically categorized results (IMPORT_ELEMENT, REFERENCE_EXPRESSION, NAMED_PARAMETER, etc.) with no false positives from `LoggedCollectStats`. Grep returned 70+ lines including false positives from `LoggedCollectStats` and noise from docstrings/comments. +- **Frequency:** High. "Who uses this?" is one of the most common codebase questions. +- **Value per hit:** Eliminates manual false-positive filtering. However, Serena's output was 64KB (with context snippets) vs Grep's ~5KB — a tradeoff of precision vs verbosity. + +### 2.4 Type hierarchy: 1 call vs 2+ grep-and-parse cycles + +- **What changes:** Getting the full type hierarchy of `BaseCollector` (supertypes: ABC → object; subtypes: Collector → AsyncCollector) required 1 Serena call. Built-in: parse the class definition line for bases + grep for inheritors + recurse = 2–3 calls, and still no access to external library types. +- **Frequency:** Medium. Common when navigating unfamiliar code. +- **Value per hit:** Saves 1–2 calls and returns transitive chains that built-ins can't produce without iteration. + +### 2.5 Structural overview: Hierarchical vs flat + +- **What changes:** `get_symbols_overview(depth=1)` on a 1551-line file returned a structured hierarchy (classes → methods + attributes) in one call. Grep for `^(class |def | def )` returned a flat 60-line list of definitions with line numbers but no attribute information or nesting. +- **Frequency:** High. Opening any unfamiliar file. +- **Value per hit:** Serena shows Protocol fields, class attributes, and method grouping. Grep shows line numbers for navigation but requires more work to understand structure. + +### 2.6 Method body retrieval: Targeted read vs range read + +- **What changes:** Reading `Collector._collect` (330+ lines) required 1 Serena `find_symbol` call using the name path. Built-in: need to know the line range (773–1103) from a prior Grep, then `Read(offset=773, limit=330)`. +- **Frequency:** High. "Show me this method" is routine. +- **Value per hit:** Serena uses stable addressing (name path). Built-in uses ephemeral line numbers that go stale after edits. The practical difference is small when you just read once, but compounds in edit-then-read-again workflows. + +### 2.7 Single-file rename: No meaningful difference + +- **What changes:** Renaming `_nullable_slice` (4 occurrences in one file): Serena rename = 1 call. Edit `replace_all` = 1 call. Identical results. +- **Frequency:** High. +- **Value per hit:** Zero. Both tools handle this equally well. + +### 2.8 Small edits (1–3 lines): Edit is more token-efficient + +- **What changes:** Changing one error message in a 22-line method: Edit sends ~200 chars (old + new string). Serena `replace_symbol_body` sends ~800 chars (entire method body). +- **Frequency:** Very high. +- **Value per hit:** Edit saves ~600 chars of payload per small edit. This reverses for full-body rewrites of 50+ line methods. + +### 2.9 Insertion: Stable address vs text anchor + +- **What changes:** Inserting a new method after `refresh_all_sequence_stats`: Serena used 1 `insert_after_symbol` call with a name path (no line number needed). Edit used 1 Read + 1 Edit with a text anchor (surrounding context for uniqueness). +- **Frequency:** Medium. +- **Value per hit:** Saves 1 Read call. Both produce identical results. + +**Verdict:** Serena's value concentrates in cross-file operations and semantic queries. For single-file text edits, the built-ins are equally capable and often more token-efficient. + +--- + +## 3. Detailed Evidence, Grouped by Capability + +### 3.1 Codebase Understanding + +#### Task 1: Repo overview +Both toolsets use the same approach (`ls`, `find`, directory listing). No Serena advantage here. + +#### Task 2: Structural overview of a large file (`collector.py`, 1551 lines) + +| Step | Serena | Built-in | +|------|--------|----------| +| Call | `get_symbols_overview(depth=1)` | `Grep "^(class \|def \| def )"` | +| Result | Hierarchical: 14 classes, their methods, attributes, and module-level functions | Flat list of 60 class/function definitions with line numbers | +| Output size | ~1.5KB structured JSON | ~3KB text | +| Next step | `find_symbol("Collector/_collect", include_body=True)` — direct | `Read(offset=773, limit=330)` — needs prior knowledge of line range | + +**Serena advantage:** Shows attributes (e.g., `CollectStats.collect_time`, `CollectStats.returns`) that Grep cannot see. Hierarchical nesting makes the file's architecture immediately clear. + +**Built-in advantage:** Line numbers enable direct `Read` calls. Flat output is compact. + +**Verdict:** Serena provides strictly more structural information in one call. The gap widens for files with deeply nested classes or dataclass fields. + +#### Task 3: Retrieve a specific method body + +| Step | Serena | Built-in | +|------|--------|----------| +| Prerequisite | Name path known (`Collector/_collect`) | Line number known from prior Grep (773) | +| Call | `find_symbol(name_path="Collector/_collect", include_body=True)` | `Read(offset=773, limit=330)` | +| Payload sent | ~50 chars (name + path) | ~30 chars (offset + limit) | +| Payload received | Exact method body (~330 lines) | Lines 773–1102 (~330 lines) | +| Correctness | Always exact | Must know/guess the correct limit | + +**Verdict:** Functionally equivalent when line numbers are known. Serena's name-path addressing degrades gracefully across edits; line numbers do not. + +#### Task 4: Find all references to `CollectStats` + +| Metric | Serena `find_referencing_symbols` | Built-in `Grep` | +|--------|----------------------------------|-----------------| +| Calls | 1 | 1 | +| Output size | 64KB (with context snippets) | ~5KB (70 lines) | +| False positives | 0 (excludes `LoggedCollectStats`) | 5+ lines from `LoggedCollectStats` | +| Noise (comments/docs) | Some (docstrings categorized) | Significant (docstrings, comments matched) | +| Semantic categories | Yes (IMPORT, PARAMETER, DECLARATION, REFERENCE) | No | + +**Serena advantage:** Zero false positives. Semantic categorization. Shows import paths and parameter usage separately from code references. + +**Built-in advantage:** 10x smaller output. Faster to scan visually. + +**Verdict:** Serena is more precise but more verbose. For "who uses this in code?" both work; for "rename this safely" Serena's precision is necessary. + +#### Task 5: Type hierarchy of `BaseCollector` + +| Metric | Serena `type_hierarchy` | Built-in (Grep chain) | +|--------|------------------------|----------------------| +| Calls | 1 | 2–3 (grep subclasses, parse superclass, recurse) | +| Result | `ABC → object` (super), `Collector → AsyncCollector` (sub) | Partial — direct sub/supertypes only, no transitive chain | +| External deps | Shows `ABC` from `abc.pyi` | Cannot access | + +**Verdict:** Serena returns complete, transitive hierarchy including external library types in one call. Built-in approach requires iteration and cannot inspect external deps. + +#### Task 6: External dependency symbol lookup + +Serena can read external dependency symbols IF you have the path from a prior tool result (e.g., `` from `type_hierarchy`). Direct search with `search_deps=True` returned empty results for `numpy.array` and `torch.Tensor`. The JetBrains IDE indexing is limited to what the language server has resolved. + +Built-in: Can `Read` site-packages files if you know the path, but discovery is manual. + +**Verdict:** Minor Serena advantage — external symbols are accessible through tool chains but not through direct search. Neither toolset makes this easy. + +### 3.2 Single-File Edits + +#### Task 7a: Small tweak (1-line change in 22-line method) + +| Metric | Edit | Serena `replace_symbol_body` | +|--------|------|------------------------------| +| Prerequisite | 1 Read (6 lines of context) | 1 `find_symbol` (gets full body) | +| Payload sent | ~200 chars (old + new string) | ~800 chars (full method body) | +| Payload received | Success message | "OK" | +| Total payload | ~200 chars edit + ~300 chars read = ~500 | ~800 chars edit + ~800 chars read = ~1600 | + +**Verdict:** Edit is 3x more token-efficient for small tweaks inside methods. + +#### Task 7b: Medium rewrite (~6 line changes in 20-line method) + +| Metric | Edit | Serena `replace_symbol_body` | +|--------|------|------------------------------| +| Payload sent | ~700 chars (old + new, full method) | ~500 chars (new body only) | +| Prerequisite read | ~500 chars | ~500 chars (from prior `find_symbol`) | + +**Verdict:** Roughly equal. For medium rewrites, payloads converge. + +#### Task 7c: Large rewrite (full body of 55+ line method) + +For a full-body rewrite of a 55-line method: +- Edit: old (~55 lines) + new (~55 lines) = ~110 lines sent +- Serena: new body only (~55 lines) sent + +**Verdict:** Serena is ~2x more token-efficient for full-body rewrites. The advantage grows linearly with method size. + +#### Task 8: Insert a new function after an existing one + +| Step | Serena | Built-in | +|------|--------|----------| +| 1 | `find_symbol("refresh_all_sequence_stats")` to confirm target | `Read(offset=250, limit=10)` to find insertion point | +| 2 | `insert_after_symbol(name_path, body)` | `Edit(old_string=anchor, new_string=anchor+new_fn)` | +| Total calls | 2 | 2 | +| Payload sent | New function body only (~300 chars) | Anchor context + new function (~400 chars) | + +**Verdict:** No meaningful difference. Both require 2 calls and produce identical results. + +#### Task 9: Rename a private helper (single-file, 4 occurrences) + +| Metric | Serena `rename` | Edit `replace_all` | +|--------|-----------------|-------------------| +| Calls | 1 | 1 | +| Prerequisites | None | Must have Read the file first | +| Result | All 4 occurrences renamed | All 4 occurrences renamed | + +**Verdict:** Functionally identical. Both are 1 call. Edit's Read-first requirement is usually satisfied from prior exploration. + +### 3.3 Multi-File Changes + +#### Task 10: Cross-file rename (`CollectStatsBase` → `BaseCollectStats`, 4 files, 10 occurrences) + +| Step | Serena | Built-in | +|------|--------|----------| +| Find references | Automatic | 1 Grep call | +| Read files | Not needed | 3 Read calls (Edit's prerequisite) | +| Apply edits | 1 rename call (atomic) | 4 Edit calls (one per file) | +| Verify | Return: "Success" | Manual (4 success messages) | +| **Total calls** | **1** | **8** (1 grep + 3 reads + 4 edits) | + +**Verdict:** Serena converts an 8-call manual pipeline into 1 atomic operation. This is the single largest efficiency gain observed. + +#### Task 11: Move symbol to another module + +Serena's `move` tool: +1. Moved `_nullable_slice` from `collector.py` to `converter.py` +2. Added import in source file: `from tianshou.data.utils.converter import _nullable_slice` +3. Added dependency import in target: `from tianshou.data.collector import _TArrLike` +4. Removed definition from source + +Built-in equivalent: Read function body → Write to target → Edit source (remove definition + add import) → Edit target (add dependency import) = 5+ calls. + +**Issue:** The move created a circular import (source ↔ target). Serena does not detect or prevent this. + +**Verdict:** Serena automates the most tedious part (import management) but doesn't guard against circular dependencies. Saves 4+ calls at the cost of needing manual circular-import review. + +#### Task 12: Move file (`segtree.py` to parent directory) + +Serena's `move` tool: +1. Moved the file +2. Updated the one direct import in `__init__.py` (`tianshou.data.utils.segtree` → `tianshou.data.segtree`) +3. Other files (`prio.py`, tests) imported via re-export and needed no changes + +Built-in equivalent: `git mv` + grep for old import path + edit each file = 3+ calls. + +**Verdict:** Serena saves 1–2 calls and automatically discovers which imports need updating. + +#### Task 12 (safe delete) + +Serena's `safe_delete` correctly refused to delete `_HACKY_create_info_batch` because it has a usage at line 730. The `propagate=true` mode (delete symbol + all call sites) failed for all tested symbols in this codebase. + +**Verdict:** The usage-check is valuable (saves you from deleting a used symbol). The propagation feature was non-functional for the tested Python symbols. + +#### Task 13: Inline + +Serena's `inline_symbol` failed for all tested symbols (`_nullable_slice`, `BaseCollector/env_num`). The tool appears to have limited Python support for inlining. + +**Verdict:** No successful inline demonstrated. Built-in manual inlining remains the only option. + +### 3.4 Reliability and Correctness + +#### Task 14: Scope precision + +Serena distinguishes `BaseCollector/_collect`, `Collector/_collect`, and `AsyncCollector/_collect` by name path. Grep for `def _collect` matches all three — manual filtering by class is required. + +**Verdict:** Serena's name-path addressing eliminates ambiguity that text search cannot resolve. + +#### Task 15: Atomicity + +Serena's cross-file rename is atomic: 4 files updated in 1 call, all-or-nothing. Built-in: 4 separate Edit calls — if call 3 fails, 2 files are updated and 2 are not. + +**Verdict:** Serena provides atomicity for cross-file operations. Built-in chains are inherently non-atomic. + +#### Task 16: Success signals + +Both return clear success/failure indicators. No meaningful difference. + +**Verdict:** Equal. + +### 3.5 Workflow Effects + +#### Task 17: Chain three edits in one file + +Edit with text matching: 3 sequential calls, no re-reads needed between them. Text anchors are immune to line-number shifts from prior edits. + +Serena `replace_symbol_body`: 3 sequential calls, no re-reads needed. Name-path addressing is also immune to line-number shifts. + +**Verdict:** No meaningful difference for chained single-file edits. + +#### Task 18: Multi-step exploration across edits + +Serena's name-path results from exploration remain valid after edits. Built-in line numbers go stale, but Edit uses text matching (not line numbers), so the practical impact is limited to `Read` calls that need updated offsets. + +**Verdict:** Minor Serena advantage. Name-path stability eliminates the need to re-scan after edits. + +### 3.6 Non-Interesting Tasks + +#### Task 19: Read non-code file + +Serena tools don't apply. `Read` is the correct tool. + +#### Task 20: Free-text pattern search + +Searching for `FIXME|HACK|TODO` across the codebase is a text search. Serena's semantic tools don't target this. `Grep` is the correct tool. + +**Verdict:** These tasks are firmly built-in territory. They represent an estimated 20–30% of daily coding work (reading configs, searching for strings, shell operations, git workflows). + +--- + +## 4. Token-Efficiency Analysis + +### Payload differences across edit sizes + +| Edit type | Edit payload | Serena payload | Winner | +|-----------|-------------|---------------|--------| +| 1-line tweak in 22-line method | ~200 chars | ~800 chars | Edit (4x) | +| 6-line change in 20-line method | ~700 chars | ~500 chars | Roughly equal | +| Full rewrite of 55-line method | ~2200 chars | ~1100 chars | Serena (2x) | +| Full rewrite of 330-line method | ~13,000 chars | ~6,500 chars | Serena (2x) | + +### Forced reads + +- Edit requires reading a file before editing it. This adds ~300–2000 chars per file. +- Serena does not require reading before editing (name-path addressing). +- For single-file edits where you already read the file, this is neutral. +- For cross-file operations on files you haven't read, Serena saves 3–4 forced reads. + +### Stable vs ephemeral addressing + +- Serena: name paths (`Collector/_collect`) are stable across edits. Results from exploration remain valid. +- Built-in: line numbers are ephemeral. `Read` results go stale after edits. `Edit` uses text matching, which is stable. +- Practical impact: Low for one-shot edits, medium for edit-then-read-again workflows. + +**Verdict:** Edit wins for small tweaks (4x more token-efficient). Serena wins for full-body rewrites (2x more efficient) and cross-file operations (eliminates forced reads). The crossover point is approximately 50% of the method body changing — below that, Edit is more efficient; above that, Serena is. + +--- + +## 5. Reliability and Correctness (Under Correct Use) + +### Precision of matching + +- Serena: Symbol-accurate. `find_referencing_symbols(CollectStats)` excludes `LoggedCollectStats`. No false positives observed. +- Grep: Text-matched. `CollectStats` matches `LoggedCollectStats`, `CollectStatsBase`, and docstring references. Requires manual filtering. +- Edit: Text-matched. `replace_all` replaces exact string matches. For unique symbol names, this is reliable. For common strings, it can over-match. + +### Scope disambiguation + +- Serena: `Collector/_collect` vs `AsyncCollector/_collect` — correctly distinguished by class-scoped name path. +- Built-in: `def _collect` matches all implementations. Must manually verify class context. + +### Atomicity + +- Serena cross-file operations: Atomic. Single call, all-or-nothing. +- Built-in multi-file chains: Non-atomic. Partial state possible if one call fails. + +### External dependency lookup + +- Serena: Can read external stubs (e.g., `abc.pyi`) through paths returned by other tools. Direct search (`search_deps=True`) returned empty for `torch.Tensor` and `numpy.array`. Limited to what JetBrains has indexed. +- Built-in: Can `Read` site-packages files if path is known. No semantic indexing. + +**Verdict:** Serena provides strictly more precise semantic matching and atomic cross-file operations. External dependency lookup is limited in both toolsets. + +--- + +## 6. Workflow Effects Across a Session + +### Where advantages compound + +1. **Explore → edit → re-explore cycle:** Serena's name-path results survive edits. In a long session making multiple changes, this saves re-scanning after each edit. The built-in's text matching also survives edits (Edit uses text, not line numbers), so the practical gap is smaller than it appears. + +2. **Cross-file refactoring chains:** Rename a class, then move it, then update all references — each Serena call is atomic and builds on the previous result. With built-ins, each step requires finding all sites, reading files, and editing — the manual equivalent of what Serena automates. + +### Where advantages diminish + +1. **Repeated small edits in one file:** Edit's text matching is equally stable and more token-efficient for small changes. No Serena advantage. + +2. **Exploration without editing:** Both toolsets provide usable results. Serena's are more structured but more verbose. + +3. **Non-Python files:** Serena's JetBrains backend provides no value for config files, shell scripts, markdown, or notebooks. + +**Verdict:** Serena's advantages compound in multi-step cross-file refactoring sessions. They do not compound for single-file iterative editing or non-code work. + +--- + +## 7. Unique Capabilities + +1. **Atomic cross-file rename/move** — No built-in equivalent. The closest manual process is a grep-find-edit chain that is non-atomic and error-prone. Frequency: medium. Impact: high (eliminates partial-update risk). + +2. **Semantic reference finding with categorization** — `find_referencing_symbols` returns zero-false-positive results categorized by usage type (import, parameter, declaration, reference). Built-in Grep cannot distinguish these. Frequency: high. Impact: medium (saves manual filtering). + +3. **Type hierarchy traversal** — Returns transitive super/subtype chains including external library types in one call. Built-in requires iteration and cannot reach external deps. Frequency: medium. Impact: medium. + +4. **Symbol-scoped body retrieval** — Read a specific method by name path without reading the surrounding file. Built-in `Read` requires line-range knowledge. Frequency: high. Impact: low (both require 1 call, difference is stable vs ephemeral addressing). + +**Verdict:** Four unique capabilities, all semantic-code operations. The most impactful is atomic cross-file refactoring. None of these have practical built-in equivalents. + +--- + +## 8. Tasks Outside Serena's Scope (Built-In Only) + +| Task | Tool | Frequency | Share of daily work | +|------|------|-----------|-------------------| +| Read config/TOML/yaml files | `Read` | High | | +| Free-text search (log strings, TODOs, URLs) | `Grep` | High | | +| File discovery by name pattern | `Glob` | Medium | | +| Shell commands (git, pip, pytest) | `Bash` | High | | +| Write new files from scratch | `Write` | Medium | | +| Read images, notebooks, PDFs | `Read` | Low | | + +Estimated share of daily work covered by built-in-only tasks: **20–30%**. The remaining 70–80% involves reading, editing, and navigating code where Serena's semantic tools are applicable. + +**Verdict:** Serena covers the code-editing and code-navigation portions of a session. Config reading, text search, and shell operations remain built-in territory. + +--- + +## 9. Practical Usage Rule + +**Use Serena for:** Any cross-file refactoring (rename, move), any "who uses this?" query, any type-hierarchy navigation, any full-method-body replacement, and any situation where you need symbol-accurate results without false positives. + +**Use built-ins for:** Small edits (1–3 lines) inside methods, free-text search, reading non-code files, file discovery, and shell operations. + +**Hybrid pattern (most efficient):** Use Serena to explore (overview, find symbol, find references) and Edit for small targeted changes. Use Serena for any cross-file refactoring. Use `Read`/`Grep`/`Glob` for non-code tasks and text search. This combination captures the strengths of both toolsets. + +**Verdict:** The optimal workflow uses Serena's semantic tools for code navigation and cross-file refactoring, and built-in Edit for small single-file changes. The two toolsets are complementary — Serena handles the structured code operations, built-ins handle the text and system operations. + +--- + +## Appendix: Call Count Summary + +| Task | Serena calls | Built-in calls | Delta | +|------|-------------|----------------|-------| +| Structural overview (1 file) | 1 | 1 | 0 | +| Method body retrieval | 1 | 1 | 0 | +| Find references (1 symbol) | 1 | 1 | 0 (but Serena has 0 false positives vs Grep's 5+) | +| Type hierarchy | 1 | 2–3 | −1 to −2 | +| Small edit (1 line in 22-line method) | 1 (+1 read) | 1 (+1 read) | 0 | +| Medium edit (6 lines in 20-line method) | 1 | 1 | 0 | +| Insert new method | 1 (+1 confirm) | 1 (+1 read) | 0 | +| Single-file rename (4 occurrences) | 1 | 1 | 0 | +| **Cross-file rename (4 files, 10 occurrences)** | **1** | **8** | **−7** | +| **Move symbol to another module** | **1** | **5+** | **−4+** | +| **Move file + update imports** | **1** | **3+** | **−2+** | +| Safe delete (usage check) | 1 | 1 grep | 0 | + +Largest single-task delta: cross-file rename saves 7 calls and provides atomicity. diff --git a/docs/04-evaluation/030_results/050_junie_plugin_on_tianshou.md b/docs/04-evaluation/030_results/050_junie_plugin_on_tianshou.md new file mode 100644 index 00000000..cfafc462 --- /dev/null +++ b/docs/04-evaluation/030_results/050_junie_plugin_on_tianshou.md @@ -0,0 +1,345 @@ +:::{admonition} Evaluation Result +:class: note +**Generated by**: Claude Opus 4.6 (coding AI agent in JetBrains Junie Plugin) +**Codebase:** [Tianshou](https://github.com/thu-ml/tianshou) — a Python reinforcement learning library (~26K lines, 43 source files) +**Date**: 2026-04-17 +::: + +# JetBrains Junie Plugin (Opus 4.6) + +> Serena gives me what my built-in tools can't — the ability to move a function between modules with all imports updated atomically, trace a class hierarchy into dependencies, and safely delete symbols with usage guards — and I'd ask my owner to add it for the move-refactoring and semantic navigation capabilities alone. + +--- + +## 1. Headline: What Serena Changes + +Serena adds three categories of capability on top of built-in tools: + +**(a) Tasks where Serena adds capability:** +- **Move symbol between modules** (with automatic import updates) — no built-in equivalent. +- **Move file/package** (with automatic import updates) — no built-in equivalent. +- **Type hierarchy traversal** — transitive supertype/subtype chains in 1 call, including into external dependencies. No built-in equivalent. +- **External dependency symbol lookup** — search and inspect symbols in installed packages (site-packages) without manual path discovery. Built-ins cannot do this. +- **Safe delete with usage checking** — refuses deletion if usages exist, reports exact locations. Built-ins require a manual search-then-decide workflow. +- **Propagated delete** — delete a symbol and automatically remove all call sites. No built-in equivalent. +- **Inline symbol** — replace all call sites with the function body. No built-in equivalent. +- **Scope-precise symbol targeting** — name paths like `AsyncCollector/_collect` disambiguate overrides that text search cannot distinguish. + +**(b) Tasks where Serena applies but offers no meaningful improvement:** +- **Cross-file rename** — both Serena (`rename`) and built-in (`rename_element`) perform semantic, cross-file renames in 1 call. Functionally equivalent. +- **Single-file rename** — same as above; both are 1-call semantic renames. +- **Structural overview of a single file** — both toolsets produce comparable results in 1 call. +- **Small edits (1-3 lines)** — Serena's `replace_symbol_body` must send the entire method body; built-in `search_replace` sends only the changed line. Built-in is more token-efficient here. + +**(c) Tasks outside Serena's scope (built-in only):** +- Reading non-code files (configs, docs, changelogs) +- Free-text search across the repo (log strings, URLs, magic constants) +- Terminal commands, test execution, git operations +- Creating new files from scratch +- Repository-level directory listing + +**Verdict:** +Serena's primary contribution is move-refactoring (symbol and file moves with import updates), semantic code navigation (type hierarchy, dependency lookup), and safe/propagated delete — capabilities with no built-in equivalent. + +--- + +## 2. Added Value and Differences by Area + +- **Move symbol/file with import updates (positive).** Frequency: occasional in refactoring work. Value per hit: saves ~7 calls and eliminates manual import tracking. Moving `get_stddev_from_dist` between modules: 1 Serena call vs. ~7 built-in calls. No built-in equivalent. + +- **Type hierarchy and reference finding (positive).** Frequency: a few times per exploration session. Value per hit: saves 2-4 search-and-read cycles and provides information (transitive supertypes, external dependency chains) that built-ins cannot produce at all. `BaseCollector`'s full hierarchy (ABC→object up, Collector→AsyncCollector down) returned in 1 call. + +- **Targeted symbol retrieval by name path (positive).** Frequency: many times per session. Value per hit: saves 1 prerequisite read (no line-number lookup needed). `find_symbol` with `include_body=true` returns exact method body in 1 call vs. 2 calls (structure + open). + +- **Safe delete with usage guard (positive).** Frequency: occasional. Value per hit: eliminates risk of orphaned references. Serena refused to delete `_dict_of_arr_to_arr_of_dicts` and reported 2 exact usages. Built-in: requires manual search first. + +- **Cross-file rename (neutral).** Frequency: several times per session. Value per hit: zero — both Serena's `rename` and built-in `rename_element` perform semantic cross-file renames in 1 call with equivalent results. + +- **Single-file edits (neutral to slightly negative for small edits).** Frequency: very high. Value per hit: for small edits, Serena's `replace_symbol_body` sends more tokens (entire method body) than `search_replace` (just the changed lines). For full method rewrites, comparable. For inserts, Serena's `insert_after_symbol` saves 1 prerequisite read. + +- **Structural overview (neutral).** Frequency: a few times per session. Value per hit: both toolsets return comparable information in 1 call. + +**Verdict:** Serena's value concentrates in move-refactoring and semantic navigation; rename is matched by the built-in `rename_element`. + +--- + +## 3. Detailed Evidence, Grouped by Capability + +### 3.1 Structural Overview (Task 2) + +**Attempted:** Get structural overview of `collector.py` (1552 lines, 15+ classes). + +| Axis | Serena (`get_symbols_overview`) | Built-in (`get_file_structure`) | +|------|------|------| +| Calls | 1 | 1 | +| Output | JSON tree: class names, method names, field names | Flat list: class/method names with line ranges and signatures | +| Unique info | Field/attribute names | Line numbers, full parameter signatures | +| Follow-up to read a method | `find_symbol` by name path (1 call) | `open` at line number (1 call) | + +**Verdict:** Functionally equivalent; each includes information the other omits. No meaningful delta. + +### 3.2 Targeted Symbol Retrieval (Task 3) + +**Attempted:** Retrieve body of `CollectStats/refresh_all_sequence_stats` without reading surrounding file. + +| Axis | Serena | Built-in | +|------|------|------| +| Calls | 1 (`find_symbol` with `include_body`) | 2 (`get_file_structure` + `open`) | +| Prerequisite | None (name path is stable) | Must know line number | +| Output payload | 4 lines (exact method body) | 100 lines (open window) | + +**Verdict:** Serena saves 1 call and returns a smaller, more precise payload. Minor but consistent advantage. + +### 3.3 Reference Finding (Task 4) + +**Attempted:** Find all references to `CollectStats` across the codebase. + +| Axis | Serena (`find_referencing_symbols`) | Built-in (`search_project`) | +|------|------|------| +| Calls | 1 | 1 | +| Output | Summarized narrative: grouped by file, annotated with context (imports, instantiations, type hints) | Raw list: 100+ text matches, includes docs, comments, strings | +| Precision | Code references only | All text mentions | +| Recall | Code references only | Everything including non-code | + +**Verdict:** Serena provides higher precision (code-only references with semantic context). Built-in provides higher recall (includes docs, comments). Different tools for different questions. + +### 3.4 Type Hierarchy (Task 5) + +**Attempted:** Full type hierarchy of `BaseCollector` — supertypes and subtypes, transitively. + +| Axis | Serena (`type_hierarchy`) | Built-in | +|------|------|------| +| Calls | 1 | 3+ (grep for class declarations, read each, trace manually) | +| Result | `BaseCollector → ABC → object` (up), `→ Collector → AsyncCollector` (down) | Partial: can find direct subclasses by grep, but supertypes of supertypes require reading external files | +| External deps | Included (ABC from `abc` module, object from `builtins`) | Not accessible | + +**Verdict:** Unique capability. No practical built-in equivalent for transitive hierarchy, especially into dependencies. + +### 3.5 External Dependency Lookup (Task 6) + +**Attempted:** Find `Distribution` class from `torch.distributions`. + +| Axis | Serena (`find_symbol` with `search_deps`) | Built-in | +|------|------|------| +| Calls | 1 | Not possible without manual site-packages navigation | +| Result | Found 41 `Distribution` classes across all installed packages with ext-identifiers | N/A | +| Follow-up | Can use ext-identifier to read body/info | Would need to find virtualenv path, navigate to package, read file | + +**Verdict:** Unique capability. Built-ins have no access to dependency source code without manual path discovery. + +### 3.6 Small Edit — 1 Line Change (Task 7a) + +**Attempted:** Change error message in `_validate_buffer` (21-line method). + +| Axis | Serena | Built-in | +|------|------|------| +| Calls | 2 (`find_symbol` + `replace_symbol_body`) | 1 (`search_replace`) | +| Input payload | ~21 lines (full method body) | ~1 line (search) + ~1 line (replace) | +| Prerequisite reads | 1 (find_symbol to get current body) | 0 (if search string is known) | + +**Verdict:** Built-in is more efficient for small, targeted edits. Serena's method-granularity addressing forces sending the entire body. + +### 3.7 Medium Rewrite — ~20 Lines (Task 7b) + +**Attempted:** Rewrite `update_at_step_batch` (20 lines), changing variable names and adding logging. + +| Axis | Serena | Built-in | +|------|------|------| +| Calls | 2 (`find_symbol` + `replace_symbol_body`) | 1 (`search_replace` with old body → new body) | +| Input payload | ~20 lines (new body) | ~40 lines (old + new body) | +| Prerequisite reads | 1 (find_symbol) | 1 (open to see current code) | + +**Verdict:** Comparable. Serena sends less payload (new body only vs. old+new), but requires a prerequisite find_symbol call. + +### 3.8 Insert New Method (Task 8) + +**Attempted:** Insert `summary_string` method after `refresh_all_sequence_stats` in `CollectStats`. + +| Axis | Serena (`insert_after_symbol`) | Built-in (`search_replace`) | +|------|------|------| +| Calls | 1 | 1-2 (need to find anchor text, then insert) | +| Addressing | By name path (stable) | By text anchor or line number (fragile) | +| Prerequisite | None | May need `open` to find insertion point | + +**Verdict:** Serena's stable addressing is a minor advantage — eliminates the need to find an anchor. + +### 3.9 Single-File Rename (Task 9) + +**Attempted:** Rename `_nullable_slice` → `_slice_if_not_none` (5 occurrences in 1 file). + +| Axis | Serena (`rename`) | Built-in (`rename_element`) | +|------|------|------| +| Calls | 1 | 1 | +| Scope | Semantic (only code references) | Semantic (only code references) | + +**Verdict:** Functionally equivalent. Both perform semantic renames in 1 call. + +### 3.10 Cross-File Rename (Task 10) + +**Attempted:** Rename `CollectStatsBase` → `CollectStatsFoundation` (used in 4 files, 10 occurrences including imports and `__all__`). + +| Axis | Serena (`rename`) | Built-in (`rename_element`) | +|------|------|------| +| Calls | 1 | 1 | +| Atomicity | All-or-nothing | All-or-nothing | +| Import handling | Automatic | Automatic | +| Result | 4 files, 10 replacements | 4 files, 10 replacements | + +**Verdict:** No delta. Both toolsets provide atomic, semantic cross-file rename in a single call. + +### 3.11 Move Symbol Between Modules (Task 11) + +**Attempted:** Move `get_stddev_from_dist` from `collector.py` to `stats.py`. + +| Axis | Serena (`move`) | Built-in | +|------|------|------| +| Calls | 1 | ~7 (read source, copy to target, delete from source, find imports, update each import, verify) | +| Files modified | 3 (source, target, test file) | Same 3, but manually | +| Import updates | Automatic | Manual | +| Atomicity | Atomic | Non-atomic | + +**Verdict:** Unique capability at this level of automation. No practical built-in equivalent at comparable effort. This is now the highest-value delta in the evaluation. + +### 3.12 Move File (Task 12) + +**Attempted:** Move `segtree.py` from `tianshou/data/utils/` to `tianshou/utils/`. + +| Axis | Serena (`move`) | Built-in | +|------|------|------| +| Calls | 1 | 3+ (bash mv, find imports, update each) | +| Import updates | Automatic (updated `__init__.py`) | Manual | + +**Verdict:** Serena advantage — single call with automatic import updates. + +### 3.13 Safe Delete (Task 13) + +**Attempted:** Delete `_dict_of_arr_to_arr_of_dicts` (has 2 usages). + +| Axis | Serena (`safe_delete`) | Built-in | +|------|------|------| +| Calls | 1 (refused, reported usages) | 2 (search_project + manual decision) | +| Safety | Refuses deletion, reports exact usages with file:line and enclosing function | Must manually verify search results | + +**Verdict:** Serena provides a safety guard with precise diagnostics. Minor but useful advantage. + +### 3.14 Scope Precision (Task 14) + +**Attempted:** Target `AsyncCollector/_collect` specifically (3 classes define `_collect`). + +Serena's name path `AsyncCollector/_collect` unambiguously selects the override. A text search for `_collect` matches all three definitions plus dozens of call sites. Built-in `get_file_structure` can disambiguate by line number, but that requires a prerequisite read and the line number goes stale after edits. + +**Verdict:** Serena's name-path addressing provides persistent, unambiguous symbol targeting. Meaningful advantage when multiple overrides exist. + +### 3.15 Chained Edits (Task 17) + +**Attempted:** Three sequential `replace_symbol_body` calls on different methods in the same file. + +All three succeeded without re-reading the file. Name paths remained stable across edits. Built-in `search_replace` also chains without re-reading (text anchors remain valid if edits are in different methods), but line numbers from `open` go stale after each edit. + +**Verdict:** Both toolsets chain well. Serena's name-path stability is a minor advantage over line-number-based addressing. + +--- + +## 4. Token-Efficiency Analysis + +| Edit size | Serena payload | Built-in payload | Winner | +|-----------|---------------|-----------------|--------| +| Small (1-3 lines in 20-line method) | ~20 lines (full body) | ~2 lines (search+replace) | Built-in (10x less) | +| Medium (rewrite ~20 lines) | ~20 lines (new body) + prerequisite find | ~40 lines (old+new) | Comparable | +| Large (rewrite 50+ lines) | ~50+ lines (new body) | ~100+ lines (old+new) | Serena (2x less, no old body needed) | +| Insert | ~N lines (new code only) | ~N lines + anchor context | Comparable | +| Cross-file rename | ~1 line (name + new name) | ~1 line (name + new name) | Tie (both semantic) | +| Move symbol | ~1 line (source + target) | ~7 calls with full bodies | Serena (>>10x less) | + +**Forced reads:** Serena's `replace_symbol_body` requires a `find_symbol` call to get the current body before editing. Built-in `search_replace` needs no prerequisite if the search text is known, but often requires `open` to discover it. + +**Stable vs. ephemeral addressing:** Serena's name paths survive edits; built-in line numbers do not. This matters in multi-edit sessions — Serena never needs to re-read for addressing purposes, while built-in may need to re-open after edits that shift line numbers. + +**Verdict:** For single-file small edits, built-ins are more token-efficient. For move-refactoring, Serena is substantially more efficient. For rename, both are equivalent. The crossover point for edits is roughly at method-level rewrites. + +--- + +## 5. Reliability & Correctness (Under Correct Use) + +- **Precision of matching:** Serena matches by semantic identity (name path in the symbol tree). Built-in `search_replace` matches by text (can over-match). However, built-in `rename_element` also matches semantically. For rename operations, precision is equivalent. + +- **Scope disambiguation:** Serena's `AsyncCollector/_collect` is unambiguous. Built-in text search for `def _collect` returns 3 matches in collector.py alone. Disambiguation requires line numbers or surrounding context. Built-in `rename_element` also requires a file path and line number to disambiguate. + +- **Atomicity:** Both Serena's and built-in's rename operations are atomic. Serena's move operations are also atomic; built-ins have no equivalent move, so the comparison is moot. For sequential text edits (`search_replace`), partial failure is possible. + +- **Semantic queries vs. text search:** Serena's `find_referencing_symbols` returns only code references with semantic context. Built-in `search_project` returns all text matches. Neither is strictly better — they answer different questions. + +- **External dependency lookup:** Serena can search and inspect symbols in installed packages via `search_deps=true`. Built-ins have no access to dependency source without manual virtualenv/site-packages navigation. This depends on the IDE having indexed the project's interpreter. + +**Verdict:** For rename, correctness guarantees are equivalent between toolsets. Serena's unique correctness advantage is in move-refactoring atomicity and external dependency access. + +--- + +## 6. Workflow Effects Across a Session + +- **Multi-edit stability:** Serena's name-path addressing remains valid across edits within a session. Built-in line numbers go stale after any edit that changes line counts. In a session with 10+ edits to the same file, Serena eliminates ~5-10 re-read calls that built-ins would need to refresh line numbers. + +- **Exploration → edit transition:** After exploring with Serena (find_symbol, type_hierarchy), the same name paths can be used directly for edits (replace_symbol_body, insert_after_symbol). Built-in exploration (get_file_structure) produces line numbers that must be used immediately before they go stale. + +- **Cross-file refactoring chains:** A rename is equivalent on both sides (1 call each). A move followed by a delete is 2 Serena calls vs. ~10+ built-in calls. The advantage compounds for move-heavy refactoring but not for rename-heavy refactoring. + +- **Session overhead:** Serena requires project activation (1 call at session start). The project can deactivate unexpectedly (observed once during this evaluation), requiring reactivation. This is a minor but real friction cost. + +**Verdict:** Serena's advantages compound for move-refactoring and exploration-heavy sessions; for rename-heavy sessions, both toolsets are equivalent. + +--- + +## 7. Unique Capabilities (No Built-In Equivalent) + +| Capability | Frequency | Impact per use | +|-----------|-----------|---------------| +| Move symbol between modules (with import updates) | Occasional | Saves ~7 calls, eliminates manual import tracking | +| Move file with import updates | Occasional | Saves ~3 calls | +| Transitive type hierarchy | A few times per exploration session | Provides information unavailable to built-ins | +| External dependency symbol lookup | Occasional | Provides information unavailable to built-ins | +| Safe delete with usage guard | Occasional | Prevents accidental breakage | +| Propagated delete (delete + remove call sites) | Rare | Saves multiple manual edits | +| Inline symbol | Rare | Saves multiple manual edits | +| Scope-precise symbol targeting by name path | Continuous (every symbol interaction) | Eliminates disambiguation overhead | + +**Verdict:** Serena provides 5-6 capabilities with no practical built-in equivalent, concentrated in move-refactoring, semantic navigation, and safe/propagated delete. + +--- + +## 8. Tasks Outside Serena's Scope (Built-In Only) + +| Task | Share of typical session | +|------|------------------------| +| Reading/editing non-code files (configs, docs, changelogs, notebooks) | ~15-20% | +| Free-text search (log strings, URLs, magic constants) | ~10% | +| Terminal commands (git, test runners, build tools) | ~15-20% | +| Creating new files from scratch | ~5-10% | +| Directory listing and repo navigation | ~5% | +| Running and debugging applications | ~10-15% | + +Estimated share of session where Serena's augmentation applies: **~35-50%** (code navigation, symbol editing, refactoring). The remainder is built-in-only territory. + +**Verdict:** Serena augments roughly a third to half of typical session work; the rest is inherently outside its scope and handled by built-ins. + +--- + +## 9. Practical Usage Rule + +| Task type | Use | +|-----------|-----| +| Cross-file rename | Either (both provide semantic rename in 1 call) | +| Move symbol/file with import updates | Serena (unique capability) | +| Safe/propagated delete | Serena (unique capability) | +| Inline symbol | Serena (unique capability) | +| Type hierarchy, dependency lookup | Serena (unique capability) | +| Find code references to a symbol | Serena (`find_referencing_symbols`) | +| Find text mentions anywhere | Built-in (`search_project` / Grep) | +| Small edit (1-3 lines) | Built-in (`search_replace`) | +| Full method rewrite | Either (comparable) | +| Insert at structural location | Serena (`insert_after_symbol`) if name path known; built-in otherwise | +| Read non-code files | Built-in (`open` / Read) | +| Terminal operations | Built-in (bash) | +| New file creation | Built-in (`create`) | +| Chained edits in one file | Either; Serena if line-number staleness is a concern | + +**Verdict:** Use Serena for move-refactoring, semantic navigation, and safe delete; use either toolset for rename; use built-ins for text-level edits, non-code files, and terminal operations.