diff --git a/README.md b/README.md index 39108a5a..0473535a 100644 --- a/README.md +++ b/README.md @@ -35,22 +35,28 @@ As the ones actually applying Serena's tools, they are in the best position to e We crafted an unbiased evaluation prompt that leads the agent to perform ~20 routine coding tasks, representative of everyday development work, -in order to compare Serena's tools with its own built-ins, measure the differences, and report the results. +in order to estimate the value added by Serena's tools when used alongside its own built-ins. Here's a one-sentence summary of what the agents had to say: -**Opus 4.6 (high effort) in Claude Code on a large Python codebase:** -> "Serena's IDE-backed semantic tools are the single most impactful addition to my toolkit — cross-file renames, moves, and reference lookups that +**Opus 4.6 (high) in Claude Code on a large Python codebase:** +> "Serena's IDE-backed semantic tools are the single most impactful addition to my toolkit – cross-file renames, moves, and reference lookups that would cost me 8–12 careful, error-prone steps collapse into one atomic call, and I would absolutely ask any developer I work with to set them up." **GPT 5.4 (high) in Codex CLI on a Java codebase:** > "As a coding AI agent, I would ask my owner to add Serena because it gives me the missing IDE-level understanding of symbols, references, and refactorings, turning fragile text surgery into calmer, faster, more confident code changes where semantics matter." -Give your agent the tools it has been asking for and add Serena MCP to your client! +**GPT 5.4 (medium) in Copilot CLI on a large, multi-language monorepo:** +> "As a coding agent, I’d absolutely ask my owner to add Serena because it makes me noticeably sharper and calmer on +real code – especially symbol-aware navigation, cross-file refactors, and monorepo dependency jumps – while I still lean +on built-ins for tiny text edits and non-code work." -See our [documentation](https://oraios.github.io/serena/04-evaluation/000_evaluation-intro.html) for the full methodology and much more detailed evaluation results -beyond these brief summaries, or run your own evaluation on a project of your choice. +Different agents in different settings independently converge on the same verdict. + +_Give your agent the tools it has been asking for and add Serena MCP to your client!_ + +See our [documentation](https://oraios.github.io/serena/04-evaluation/000_evaluation-intro.html) for the full methodology and much more detailed evaluation results, or run your own evaluation on a project of your choice. ## How Serena Works diff --git a/docs/02-usage/030_clients.md b/docs/02-usage/030_clients.md index e608b272..531d750b 100644 --- a/docs/02-usage/030_clients.md +++ b/docs/02-usage/030_clients.md @@ -367,8 +367,10 @@ Copilot should now show that Serena is running, though you may have to restart i ## JetBrains Junie -For the Junie plugin in JetBrains IDEs, go to the three dots in the top right corner, -then Settings / MCP Settings and add Serena to Junie's global MCP server configuration: +For the Junie plugin in JetBrains IDEs you can add Serena either to the global configuration in `~/.junie/mcp/mcp.json` +or to the project configuration in `/.junie/mcp/mcp.json`. Important, don't add both! +In both cases the entry should be: + ```json { @@ -385,14 +387,19 @@ then Settings / MCP Settings and add Serena to Junie's global MCP server configu } ``` -You will have to prompt Junie to "Activate the current project using serena's activation tool" at the -start of each session. +With the global configuration, Serena will be available in all projects. However, +within the Junie plugin, projects will not be automatically activated in Serena. +You may thus have to prompt +Junie to "Activate the current project using serena's activation tool" at the start of each session (though some models are +smart enough to activate the project automatically). + +With the project-scoped configuration, Serena will be available only in that project, and the project will automatically +be recognized as active by Serena. -For Junie CLI, add the same setting to `~/.junie/mcp/mcp.json`. ## JetBrains AI Assistant -Go to Settings / Tools / AI Assistant / MCP and add a new **global** configuration: +Go to Settings / Tools / AI Assistant / MCP and enter the following configuration: ```json { @@ -409,7 +416,8 @@ Go to Settings / Tools / AI Assistant / MCP and add a new **global** configurati } ``` -Then make sure to configure the working directory to be the project root. +Like for Junie, you have the choice between the global and the project-scoped configuration, +with the same trade-off. ## Antigravity diff --git a/docs/04-evaluation/000_evaluation-intro.md b/docs/04-evaluation/000_evaluation-intro.md index 506c60d3..64efedee 100644 --- a/docs/04-evaluation/000_evaluation-intro.md +++ b/docs/04-evaluation/000_evaluation-intro.md @@ -1,15 +1,19 @@ # Evaluation -> **Claude Code (Opus 4.6):** "Serena's IDE-backed semantic tools are the single most impactful addition to my toolkit — cross-file renames, moves, and reference lookups that would cost me 8–12 careful, error-prone steps collapse into one atomic call, and I would absolutely ask any developer I work with to set them up." +> **Claude Code (Opus 4.6, medium):** "Serena's IDE-backed semantic tools are the single most impactful addition to my toolkit — cross-file renames, moves, and reference lookups that would cost me 8–12 careful, error-prone steps collapse into one atomic call, and I would absolutely ask any developer I work with to set them up." -> **Codex (GPT 5.4):** "As a coding agent, I would ask my owner to add Serena because it turns fragile text-and-line-number work into precise symbol-aware navigation and refactoring, which makes real code changes feel faster, safer, and far less blind." +> **Codex (GPT 5.4, high):** "As a coding agent, I would ask my owner to add Serena because it turns fragile text-and-line-number work into precise symbol-aware navigation and refactoring, which makes real code changes feel faster, safer, and far less blind." -These are not marketing quotes — they are the agents' own one-sentence verdicts after -performing hands-on tasks each, using both Serena's tools and their built-in +> **Codex (GPT 5.4, medium):** "As a coding agent, I’d absolutely ask my owner to add Serena because it makes me noticeably sharper and calmer on +real code – especially symbol-aware navigation, cross-file refactors, and monorepo dependency jumps – while I still lean +on built-ins for tiny text edits and non-code work." + +These are not marketing quotes – they are the agents' own one-sentence verdicts after +performing hands-on tasks, using both Serena's tools and their built-in alternatives side by side, on real codebases (large Python library in the case of Claude Code, medium-sized Java project in the case of Codex). -Both agents independently converge on the same core finding: Serena's strongest +Different agents in different settings independently converge on the same core finding: Serena's strongest contribution is collapsing multi-file, semantically-aware operations into single atomic calls, while built-in tools remain the better choice for small local edits, text search, config files, and shell work. diff --git a/docs/04-evaluation/030_results/000_evaluation-results.md b/docs/04-evaluation/030_results/000_evaluation-results.md index e0b6bfc6..eecd29e8 100644 --- a/docs/04-evaluation/030_results/000_evaluation-results.md +++ b/docs/04-evaluation/030_results/000_evaluation-results.md @@ -12,6 +12,7 @@ evaluation can easily be repeated with the LSP-based backend to assess its subse - [Claude Code (Opus 4.6) on a large Python codebase](010_cc_on_tianshou) - [Codex (GPT 5.4) on a Java codebase](020_codex_on_jbplugin) +- [Copilot CLI (GPT 5.4) on a large, multi-language monorepo](030_copilot_cli_on_ente.md) You can run your own evaluation on a project of your choice by reusing our [evaluation prompt](../020_prompts/010_evaluation-prompt.md). diff --git a/docs/04-evaluation/030_results/010_cc_on_tianshou.md b/docs/04-evaluation/030_results/010_cc_on_tianshou.md index 8b0b53fe..763f5f6f 100644 --- a/docs/04-evaluation/030_results/010_cc_on_tianshou.md +++ b/docs/04-evaluation/030_results/010_cc_on_tianshou.md @@ -4,7 +4,7 @@ **Codebase:** [Tianshou](https://github.com/thu-ml/tianshou) — a Python reinforcement learning library (~26K lines, 43 source files) ::: -# Claude Code (Opus) +# Claude Code (Opus 4.6, medium) > **One-line summary:** Serena's IDE-backed semantic tools are the single most impactful addition to my toolkit — cross-file renames, moves, and reference lookups that would cost me 8–12 careful, error-prone steps collapse into one atomic call, and I would absolutely ask any developer I work with to set them up. diff --git a/docs/04-evaluation/030_results/020_codex_on_jbplugin.md b/docs/04-evaluation/030_results/020_codex_on_jbplugin.md index 27fa09f3..ee0f78d8 100644 --- a/docs/04-evaluation/030_results/020_codex_on_jbplugin.md +++ b/docs/04-evaluation/030_results/020_codex_on_jbplugin.md @@ -4,7 +4,7 @@ **Codebase:** Serena JetBrains Plugin (Java) ::: -# Codex (GPT-5.4) +# Codex (GPT-5.4, high) > **One-line summary:** As a coding agent, I would ask my owner to add Serena because it turns fragile text-and-line-number work into precise symbol-aware navigation and diff --git a/docs/04-evaluation/030_results/030_copilot_cli_on_ente.md b/docs/04-evaluation/030_results/030_copilot_cli_on_ente.md new file mode 100644 index 00000000..bef518e2 --- /dev/null +++ b/docs/04-evaluation/030_results/030_copilot_cli_on_ente.md @@ -0,0 +1,348 @@ +:::{admonition} Evaluation Result +:class: note +**Generated by:** GPT-5.4 (medium) in Copilot Cli +**Codebase:** [ente](https://github.com/ente-io/ente) - a large monorepo in Dart, TypeScript, Go, Rust, and other languages. +::: + +# Copilot CLI (GPT-5.4, medium) + +> As a coding agent, I’d absolutely ask my owner to add Serena because it makes me noticeably sharper and calmer on +real code—especially symbol-aware navigation, cross-file refactors, and monorepo dependency jumps—while I still lean +on built-ins for tiny text edits and non-code work + + +**Baseline.** I started from source code only, avoided repo docs/notes, and ran each reversible experiment against the repo directly. After every edit/refactor experiment, the working tree returned to its baseline state. + +## 1. Headline: what Serena changes + +Serena changes the workflow **when the task is about code symbols rather than raw text**. In this repo, the practical delta was: + +1. **Added capability / materially better workflow.** Symbol-aware navigation and refactors in TypeScript: structural overviews, precise code-only references, hierarchy queries, symbol-targeted rename, file move with import updates, and inline. These usually collapsed a 2-6 step built-in chain into **one semantic operation after discovery**, and reduced manual scope verification. +2. **Applies but offers little or no improvement.** Small local edits inside an already-understood method. Built-ins can patch only the changed lines; Serena's body replacement resends the whole symbol, so it was often **less payload-efficient** for 1-3 line tweaks. +3. **Outside Serena's scope.** Non-code reads, free-text search, git inspection, config/package files, and other text-first tasks. Built-ins remained the natural tools there. + +Two important observed limits constrained Serena's delta here: **the strongest measured gains were concentrated in the TypeScript desktop app and the Rust core crate where I ran the hands-on comparisons**, and **some refactors still carried diff-shape tradeoffs** such as formatting churn or unexpected target-file choices. + +**Verdict:** In this repo, Serena was a strong TypeScript symbol layer on top of the built-ins, not a general replacement for text/file work. + +## 2. Added value and differences by area + +| Area | What changed vs built-ins | Frequency | Value per hit | +| --- | --- | --- | --- | +| **Cross-file symbol refactors** | `rename`, file `move`, and `inline` turned manual search/edit/update chains into one semantic op. The `wait -> delay` rename updated 4 files from 1 symbol definition; moving `http.ts` updated the importing file automatically. | Medium | High: typically **2-5 calls saved** plus less manual scope checking | +| **Code-only discovery** | Symbol overview, symbol body retrieval, reference search, and type hierarchy returned code structure directly instead of raw text matches. For `wait`, Serena returned **3 real code-use files**; `rg` returned **7 files**, including docs/comments and English-word hits. | High | Medium: usually **1-3 follow-up reads/filters avoided** | +| **Stable addressing** | Name paths stayed reusable across multiple edits (`createMainWindow`, `openStreetMapUserAgent`, `wait`); built-in line ranges had to be reacquired after edits. | Medium | Medium: less re-reading, less stale-context risk | +| **Small in-method edits** | Serena was not more efficient. Replacing `AutoLauncher/toggleAutoLaunch` required resending the full method body, while the built-in patch changed only the touched lines. | High | Low negative: built-ins used **smaller edit payloads** | +| **External dependency lookup in a monorepo** | Once indexing was available, Serena resolved Electron types from `desktop/node_modules`, `next-electron-server` declarations from the desktop package, and Rust crate symbols from Cargo registry sources. In a monorepo, that removes a manual "which package owns this dependency?" step. | Medium | Medium-High: usually **1-3 searches plus path discovery avoided** | + +**Verdict:** Serena's highest-value delta was semantic refactoring plus dependency-aware code lookup in the TypeScript/Rust parts of the monorepo; its weakest area was tiny local edits. + +## 3. Detailed evidence, grouped by capability + +### 3.1 Codebase understanding + +#### Task 1: high-level repository overview + +- **Attempted:** top-level layout and likely code-heavy areas. +- **Serena chain:** `serena-list_dir(.)` -> directory list. +- **Built-in chain:** not separately needed; Serena added no unique value beyond a normal directory listing here. +- **Payloads:** one short directory listing either way. +- **Finding:** **no meaningful delta**; this is plain filesystem exploration. + +**Verdict:** For repo layout, Serena was neutral. + +#### Task 2: structural overview of a large file, plus the concrete next step + +- **Target file:** `desktop/src/main.ts` (753 lines). +- **Serena chain:** `get_symbols_overview(main.ts, depth=1)` -> concise symbol map of top-level functions and nested locals under `main`; next step `find_symbol(createMainWindow, include_body=true)`. +- **Built-in chain:** `rg` on `const|function|class|export` in `main.ts` -> flat text hits; next step `view` of lines `331-439` to read `createMainWindow`. +- **Payloads observed:** + - Serena overview output: compact symbol list for the file; next-step body fetch returned only the selected symbol body. + - Built-in overview output: many matching lines without structure; next-step read required **~109 lines** of file content. +- **Delta:** Serena's overview was not just shorter; it also supplied **stable symbol names** for the follow-up call. Built-ins could answer the question, but only after a second text-localizing step. + +**Verdict:** Serena materially improved the "overview -> inspect one function" flow by making the follow-up call symbol-based instead of line-based. + +#### Task 3: retrieve a specific class method body without reading the surrounding file + +- **Target symbol:** `AutoLauncher/toggleAutoLaunch` in `desktop/src/main/services/auto-launcher.ts`. +- **Serena chain:** `find_symbol(AutoLauncher/toggleAutoLaunch, include_body=true)` -> exact method body. +- **Built-in chain:** `view` of the relevant file range (`20-40`) after locating the method. +- **Payloads observed:** + - Serena returned the **10-line method body** only. + - Built-in read returned **21 lines** of surrounding class context. +- **Delta:** Serena saved one localization step and avoided unrelated lines. + +**Verdict:** Serena added a real but modest efficiency gain for targeted method retrieval. + +#### Task 4: find all references for one non-trivial symbol; compare code-use recall/precision vs text mention search + +- **Target symbol:** `wait` in `desktop/src/main/utils/common.ts`. +- **Serena chain:** `find_referencing_symbols(wait)` -> 3 files with symbol contexts: `main.ts`, `ffmpeg-worker.ts`, `ml-worker.ts`. +- **Built-in chain:** `rg \bwait\b desktop/src` -> 7 files including: + - real uses/imports, + - comments/doc strings (`preload.ts`, `watch.ts`, `main.ts` prose), + - the definition itself, + - doc mentions in `desktop/docs/release.md` when searching broader scope. +- **Payloads observed:** + - Serena: one structured result grouped by referencing symbol. + - Built-ins: one broader result, but it needed human filtering to answer "who uses this in code?" +- **Delta:** Serena improved **precision**, not just convenience. For the code-only question, built-ins needed extra filtering or extra reads. + +**Verdict:** Serena clearly improved reference search when the question is semantic ("who uses this in code?") rather than textual. + +#### Task 5: supertypes / subclasses / implementations + +- **Equivalent used:** interface hierarchy in `web/apps/ensu/src/services/llm/inference.ts`, because this TS area had interface implementations rather than rich class inheritance. +- **Serena chain:** `type_hierarchy(InferenceBackend, both)` -> `WasmInference`, `TauriInference`; `type_hierarchy(WasmInference, both)` -> supertype `InferenceBackend`. +- **Built-in chain:** `rg InferenceBackend|implements InferenceBackend` -> manual reconstruction from four text matches. +- **Payloads observed:** Serena returned the hierarchy directly; built-ins returned only raw declarations/usages. +- **Delta:** Serena removed the manual synthesis step. Built-ins were sufficient here because the hierarchy was shallow, but that was because the example was small. + +**Verdict:** Serena added moderate value for hierarchy queries; the value grows with hierarchy depth. + +#### Task 6: external dependency symbol lookup + +- **Targets used after indexing was available:** `BrowserWindow` and `serveNextAt` in the desktop TypeScript app, plus `Url` and `Zeroizing` in `rust/core`. +- **Serena chain (TS):** + - `find_declaration(new BrowserWindow(...), include_body=true)` -> `desktop/node_modules/electron/electron.d.ts`, body `class BrowserWindow extends Electron.BrowserWindow {}` + - `find_declaration(import serveNextAt ... , include_body=true)` -> `desktop/node_modules/next-electron-server/index.d.ts`, body `declare function serveNextAt(uri: string, options?: Options): void;` + - `find_symbol(..., search_deps=true)` on those dependency files returned dependency-side docs. +- **Serena chain (Rust):** + - `find_declaration(use reqwest::{Response, Url};, include_body=true)` -> `` external symbol `Url[0]` with the struct body + - `find_declaration(use zeroize::Zeroizing;, include_body=true)` -> `` external symbol `Zeroizing[0]` with the struct body + - `find_symbol(..., relative_path=, search_deps=true)` returned dependency-side docs for those external symbols. +- **Built-in equivalent chain:** + - Manually infer the correct monorepo-local dependency root (`desktop/node_modules`, not repo root), + - or manually inspect Cargo metadata / `Cargo.lock`, + - then open the resolved dependency files directly (for Rust, under the Cargo registry). +- **Payloads observed:** Serena returned the declaration target and a small signature/body directly; built-ins required **package-root discovery first**, which is a real extra step in a monorepo. +- **Delta:** Serena **does add capability and efficiency here** once indexing exists. The gain is larger in this monorepo than in a single-package repo because dependency ownership is split across package-local Node dependencies and shared Cargo registry sources. + +**Verdict:** With indexing available, Serena added meaningful external dependency lookup, and the value was amplified by the monorepo layout. + +### 3.2 Single-file edits + +#### Task 7a: small tweak (1-3 lines inside a method) + +- **Change:** rename local `autoLaunch` -> `launcher` inside `AutoLauncher/toggleAutoLaunch`. +- **Built-in chain:** `view(auto-launcher.ts, 20-40)` -> `apply_patch` on 3 changed lines -> `git diff`. +- **Serena chain:** `find_symbol(toggleAutoLaunch, include_body=true)` -> `replace_symbol_body(toggleAutoLaunch)` -> `git diff`. +- **Payloads observed:** + - Built-ins: read **21 lines**, patch changed **3 logical lines**. + - Serena: fetched **10-line body**, resent **full 10-line body**. +- **Delta:** same result, same number of main steps, but the symbolic edit resent untouched lines. + +**Verdict:** For tiny in-method tweaks, built-ins were more payload-efficient and Serena added no real workflow advantage. + +#### Task 7b: medium rewrite (~10-30 lines) + +- **Change:** rewrite `uniqueSavePath` to use `candidatePath` + `for` loop. +- **Built-in chain:** `view(main.ts, 500-540)` -> `apply_patch` replacing the function body -> `git diff`. +- **Serena chain:** `find_symbol(uniqueSavePath, include_body=true)` -> `replace_symbol_body(uniqueSavePath)` -> `git diff`. +- **Payloads observed:** + - Built-ins: read **41 lines** to safely anchor a **~10-line** rewrite. + - Serena: fetched the **11-line symbol body** and resent the rewritten body only. +- **Delta:** Here Serena was more efficient: less prerequisite read volume and no dependence on surrounding file context. + +**Verdict:** Serena was better for medium symbol-sized rewrites. + +#### Task 7c: large / whole-body rewrite + +- **Change:** rewrite the entire `createMainWindow` body. +- **Built-in chain:** `view(main.ts, 331-439)` -> `apply_patch` replacing the function body -> `git diff`. +- **Serena chain:** `find_symbol(createMainWindow, include_body=true)` -> `replace_symbol_body(createMainWindow)` -> `git diff`. +- **Payloads observed:** + - Built-ins: read **~109 lines** and patched the whole function. + - Serena: fetched the **same symbol body** and resent the whole rewritten body. +- **Delta:** Serena still avoided a file-range read, but once the symbol itself dominates the payload, the token gap mostly disappears. + +**Verdict:** For whole-body rewrites, Serena's gain was modest: better addressing, not dramatically smaller payload. + +#### Task 8: insert a new function at a structural location + +- **Insertion:** `waitSeconds` after `wait` in `desktop/src/main/utils/common.ts`. +- **Built-in chain:** `view(common.ts, 1-40)` -> `apply_patch` inserting after the existing function. +- **Serena chain:** `find_symbol(wait)` -> `insert_after_symbol(wait)`. +- **Payloads observed:** + - Built-ins: read **26 lines** to place a **1-line** function. + - Serena: no extra file-range read once the symbol name was known. +- **Delta:** Serena made the location structural instead of textual. + +**Verdict:** Serena improved insertions when the location is "after symbol X" rather than "after line Y". + +#### Task 9: rename a private helper used only within one file + +- **Target symbol:** `openStreetMapUserAgent` in `desktop/src/main.ts`. +- **Built-in chain:** `view`/`rg` to find call site + definition -> `apply_patch` updating both. +- **Serena chain:** `rename(openStreetMapUserAgent -> buildOpenStreetMapUserAgent)`. +- **Payloads observed:** + - Built-ins: two textual sites had to be found and updated manually. + - Serena: one rename call, terse success response (`"Success"`). +- **Delta:** small call-count improvement, larger correctness improvement when the file is bigger or the name is less unique. + +**Verdict:** Serena was somewhat better even for single-file private renames because it removed manual site enumeration. + +### 3.3 Multi-file changes + +#### Task 10: rename a symbol across several files including imports + +- **Target symbol:** `wait` -> `delay`. +- **Built-in chain:** read `common.ts`, `main.ts`, `ffmpeg-worker.ts`, `ml-worker.ts` -> one multi-file `apply_patch` -> `git diff`. +- **Serena chain:** `rename(wait -> delay)` on the defining symbol -> `git diff`. +- **Payloads observed:** + - Built-ins: required reading **4 files** and manually updating export, imports, and call sites. + - Serena: one semantic rename updated the same 4 files. +- **Success signals:** built-ins only showed success via resulting diff; Serena returned `"Success"`. +- **Delta:** this is one of Serena's clearest wins: same final diff, far less manual scope work. + +**Verdict:** Serena strongly improved multi-file renames by collapsing discovery + edit into one symbol-based refactor. + +#### Task 11: move a symbol from one module to another, updating imports + +- **Target:** move `nullToUndefined` out of `common.ts`. +- **Serena chain executed:** `move(nullToUndefined, target_relative_path=http.ts)` -> `git diff`. +- **Observed result:** Serena removed the symbol from `common.ts`, updated `ffmpeg-worker.ts`, but created a **new file** `desktop/src/main/utils/nullToUndefined.ts` instead of merging into `http.ts`. +- **Built-in equivalent:** would require manually copying the symbol into the intended target module, updating imports, then deleting the old definition. +- **Delta:** Serena still automated the cross-file update, but **did not provide the specific "move into existing module" behavior I was testing**. + +**Verdict:** Serena added partial value for symbol moves here, but not the full capability of "move into a chosen existing TS file". + +#### Task 12: move a file/package and update imports + +- **Target:** move `desktop/src/main/utils/http.ts` to `desktop/src/main/services/http.ts`. +- **Serena chain:** `move(file http.ts -> services/)` -> `git diff`. +- **Observed result:** the file was renamed/moved and `ffmpeg-worker.ts` import updated from `../utils/http` to `./http`. +- **Built-in equivalent:** locate all imports, move the file, patch each import path, then verify. +- **Delta:** this was a real one-call semantic file move. + +**Verdict:** Serena materially improved file moves that require import updates. + +#### Task 12 (safe delete with no remaining usages) + +- **Attempted:** searched for naturally unused TS symbols in the working areas (`main.ts`, `common.ts`, `temp.ts`, `inference.ts`) and checked several candidates (`registerForEnteLinks`, `minimumWindowSize`, `AutoLauncher/isEnabled`, `openStreetMapUserAgent`, `safeJson`, `buildSamplingConfig`). +- **Observed result:** every plausible candidate still had live references. +- **Outcome:** **no suitable candidate found in the TS areas where Serena was operational**, so I skipped this comparison instead of forcing an invalid input. + +**Verdict:** No evidence either way here because the repo did not offer a clean unused-symbol candidate in the code areas Serena handled reliably. + +#### Task 13: delete a symbol and propagate deletion to call sites + +- **Attempted:** looked for a helper whose call sites could be semantically removed rather than inlined or manually rewritten. +- **Observed result:** the good candidates in this repo were better modeled as **inline** refactors, not delete-with-propagation. +- **Outcome:** **no suitable candidate**; skipped rather than using an unsafe input. + +**Verdict:** No measured delta here because the available candidates were inline candidates, not safe propagate-delete candidates. + +#### Task 13 (inline a small helper) + +- **Target symbol:** `waitForRendererDevServer`. +- **Built-in chain:** `view` call site + definition -> `apply_patch` replacing `await waitForRendererDevServer()` with `await wait(1000)` and deleting the helper -> `git diff`. +- **Serena chain:** `inline(waitForRendererDevServer, keep_definition=false)` -> `git diff`. +- **Observed result:** both achieved the inline, but Serena also rewrote unrelated import formatting at the top of the file. +- **Success signals:** built-ins: resulting diff; Serena: `{"status":"SUCCESS"}`. +- **Delta:** Serena added the unique semantic refactor, but in this run it also introduced **format churn outside the logical change**. + +**Verdict:** Serena added real inline capability, with a low-frequency but real tradeoff of broader formatting churn. + +### 3.4 Reliability & correctness-oriented checks + +#### Task 14: scope precision + +- **Demonstrated with:** `AutoLauncher/toggleAutoLaunch`, `openStreetMapUserAgent`, and `InferenceBackend`. +- **Serena:** symbol names and name paths targeted the exact code entity. +- **Built-ins:** text search for names such as `wait` or `writeToTemporaryFile` over-matched comments, docs, and multiple textual occurrences. +- **Delta:** Serena's unit of work was the symbol; built-ins' unit was the matching line. + +**Verdict:** Serena was reliably more precise whenever the target was a symbol rather than a string. + +#### Task 15: atomicity + +- **Observed:** Serena rename/file-move/inline each ran as one refactor operation after symbol selection. +- **Built-ins:** a single `apply_patch` can update multiple files atomically as a patch, but it **cannot discover missed sites**; semantic completeness remains manual. +- **Delta:** Serena's advantage was not transactional all-or-none patching; it was **scope computation**. + +**Verdict:** Serena improved semantic completeness more than patch atomicity. + +#### Task 16: success signals + +- **Observed Serena success outputs:** `OK` for body replacement, `"Success"` for rename, JSON result for move, `{"status":"SUCCESS"}` for inline. +- **Observed built-in success outputs:** only indirect evidence via `git diff` / clean revert. + +**Verdict:** Serena gave clearer machine-readable success signals for refactors than the built-ins did. + +## 4. Token-efficiency analysis + +### By edit size + +| Edit size | Built-ins | Serena | More efficient | +| --- | --- | --- | --- | +| **Small tweak** (`toggleAutoLaunch`) | Read ~21 lines, patch only changed lines | Fetch 10-line body, resend full 10-line body | **Built-ins** | +| **Medium rewrite** (`uniqueSavePath`) | Read ~41 lines to safely patch ~10 lines | Fetch 11-line body, resend 11-line body | **Serena** | +| **Large rewrite** (`createMainWindow`) | Read ~109 lines, patch whole body | Fetch ~same symbol body, resend whole body | **Near tie**, slight Serena advantage only from structural targeting | +| **Cross-file rename** (`wait -> delay`) | Read 4 files, craft 4-file patch | One rename after discovery | **Serena by a large margin** | + +### Forced reads + +- Built-ins often needed a **localization read** before the edit. +- Serena avoided that when the symbol was already known, but not when the task itself required understanding the body. + +### Stable vs ephemeral addressing + +- Serena's addresses (`createMainWindow`, `wait`, `openStreetMapUserAgent`) stayed useful across later operations. +- Built-in line ranges from `view` became stale after edits, so later text operations required re-grepping or re-viewing. + +**Verdict:** Serena is most token-efficient for medium-to-large symbol work and cross-file refactors; built-ins stay leaner for tiny local edits. + +## 5. Reliability & correctness (under correct use) + +- **Precision of matching:** Serena's reference search answered "who uses this in code?" better than `rg`, which mixed real uses with prose/comment matches. +- **Scope disambiguation:** Serena targeted exact symbols (`AutoLauncher/toggleAutoLaunch`, `InferenceBackend`) rather than relying on unique text strings. +- **Atomicity:** Serena computed and updated semantic scope in one refactor call; built-ins could batch edits, but only after manual scope discovery. +- **Semantic queries vs text search:** hierarchy and references were the strongest examples. Built-ins could reconstruct them, but only with manual interpretation. +- **External dependencies:** after indexing was available, Serena resolved desktop TypeScript dependencies into package-local declaration files under `desktop/node_modules` and Rust dependencies into external Cargo sources such as `url` and `zeroize`. Built-ins could still reach those files, but only after manual package-root or registry-path discovery. +- **Monorepo effect:** this repo magnified Serena's dependency-lookup value because "the dependency source" was not at one obvious global root. Serena jumped from app code to the right package-local or registry-backed dependency context directly. + +**Verdict:** Serena improved correctness by narrowing work to exact symbols and by resolving dependencies across monorepo boundaries. + +## 6. Workflow effects across a session + +- **Advantages compounded** when I stayed in symbol space. Example: `get_symbols_overview(main.ts)` produced symbol names that I later reused for `find_symbol(createMainWindow)`, `rename(openStreetMapUserAgent)`, and `inline(waitForRendererDevServer)`. +- **Built-in workflows required refreshes**. Across repeated `main.ts` experiments, I repeatedly had to reacquire ranges with `view`/`rg` before editing because prior line-based context was no longer trustworthy. +- **In the monorepo, Serena also compounded by removing package-boundary bookkeeping.** In the desktop app I could jump from `main.ts` into Electron and `next-electron-server` declarations without first reasoning about workspace roots; in `rust/core` I could jump into Cargo-registry dependencies through external symbol handles instead of manually reconstructing registry paths from `Cargo.lock`. +- **The compounding effect disappeared** for tiny edits and non-code work, where built-ins were already direct and minimal. +- **One tradeoff compounded too:** some Serena refactors carried formatting side effects (notably `inline`), so the semantic benefit does not guarantee a surgically small diff. + +**Verdict:** Serena's advantages compound most in code-centric monorepo sessions, where symbol reuse and dependency jumps save both re-reading and package-root discovery work. + +## 7. Unique capabilities + +| Capability with no practical one-step built-in equivalent | Frequency | Impact | +| --- | --- | --- | +| **Semantic cross-file rename from a single symbol definition** | Medium | High | +| **Type hierarchy query (implementations / supertypes)** | Low-Medium | Medium | +| **Inline refactor across call sites** | Low | High when applicable | +| **File move with import updates** | Low-Medium | High | +| **External dependency resolution from in-repo code into package-local or registry-backed sources** | Medium | Medium-High | + +Built-ins can approximate all of these manually, but not as a single semantic operation. + +**Verdict:** Serena did add unique practical capabilities, especially around refactors that require scope computation rather than text replacement. + +## 8. Tasks outside Serena's scope (built-in only) + +- Reading non-code files like `desktop/package.json` +- Free-text search such as `ente://app` or URL strings +- Git inspection / diff / cleanup +- Config/package/changelog/docs/notebook reading +- Exact textual patching once the line range is already known + +In this session, these built-in-only tasks were **roughly 40% of the total operational steps by count**, but they were usually the low-complexity steps around the more valuable semantic work. + +**Verdict:** A substantial share of everyday terminal work remains built-in-only, but Serena targets the higher-value symbol-heavy slice rather than the whole session. + +## 9. Practical usage rule + +Use **Serena first** when the task is about a **code symbol** and especially when it spans **multiple files, references, or a whole symbol body**. Use **built-ins first** when the task is about **text, config/docs, free-text search, git state, or a 1-3 line local tweak**. The highest-yield mixed workflow in this repo was: **discover/refactor with Serena, inspect non-code and do tiny patches with built-ins**. + +**Verdict:** Choose Serena for symbol semantics and built-ins for text locality.