diff --git a/.github/workflows/pytest.yml b/.github/workflows/pytest.yml index f1d969e9..45601361 100644 --- a/.github/workflows/pytest.yml +++ b/.github/workflows/pytest.yml @@ -576,3 +576,5 @@ jobs: - name: Type-checking with mypy shell: bash run: uv run poe type-check + - name: Make sure documentation builds + run: uv run poe doc-build diff --git a/CHANGELOG.md b/CHANGELOG.md index 30a9c809..4c0ab4b7 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,6 +6,12 @@ Status of the `main` branch. Changes prior to the next official version change w - Support environment variable `SERENA_USAGE_REPORTING` (set to `false` to disable usage reporting) - Extended the list of always ignored directories (by language servers) with common cases. +JetBrains: + - Improve handling of `relative_path` parameter + - Improve its documentation to avoid usage errors + - Replace escaped characters in `relative_path` with their unescaped counterparts (< and >) + - `FindSymbolTool`: Force `search_deps=True` if `relative_path` pertains to external dependencies. + * Language Servers: - Add mSL (mIRC Scripting Language) support (custom pygls-based language server; symbols, references, definitions) diff --git a/README.md b/README.md index f1b513d1..39108a5a 100644 --- a/README.md +++ b/README.md @@ -28,6 +28,31 @@ more complex codebases. > Do not install Serena via an MCP or plugin marketplace! They contain outdated and suboptimal installation commands. > Instead, follow our [Quick Start](#quick-start) instructions. +## What Our "End Users" Say + +While it is humans who download and set up Serena, our end users are essentially AI agents. +As the ones actually applying Serena's tools, they are in the best position to evaluate Serena. + +We crafted an unbiased evaluation prompt that leads the agent to perform ~20 routine coding tasks, +representative of everyday development work, +in order to compare Serena's tools with its own built-ins, measure the differences, and report the results. + +Here's a one-sentence summary of what the agents had to say: + +**Opus 4.6 (high effort) in Claude Code on a large Python codebase:** +> "Serena's IDE-backed semantic tools are the single most impactful addition to my toolkit — cross-file renames, moves, and reference lookups that +would cost me 8–12 careful, error-prone steps collapse into one atomic call, and I would absolutely ask any developer I work with to set them up." + +**GPT 5.4 (high) in Codex CLI on a Java codebase:** +> "As a coding AI agent, I would ask my owner to add Serena because it gives me the missing IDE-level understanding of symbols, references, and +refactorings, turning fragile text surgery into calmer, faster, more confident code changes where semantics matter." + +Give your agent the tools it has been asking for and add Serena MCP to your client! + +See our [documentation](https://oraios.github.io/serena/04-evaluation/000_evaluation-intro.html) for the full methodology and much more detailed evaluation results +beyond these brief summaries, or run your own evaluation on a project of your choice. + + ## How Serena Works Serena provides the necessary [tools](https://oraios.github.io/serena/01-about/035_tools.html) for coding workflows, @@ -58,7 +83,7 @@ complex projects! There are two alternative technologies powering these capabilities: -* **Language servers** implementing the language server Protocol (LSP) — the free/open-source alternative +* **Language servers** implementing the language server protocol (LSP) — the free/open-source alternative which is used by default. * The **Serena JetBrains Plugin**, which leverages the powerful code analysis and editing capabilities of your JetBrains IDE (paid plugin; free trial available). @@ -231,4 +256,3 @@ Please refer to the [user guide](https://oraios.github.io/serena/02-usage/000_in A significant part of Serena, especially support for various languages, was contributed by the open source community. We are very grateful for the many contributors who made this possible and who played an important role in making Serena what it is today. - diff --git a/docs/02-usage/030_clients.md b/docs/02-usage/030_clients.md index 8280d604..e608b272 100644 --- a/docs/02-usage/030_clients.md +++ b/docs/02-usage/030_clients.md @@ -266,6 +266,15 @@ To set this up, create the file `~/.copilot/hooks/serena-hooks.json` with the fo The `SessionStart` hook also addresses the global configuration limitation mentioned above — it will automatically prompt the agent to activate the project directory, so you no longer need to do this manually. +## Copilot CLI + +Use the interactive `/mcp add` slash command, choose Serena as the name, STDIO as the server type, and +`serena start-mcp-server --context=copilot-cli --project-from-cwd` as command. Copilot CLI will immediately notify you +that Serena is running if everything is set up correctly or display an error otherwise. + +You should add the same **hooks** as in VSCode (see above) if Copilot CLI didn't pick them up automatically. + + ## Codex (CLI and App) You can simply run `serena setup codex`. @@ -356,30 +365,11 @@ Alternatively, add the following to `~/.copilot/mcp-config.json` (create the fil Copilot should now show that Serena is running, though you may have to restart it. -## JetBrains Junie +## JetBrains Junie For the Junie plugin in JetBrains IDEs, go to the three dots in the top right corner, then Settings / MCP Settings and add Serena to Junie's global MCP server configuration: -```json -{ - "mcpServers": { - "serena": { - "command": "serena", - "args": [ - "start-mcp-server", - "--context=junie" - ] - } - } -} -``` - -You will have to prompt Junie to "Activate the current project using serena's activation tool" at the -start of each session. - -For Junie CLI, where the project can be activated automatically on startup, edit `~/.junie/mcp/mcp.json` and add - ```json { "mcpServers": { @@ -395,12 +385,14 @@ For Junie CLI, where the project can be activated automatically on startup, edit } ``` +You will have to prompt Junie to "Activate the current project using serena's activation tool" at the +start of each session. + +For Junie CLI, add the same setting to `~/.junie/mcp/mcp.json`. + ## JetBrains AI Assistant -Here you can set up the more convenient per-project MCP server configuration, as the AI assistant supports specifying -the launch working directory. - -Go to Settings / Tools / AI Assistant / MCP and add a new **local** configuration via the `as JSON` option: +Go to Settings / Tools / AI Assistant / MCP and add a new **global** configuration: ```json { @@ -410,8 +402,7 @@ Go to Settings / Tools / AI Assistant / MCP and add a new **local** configuratio "args": [ "start-mcp-server", "--context=jb-ai-assistant", - "--project", - "$(pwd)" + "--project-from-cwd" ] } } diff --git a/docs/04-evaluation/000_evaluation-intro.md b/docs/04-evaluation/000_evaluation-intro.md new file mode 100644 index 00000000..506c60d3 --- /dev/null +++ b/docs/04-evaluation/000_evaluation-intro.md @@ -0,0 +1,30 @@ +# Evaluation + +> **Claude Code (Opus 4.6):** "Serena's IDE-backed semantic tools are the single most impactful addition to my toolkit — cross-file renames, moves, and reference lookups that would cost me 8–12 careful, error-prone steps collapse into one atomic call, and I would absolutely ask any developer I work with to set them up." + +> **Codex (GPT 5.4):** "As a coding agent, I would ask my owner to add Serena because it turns fragile text-and-line-number work into precise symbol-aware navigation and refactoring, which makes real code changes feel faster, safer, and far less blind." + +These are not marketing quotes — they are the agents' own one-sentence verdicts after +performing hands-on tasks each, using both Serena's tools and their built-in +alternatives side by side, on real codebases (large Python library in the case of Claude Code, +medium-sized Java project in the case of Codex). + +Both agents independently converge on the same core finding: Serena's strongest +contribution is collapsing multi-file, semantically-aware operations into single atomic +calls, while built-in tools remain the better choice for small local edits, text search, +config files, and shell work. + +## Approach + +We gave each agent a single evaluation prompt and pointed it at a codebase. The agent +then performed a wide variety of tasks across five areas (navigation, small edits, large edits, +cross-file refactoring, and workflow effects), executing each task with both toolsets and +recording call counts, payload sizes, and prerequisite steps. Every finding was classified +as either (a) Serena adds capability, (b) Serena applies but offers no improvement, +or (c) outside Serena's scope — a structure that requires reporting negative and neutral +results, not just positive ones. + +The agent evaluates itself. This is deliberate: the agent is the actual user of the tools, +so it can judge workflow improvements from direct experience rather than through proxy +metrics. And because the prompt defines task *categories* rather than fixed tasks, anyone +can rerun the evaluation on their own project with their own agent. diff --git a/docs/04-evaluation/010_methodology.md b/docs/04-evaluation/010_methodology.md new file mode 100644 index 00000000..feb7f282 --- /dev/null +++ b/docs/04-evaluation/010_methodology.md @@ -0,0 +1,160 @@ +# Methodology + +In this section we describe the methodology we applied in evaluating the performance of Serena's tools. + +The evaluation measures the **concrete delta** that Serena's tools provide on top of an agent's +built-in capabilities (file reads, text edits, grep, shell, etc.). +Rather than a simple thumbs-up/thumbs-down, it produces a detailed, evidence-based report +covering capabilities, efficiency and reliability. + +## Design Goals + +We designed an evaluation methodology with three goals: + +1. **Generality.** The evaluation mechanism should be broadly applicable and repeatable by any user, on any project, with any agent. + The evaluation is a single prompt that you give to your agent of choice, pointed at your codebase + of choice, in your client of choice. This means the results directly reflect the value Serena + would add to your actual workflow — not to an artificial benchmark setup. There is nothing to + install, configure, or script beyond what you already have. + +2. **The agent evaluates itself.** + We deliberately let the AI agent be both the executor and the evaluator. + Since most evaluations are strictly quantitative, this may seem unusual, but it is a reasonable choice: + The agent is the actual end user of the + tools, so it is in the best position to judge whether a semantic tool improves its workflow + compared to its built-in alternatives. It can measure call counts, payload sizes, and + prerequisite steps from direct experience rather than from proxy metrics. It also avoids the + problem of a human evaluator having to simulate how an agent would use the tools — the agent + simply uses them and reports what it observes. + +3. **Comprehensive and unbiased by design.** + Rather than selecting specific tasks, the prompt defines *task categories* that systematically + span Serena's capabilities: codebase understanding, single-file edits of varying sizes, multi-file + refactoring, reliability properties, and workflow effects. The agent picks concrete instances + from the codebase at hand, performs each task using both toolsets side by side, and classifies + every finding as a positive delta, a neutral/negative delta, or out of scope. The prompt + explicitly requires reporting negative deltas and cases where Serena offers no improvement, + structurally counterbalancing any tendency to favour the tool being evaluated. + +## Method + +We give an AI coding agent a single, detailed [evaluation prompt](020_prompts/010_evaluation-prompt.md) in a one-shot session. +The prompt instructs the agent to perform approximately 20 hands-on tasks across five areas: + +1. **Codebase understanding** — structural overviews, targeted symbol retrieval, reference finding, type hierarchies, and external dependency lookup. +2. **Single-file edits** — small tweaks, medium rewrites, full-body replacements, insertions, and local renames. +3. **Multi-file changes** — cross-file renames, symbol and file moves, safe deletes, and inlining. +4. **Reliability & correctness** — scope precision, atomicity, and success signals. +5. **Workflow effects** — chained edits, stable vs. ephemeral addressing, and multi-step exploration. + +For every task, the agent executes the full end-to-end workflow using **both** toolsets (Serena's semantic +tools and its own built-in tools), applies real edits verified via `git diff`, and records call counts, +payload sizes and prerequisite steps. Edits are reverted after each experiment to keep the working tree clean. + +The resulting report classifies each finding into one of three categories: + +* **(a)** tasks where Serena adds capability, +* **(b)** tasks where Serena applies but offers no improvement, and +* **(c)** tasks outside Serena's scope. + +Only category (b) constitutes a neutral or negative finding; category (c) is context, not a finding. + +After the evaluation, a separate [follow-up prompt](020_prompts/020_summary-prompt.md) asks the agent for a +one-sentence, user-facing recommendation — the quotes shown on the [main page](https://github.com/oraios/serena). + +## Assessment of the Methodology + +_The following assessment was written by Claude Opus 4.6 (high effort) after reading the full +evaluation methodology, evaluation prompt, and all result documents._ + +**The methodology is sound**, and the two published results demonstrate that it produces meaningful, +detailed evaluations. Both reports follow the prompt's structure faithfully: they perform all ~20 +tasks using both toolsets, record concrete measurements (call counts, payload sizes, prerequisite +steps), and classify findings into the three required categories. Crucially, both agents report +neutral and negative deltas honestly — the Claude Code report explicitly notes that small edits are +more efficient with built-ins (~4.5x less payload), and the Codex report flags that tiny intra-method +changes and simple one-file renames see no benefit from Serena. Neither report reads as promotional; +they read as technical comparisons with quantified evidence on both sides. + +The two reports also validate the methodology's design goal of generalisability: despite being +produced by different agents (Opus 4.6 vs GPT 5.4), on different codebases (Python RL library vs +Java IDE plugin), they converge on the same core findings — cross-file refactoring is Serena's +highest-value contribution, structural navigation provides a moderate advantage, and small local +edits are better handled by built-ins. This convergence across independent runs increases confidence +that the findings reflect genuine properties of the toolset rather than artefacts of a particular +agent or codebase. + +The methodology's core strengths are: + +- **Self-evaluation by the agent is the right design choice.** The agent is the actual consumer of the + tools, so it can report on workflow friction, payload overhead, and call counts from first-hand + experience. A human evaluator would have to guess at these. +- **Task categories instead of fixed tasks** avoid cherry-picking while still ensuring coverage. Letting + the agent pick concrete instances from the codebase at hand means the evaluation naturally adapts to + what the project actually contains (e.g. skipping inline if no suitable candidate exists, as the + Claude Code report did for Python, while the Codex report found a suitable Java candidate). +- **The three-category classification** (adds value / applies but no improvement / out of scope) is the + right framing for an augmentation layer. It prevents the common trap of penalising a tool for not + covering things it was never designed to cover. +- **Reproducibility by users** is a strong differentiator. Anyone can validate the claims on their own + codebase with their own agent. + +The main limitation is scope: the published results cover two agent/codebase combinations using the +JetBrains backend. Single-shot variance means a second run of the same agent could produce different +task selections and slightly different verdicts, and the LSP backend — which most users will +encounter first — has not yet been evaluated. These are practical constraints of cost, not +methodological flaws, and the reproducible design explicitly invites users to fill these gaps +themselves. + +One thing to watch for is whether agents that are less capable than Opus 4.6 or GPT 5.4 produce +meaningful evaluations — the prompt is quite demanding. But that is a question about the agent, not +about the method. + +### Prompt Fairness + +We put significant effort into crafting an unbiased, fair evaluation prompt. +To assess the prompt's fairness, we asked Claude Opus 4.6 to evaluate the prompt itself. + +**Prompt**: +> The attached prompt is designed to evaluate the additive value of Serena as an augmentation layer on top of built-in tools +> (not to compare two competing alternatives). +> Is the methodology suitable for that goal, or does it introduce bias that would distort the measurement of what Serena adds? +> Note that we assume that sufficiently intelligent models use the tools correctly, +> which is why misuse and failure modes are out of scope for this evaluation. +> Do not question the validity of this assumption in your evaluation of the prompt. +> Give a brief answer, maximally two sentences. +> +> \ + +**Claude Opus' Answer**: +> "The methodology is well-suited to its stated goal: by explicitly scoping to correct use, treating Serena as an +> augmentation layer rather than a competitor, and classifying out-of-scope tasks as context rather than negative +> findings, it avoids the most common biases that would distort a delta measurement. The one risk is that the "correct-use +> rule" could subtly favor Serena by preemptively filtering out any scenario where its tools underperform — but the prompt +> mitigates this by requiring category (b) findings (applies but no improvement) and explicitly demanding negative deltas +> be reported, so the bias is acknowledged and structurally counterbalanced." + +According to Claude, the only "biased" aspects are some sentences about misuse of tools, which we consider irrelevant for the purpose of evaluation +(sufficiently intelligent models use the tools correctly). + + +## Why Not Benchmarks? + +Standard coding benchmarks (SWE-bench, HumanEval, etc.) measure an agent's ability to solve +predefined tasks with a known correct answer. They are valuable for comparing models and agents, +but they are a poor fit for evaluating a **tool augmentation layer** like Serena for several reasons: + +- **Benchmarks don't reflect real usage patterns.** Benchmark tasks are typically small, self-contained + problems that can be solved by reading and editing a handful of files. They rarely exercise the + workflows where Serena's tools shine — cross-file refactoring, navigating large codebases by symbol + structure, chaining multiple edits with stable addressing, or querying type hierarchies and + external dependencies. A benchmark score would mostly measure performance on tasks where Serena + is not expected to help. +- **Results would not generalise to the user's project.** Serena's value depends on the codebase + (size, language, complexity), the agent (model, built-in tools), and the client harness + (Claude Code, Codex, IDE plugins, etc.). A fixed benchmark on a fixed codebase with a fixed + agent tells you little about what Serena would add to *your* setup. +- **Predefined tasks bias the measurement.** Choosing specific tasks to evaluate inevitably + introduces selection bias — we would end up picking tasks that either favour or disfavour Serena. + We wanted an evaluation that systematically covers the full surface area of Serena's capabilities + without cherry-picking. \ No newline at end of file diff --git a/docs/04-evaluation/020_prompts/000_prompts.md b/docs/04-evaluation/020_prompts/000_prompts.md new file mode 100644 index 00000000..5033e6fa --- /dev/null +++ b/docs/04-evaluation/020_prompts/000_prompts.md @@ -0,0 +1,6 @@ +# Prompts + +Our evaluation uses two prompts, which are passed to the LLM in order to generate the evaluation results: + +1. the [main evaluation prompt](010_evaluation-prompt), which performs the analysis +2. the [summary prompt](020_summary-prompt), which condenses the results diff --git a/docs/04-evaluation/020_prompts/010_evaluation-prompt.md b/docs/04-evaluation/020_prompts/010_evaluation-prompt.md new file mode 100644 index 00000000..45009c4d --- /dev/null +++ b/docs/04-evaluation/020_prompts/010_evaluation-prompt.md @@ -0,0 +1,333 @@ +# Evaluation Prompt + +We use the prompt below to evaluate the added value of Serena's tools against the +agent's built-in tools on a given project. +The evaluations were created in one-shot sessions, only using this +prompt and the follow-up [prompt for summarization](020_summary-prompt) + +``` +# Evaluate Serena's Tools Against Built-Ins + +You have access to Serena's coding tools alongside your built-in tools (Read, +Edit, Write, Glob, Grep, Bash, etc.). I want a thorough, evidence-based +evaluation of **what Serena's tools add on top of the built-ins**, assuming both +toolsets are used correctly. + +This is an evaluation, not a user guide, and it is not a binary adoption pitch. +Your job is to answer: *if a competent user of both toolsets had only the +built-ins, what concrete capabilities and efficiency differences would they +experience, and by how much?* A reader who finishes your report should have a +clear, specific picture of what Serena adds — as well as where it provides no +meaningful improvement or introduces tradeoffs — in terms of capabilities, +workflows, and efficiency. Not a thumbs-up/thumbs-down, but a sharp description +of the delta. + +Failure modes from misuse, silent-failure traps, gotcha comparisons, and "be +careful of X" warnings are out of scope. They belong in onboarding material for +a developer learning the tools, not in a delta analysis of what the tools add. + +**Describe the measured differences clearly and neutrally.** Avoid generic or +non-informative framing such as "both have their place" unless supported by +concrete findings. If Serena adds substantial capabilities, name and quantify +them. If it adds marginal or no capabilities, say that and show why. If there +are regressions or tradeoffs, include them explicitly. The two toolsets are +complementary — that's a given, not the answer. Serena is an augmentation layer, +not a replacement. Do not penalize it for tasks it was not designed to address — +instead, note those tasks as "built-in only" and move on. The evaluation should +measure what Serena adds where it applies, not what it fails to add where it +doesn't.The answer is a specific list of what Serena contributes (or does not +contribute) to a correct-use workflow relative to built-ins. + +Write the report to serena-evaluation.md in the repo root. + + + +## Ground rules + +### Starting conditions + +- Start fresh. Do not read project memories, CLAUDE.md shortcuts, or prior notes + about the repo. Do not read documentation files either. Explore as if you've + never seen it, focusing on code. +- Use git as your safety net — experiment freely. Any edit can be reverted with + `git checkout -- ` or `git stash`. Run edits for real; don't simulate. A + hands-on comparison is worth far more than a thought experiment. +- After each experiment, verify the working tree is clean with + `git status --short` before moving on. + +### How to compare — correct use only + +- **Correct-use rule.** Evaluate each tool on inputs and tasks it was designed + for, called the way a competent user would call it. A tool doing exactly what + its contract says is not a finding, even if a careless caller could misuse it. +- **Know the contract before you call.** Before invoking any tool, have a + one-sentence understanding of what it does. If you expect an error or "not + applicable," don't make the call. +- **Refactoring semantics are real.** Inlining requires a substitutable function + (typically single-expression, no side effects); moving requires a legal + target; safe-delete requires no surviving usages. If the repo has no suitable + candidate for a given refactoring, report "no suitable candidate in this + codebase" and skip it — don't contrive a broken input. + +### How to compare — workflow level, not single-call level + +- For every task, write out the full end-to-end call chain on each side before + drawing conclusions. Include prerequisite reads and follow-up steps. +- Do not evaluate a tool based on criteria that only arise from mixing workflows + incorrectly. +- Ephemeral addressing is a liability. Line numbers and byte offsets go stale + after edits; stable addressing (name paths) may reduce rework. + +### How to measure + +- Track observations during execution. For every tool call, note: number of + calls, approximate input size, output size, and any prerequisite or + verification steps. +- Separate call count, input payload, output payload, and verification cost as + distinct axes. +- Include prerequisite Reads and post-hoc verification steps in comparisons. + +When a task falls entirely outside Serena's design scope (e.g., reading config +files, small text edits where Edit already sends minimal payload), classify it +as "not applicable" rather than as a negative delta. A negative delta requires +that Serena targets the task and performs worse, not that a tool designed for +something else is suboptimal when misapplied to it. + + + +## Exploration phase — tasks to actually perform + +Work through the following. Each item exercises a specific capability under +correct use; substitute an equivalent if an item isn't applicable. + +### Codebase understanding + +1. Get a high-level overview of the repository structure — top-level layout, + main packages, entry points. +2. Pick one large source file (300+ lines). Get a structural overview of it. Do + it with semantic overview tools and with Glob/Grep/Read. Then write out the + concrete next step on each side and compare the pair of calls, not just the + overview call. +3. Pick a specific method inside a class and retrieve its body without reading + the surrounding file. +4. For one non-trivial symbol, find all references across the codebase. Compare + recall and precision under the question "who uses this in code?" vs "where is + this mentioned anywhere, including docs?" +5. For a class, list its subclasses / implementations and its supertypes, + including transitively. Compare against what text search would need to do. +6. For at least one symbol from an external dependency (a third-party library), + try to retrieve its definition or signature. Note whether each toolset can do + this at all and what infrastructure it requires (environment activation, + site-packages discovery, language-server indexing, etc.). + +### Single-file edits — span the full range of edit sizes + +7a. Small tweak (1–3 lines inside a method). Change an error message or rename a +local variable inside a larger method. Do it with `Edit` and with symbolic body +replacement. Compare payload sent, payload received, and prerequisite reads. + +7b. Medium rewrite (replace ~10–30 lines — most of a method body). Rewrite the +main logic of a method while keeping its signature. Do it both ways. + +7c. Large/whole-body rewrite. Pick a method of 50+ lines and rewrite the entire +body. Do it both ways. + +8. Insert a new function/method at a specific structural location (for example, + right after an existing method). Try both the symbolic-insert path and the + manual Edit path. +9. Rename a private helper used only within one file. Compare doing it by hand + vs. using a semantic rename. + +### Multi-file changes + +10. Rename a symbol (function, class, or method) used across several files + including imports. Compare the semantic path against the built-in equivalent + chain. +11. Move a symbol from one module to another, updating imports at all call + sites. Use the semantic move tool if available; plan the built-in equivalent + honestly. +12. Move a file or package to a different location, updating imports at all call + sites. Use the semantic move tool if available; plan the built-in equivalent + honestly. +12. Delete a symbol safely, checking it has no remaining usages. Compare + search-then-delete with a safe-delete tool. +13. Delete a symbol and propagate the deletion to all call sites. Compare to how + the built-in equivalent would work. +13. Inline a small helper into its call sites — only if the codebase contains a + function that is legally inlinable. If no such candidate exists, report "no + suitable candidate" and skip it. + +### Reliability & correctness under correct use + +14. Scope precision. Demonstrate that semantic tools address symbols by name + path and can target a specific class method, override, or overload that text + search would over-match. +15. Atomicity. A semantic cross-file refactoring is atomic: either all sites are + updated or none. A chain of `Edit` calls is not. +16. Success signals. For each completed refactor, note what each tool returns on + success. + +### Workflow effects across multiple edits + +17. Chain at least three edits in one file. Report what each toolset requires + between edits. +18. Multi-step exploration across the repo. Note whether intermediate results + remain useful across later edits or have to be refreshed. + +### Things where the comparison shouldn't be interesting + +19. Read and understand a non-code file (config, changelog, docs, notebook). + Semantic-code tools don't apply — use `Read`. +20. Search for a free-text pattern across the repo (log string, magic constant, + URL). Use `Grep`. + + + +## Evaluation phase + +Write a report structured for progressive disclosure. + +**Value-weighting is required.** For every contribution or difference you +identify — positive, neutral, or negative — estimate: + +- **Frequency:** how often this arises in typical coding work +- **Value per hit:** calls saved, tokens saved, or correctness impact + +Order findings by **frequency × value-per-hit**, not novelty. + +**Every section must end with a one-sentence verdict** summarizing the practical +takeaway. + + + +### 1. Headline: what Serena changes + +Open with a precise description of the delta Serena provides. Distinguish +between three categories: (a) tasks where Serena adds capability, (b) tasks +where Serena applies but offers no improvement, and (c) tasks outside Serena's +scope. Only category (b) constitutes a neutral or negative finding. Category (c) +is context, not a finding. + +A reader stopping here should understand both what is gained and what is not. + +**Verdict:** (one sentence) + + + +### 2. Added value and differences by area (3–6 bullets) + +Each bullet must describe: + +- What Serena changes relative to built-ins (positive, neutral, or negative) +- Frequency +- Value per hit + +Avoid framing in terms of “wins”; describe concrete differences. + +**Verdict:** (one sentence) + + + +### 3. Detailed evidence, grouped by capability + +Per task: + +- What you attempted +- Full call chain on both sides +- Payloads sent and received + +Include cases where: + +- Serena is better +- Built-ins are better +- No meaningful difference exists + +End each subsection with a verdict. + + + +### 4. Token-efficiency analysis + +Address: + +- Payload differences across edit sizes +- Forced reads +- Stable vs ephemeral addressing + +Include cases where each toolset is more efficient. + +**Verdict:** (one sentence) + + + +### 5. Reliability & correctness (under correct use) + +Address: + +- Precision of matching +- Scope disambiguation +- Atomicity +- Semantic queries vs text search +- External dependency symbol lookup and what setup it depends on + +Include both strengths and limitations of each toolset. + +**Verdict:** (one sentence) + + + +### 6. Workflow effects across a session + +Evaluate multi-step workflows and whether advantages compound or diminish. +Include neutral or negative findings where applicable. + +**Verdict:** (one sentence) + + + +### 7. Unique capabilities (if any) + +List capabilities that have no practical built-in equivalent. If none exist, +explicitly state that. Annotate each with frequency and impact. + +**Verdict:** (one sentence) + + + +### 8. Tasks outside Serena's scope (built-in only) + +Identify tasks where built-ins are the natural choice because Serena's tools +don't target them. List these briefly for completeness but do not frame them as +Serena shortcomings — they are outside its scope. Estimate their share of daily +work to contextualize how much of a session Serena's augmentation covers. + +**Verdict:** (one sentence) + + + +### 9. Practical usage rule + +Provide a decision rule for choosing between toolsets based on task type. + +**Verdict:** (one sentence) + + + +## What I'm looking for + +- Claims grounded in observed evidence +- Explicit reporting of **positive, neutral, and negative deltas** +- Clear quantification of impact +- Honest workflow-level comparisons + + + +## What I am not looking for + +- Misuse-based failure analysis +- Gotcha comparisons +- Neutral statements without evidence +- Binary recommendations +- Novelty-driven ordering +- Unquantified claims +``` \ No newline at end of file diff --git a/docs/04-evaluation/020_prompts/020_summary-prompt.md b/docs/04-evaluation/020_prompts/020_summary-prompt.md new file mode 100644 index 00000000..c09fb8a6 --- /dev/null +++ b/docs/04-evaluation/020_prompts/020_summary-prompt.md @@ -0,0 +1,12 @@ +# Summary Prompt + +We used the prompt below to summarize the evaluation of Serena. + +``` +Write a one-sentence user-facing summary about the value that Serena's tools +provide for coding. With a bit of emotion but grounded in the evaluation that +you have done above. It should be written from the perspective of a coding AI +agent and make it clear whether an AI agent would ask its owner to add Serena's +tools or not. It will be the first sentence that a potential new user will see +about how useful Serena is for an AI agent. +``` \ No newline at end of file diff --git a/docs/04-evaluation/030_results/000_evaluation-results.md b/docs/04-evaluation/030_results/000_evaluation-results.md new file mode 100644 index 00000000..e0b6bfc6 --- /dev/null +++ b/docs/04-evaluation/030_results/000_evaluation-results.md @@ -0,0 +1,17 @@ +# Results + +This section presents the results of the evaluation. + +We performed evaluations using popular AI coding agents in representative scenarios — different +agents, different programming languages, and different codebases — to show that the results are +not specific to a single setup. + +All evaluations were conducted using the **JetBrains-powered version** of Serena, as it is the +more powerful backend with a broader set of refactoring and navigation capabilities. The +evaluation can easily be repeated with the LSP-based backend to assess its subset of capabilities. + +- [Claude Code (Opus 4.6) on a large Python codebase](010_cc_on_tianshou) +- [Codex (GPT 5.4) on a Java codebase](020_codex_on_jbplugin) + +You can run your own evaluation on a project of your choice by reusing our +[evaluation prompt](../020_prompts/010_evaluation-prompt.md). diff --git a/docs/04-evaluation/030_results/010_cc_on_tianshou.md b/docs/04-evaluation/030_results/010_cc_on_tianshou.md new file mode 100644 index 00000000..8b0b53fe --- /dev/null +++ b/docs/04-evaluation/030_results/010_cc_on_tianshou.md @@ -0,0 +1,358 @@ +:::{admonition} Evaluation Result +:class: note +**Generated by**: Claude Opus 4.6 (coding AI agent in Claude Code CLI) +**Codebase:** [Tianshou](https://github.com/thu-ml/tianshou) — a Python reinforcement learning library (~26K lines, 43 source files) +::: + +# Claude Code (Opus) + +> **One-line summary:** Serena's IDE-backed semantic tools are the single most impactful addition to my toolkit — cross-file renames, moves, and reference lookups that would cost me 8–12 careful, error-prone steps collapse into one atomic call, and I would absolutely ask any developer I work with to set them up. + +**Method:** Hands-on, side-by-side execution of 20 tasks using both toolsets. All edits were applied to real files and verified via `git diff`, then reverted. + +--- + +## 1. Headline: What Serena Changes + +Serena adds an IDE-backed semantic layer (powered by JetBrains) on top of the built-in text-level tools. The practical delta breaks down into three categories: + +**(a) Tasks where Serena adds capability:** +- **Cross-file refactoring** (rename, move symbol, move file): 1 atomic call vs. N×(Grep+Read+Edit) chains. This is Serena's strongest contribution — it collapses multi-file, import-aware operations into single calls that are atomic and semantically correct. +- **Structural code navigation**: Symbol overviews, type hierarchies, and reference finding return structured, scope-disambiguated results that text search cannot produce without manual interpretation. +- **External dependency introspection**: Serena can look up symbols in installed packages via the IDE index without requiring manual environment discovery. + +**(b) Tasks where Serena applies but offers no improvement:** +- **Single-file rename of a unique string**: `Edit` with `replace_all=true` achieves the same result in 1 call (after a Read), with comparable effort. +- **Small edits (1–3 lines inside a method)**: Edit's substring matching is more efficient — less payload sent, no requirement to supply the full symbol body. +- **Inserting new code at a known location**: Both approaches require ~1–2 calls. Edit can target by surrounding text; Serena targets by name path. The effort is comparable. + +**(c) Tasks outside Serena's scope (built-in only):** +- Non-code files (config, docs, notebooks, changelogs) +- Free-text search across the repo +- Shell commands, git operations, test execution +- File creation from scratch + +**Verdict:** Serena's primary contribution is collapsing multi-file, semantically-aware operations (rename, move, reference-finding, type hierarchy) from multi-step manual processes into single atomic calls; for single-file text edits, it is comparable or slightly less efficient. + +--- + +## 2. Added Value and Differences by Area + +**1. Cross-file rename/move (Serena: strong positive)** +- Serena: 1 call updates definition + all imports + all usages across N files atomically. +- Built-in: 1 Grep + N×(Read+Edit) = 2N+1 calls, non-atomic. +- Frequency: Moderate (a few times per working session during refactoring work). Value per hit: High — saves 6–10+ calls and eliminates the risk of partial updates. + +**2. Structural overview / targeted symbol reading (Serena: moderate positive)** +- `get_symbols_overview(depth=1)`: 1 call returns all classes, methods, and attributes in structured JSON, via the IDE's parser — language-agnostic and always correct. Built-in equivalent: `Grep` for a language-specific heuristic pattern like `^(class |def )`. This works for simple Python files but is inherently fragile: the pattern must be hand-tuned per language, misses decorated or multi-line declarations, and can match inside strings or comments. For non-Python languages, an entirely different regex would be needed. +- `find_symbol(include_body=True)`: Retrieves a specific method body by name path without reading surrounding code. Built-in: requires knowing the line number (via Grep) then Read with offset — 2 calls. +- Frequency: High (many times per session). Value per hit: Low-to-moderate — saves ~1 call and some context window tokens per navigation. The reliability advantage over heuristic Grep patterns is consistent across all languages. + +**3. Reference finding with structural context (Serena: moderate positive)** +- `find_referencing_symbols`: Returns which *symbols* reference a target, with file grouping and usage type (import, call, type annotation). Built-in `Grep` returns all text matches including docs, comments, and string literals — same recall but lower precision. +- Frequency: Moderate. Value per hit: Moderate — precision matters when planning refactors of widely-used symbols. + +**4. Type hierarchy (Serena: positive, niche)** +- 1 call returns full super/sub-type chains transitively. Built-in: requires iterative Grep for `class X(Y)` patterns, manual transitive closure. +- Frequency: Low (occasional during architecture exploration). Value per hit: High per occurrence — several calls saved. + +**5. Stable addressing across edits (Serena: moderate positive)** +- Serena addresses symbols by name path (`CollectStats/refresh_return_stats`), which is invariant under edits elsewhere in the file. The built-in workflow is fundamentally line-number-mediated: Grep returns line numbers, Read takes line offsets — and both go stale after any edit to the file. Edit's text matching (`old_string`) is more resilient than line numbers, but it sits at the end of a chain that starts with position-based lookups. After an edit shifts lines, previously noted Grep results and Read offsets are invalid and must be re-acquired. +- This means a multi-edit session with built-ins requires re-Grep or re-Read between edits to re-establish positions, while Serena's name paths remain valid throughout. In Task 17 (three chained edits), Serena needed 3 calls total; the built-in path needed 4 (1 Read + 3 Edits) only because the text anchors happened to be unique — but had any Edit failed uniqueness, a re-Read would have been required, pushing the count to 5–7. +- Frequency: High (any session with multiple edits to the same file). Value per hit: Moderate — saves 1–2 re-read calls per edit chain, and eliminates the class of errors where stale line numbers cause edits to land in the wrong place. + +**6. Single-file small/medium edits (Serena: neutral to slight negative)** +- For a 1-line change inside a 13-line method, Serena's `replace_symbol_body` requires sending the entire 13-line body. Edit sends just the changed line. The prerequisite cost differs: Serena needs `find_symbol(include_body=True)` to get the body; Edit needs `Read` of the relevant lines. Both are 1 prerequisite call + 1 edit call. +- Frequency: Very high. Value per hit: Slightly negative for Serena — more payload for small changes. + +**Verdict:** Serena's strongest contributions are in cross-file refactoring (high value, moderate frequency) and structured navigation (moderate value, high frequency); for small text edits, built-ins are slightly more efficient. + +--- + +## 3. Detailed Evidence, Grouped by Capability + +### 3.1 Structural Overview (Task 2) + +| Metric | Serena `get_symbols_overview(depth=1)` | Built-in `Grep` for class/def | +|---|---|---| +| Calls | 1 | 1 | +| Output structure | JSON tree: 14 classes with nested methods + attributes | Flat list: 54 `class`/`def` lines with line numbers | +| Nesting visible? | Yes — methods grouped under classes | No — indentation implies nesting but no grouping | +| Attributes visible? | Yes | No | +| Output size | ~2.5KB structured JSON | ~3KB flat text | +| Correctness | Always correct — uses IDE parser, language-agnostic | Heuristic — the regex `^(class \|def \| def )` is Python-specific and can miss decorated methods, multi-line signatures, or match inside strings/comments | +| Language portability | Works unchanged for any language the IDE supports | Requires a new hand-tuned regex per language | +| Next step | `find_symbol("ClassName/method", include_body=True)` — 1 call | `Read(file, offset=line, limit=N)` — 1 call | + +Both paths need 1 call for overview + 1 call for drill-down. Serena's output is more structured and reliably correct; Grep's is simpler but fragile. The Grep pattern used here happened to work well for this Python codebase, but would need to be rewritten for Java (`class|interface|enum`, brace-delimited), Rust (`fn|struct|impl|trait`), TypeScript (`class|function|interface`), etc. — and even within Python, it misses cases like `@overload`-decorated methods or methods whose `def` line is preceded by a long decorator stack that pushes `def` to a non-standard indentation. + +**Verdict:** Serena provides a reliably correct structural overview across languages; Grep-based overviews are heuristic approximations that work in simple cases but degrade for complex declarations or non-Python codebases. + +### 3.2 Targeted Method Retrieval (Task 3) + +| Metric | Serena | Built-in | +|---|---|---| +| Calls | 1 (`find_symbol` with `include_body=True`) | 2 (Grep to find line + Read with offset) | +| Prerequisite | Know the name path | Know the method name | +| Output | Method body only (no surrounding code) | Surrounding code included (must estimate limit) | + +For `Collector/_collect` (330 lines): Serena returned exactly 330 lines. Built-in Read returned 332 lines (including the next method's signature). + +**Verdict:** Serena saves 1 call and returns precisely the requested body; built-in is adequate but requires line-number discovery. + +### 3.3 Cross-File References (Task 4) + +| Metric | Serena `find_referencing_symbols` | Built-in `Grep` | +|---|---|---| +| Calls | 1 | 1 | +| Files found | 63 (code files only, structured by symbol type) | 83 (includes docs, notebooks, changelogs, README) | +| Output structure | Grouped by file, categorized (import, call, type annotation) | Flat file list | +| Precision | Code-only references | All textual mentions | + +Grep finds 83 files including `README.md`, `CHANGELOG.md`, and `.ipynb` notebooks. Serena finds 63 code files. For the question "who uses this in code?", Serena's result is directly usable. For "where is this mentioned anywhere?", Grep is the right tool. + +**Verdict:** Serena provides higher-precision code-usage results; Grep provides broader text-level coverage. Different tools for different questions. + +### 3.4 Type Hierarchy (Task 5) + +| Metric | Serena `type_hierarchy` | Built-in | +|---|---|---| +| Calls | 1 | 2+ (Grep for `class X(BaseCollector` + Grep for `class Collector(` + potential transitive search) | +| Result | `BaseCollector → ABC → object` (supers), `BaseCollector → Collector → AsyncCollector` (subs), with file locations | `class Collector(BaseCollector[TCollectStats], ...)` at line 551 — requires further reads for AsyncCollector's superclass | + +**Verdict:** Serena produces the full transitive hierarchy in 1 call; built-in requires iterative searches. + +### 3.5 External Dependency Lookup (Task 6) + +| Metric | Serena | Built-in | +|---|---|---| +| Can retrieve? | Yes — `find_declaration` + `search_deps=True` | Requires environment discovery: `python -c "import X; print(inspect.getfile(X))"` then `Read` | +| Infrastructure | IDE index (pre-built) | Working Python environment, correct venv activated | +| Result | Symbol location + documentation | Full source file (if env is set up) | + +In this session, the Python environment wasn't directly accessible from bash, so built-in lookup would have required additional setup. Serena retrieved `torch.distributions.Distribution`'s location and docstring via the IDE index. + +**Verdict:** Serena provides dependency introspection without environment setup; built-in requires a working interpreter. + +### 3.6 Small Edit — 1 Line Change (Task 7a) + +| Metric | Serena `replace_symbol_body` | Built-in `Edit` | +|---|---|---| +| Prerequisite calls | 1 (`find_symbol` with `include_body=True`) — already done in Task 3 | 1 (`Read` of ~15 lines) | +| Edit call | 1 (send full 13-line body) | 1 (send 1-line old + 1-line new) | +| Payload sent (edit call) | ~550 chars (full body) | ~120 chars (changed line only) | +| Total calls | 1–2 | 2 | + +**Verdict:** For small changes, Edit sends ~4.5× less payload in the edit call. Total call count is the same. + +### 3.7 Medium Rewrite — ~19 Lines (Task 7b) + +| Metric | Serena | Built-in | +|---|---|---| +| Prerequisite | 1 `find_symbol` (already done) | 1 `Read` (~22 lines) | +| Edit payload | ~550 chars (full body) | ~1000 chars (old 19 lines + new 20 lines) | +| Total calls | 1–2 | 2 | + +At this scale, Serena's payload is actually *smaller* because Edit must send both old and new text, while Serena sends only the new body. The crossover point is approximately where the changed region exceeds half the method body. + +**Verdict:** At medium scale, payloads converge; Serena's is slightly smaller due to only sending the replacement. + +### 3.8 Large Rewrite — 55 Lines (Task 7c) + +| Metric | Serena | Built-in | +|---|---|---| +| Prerequisite | 1 `find_symbol` (already done) | 1 `Read` (~67 lines) | +| Edit payload | ~2200 chars (new body) | ~4400 chars (old 63 lines + new 62 lines) | +| Total calls | 1–2 | 2 | + +**Verdict:** For full-method rewrites, Serena sends ~50% less payload because it only sends the replacement, not the original. + +### 3.9 Cross-File Rename (Task 10) + +| Metric | Serena `rename` | Built-in chain | +|---|---|---| +| Calls | 1 | 1 Grep + 4 Read + 4 Edit = 9 | +| Files affected | 4 (automatically discovered) | 4 (manually discovered via Grep) | +| Import handling | Automatic | Manual — must identify and rewrite each import statement | +| Atomicity | Atomic — all-or-nothing | Sequential — intermediate states are inconsistent | + +**Verdict:** Serena reduces a 9-call chain to 1 call for cross-file rename, with atomicity. + +### 3.10 Move Symbol Across Modules (Task 11) + +| Metric | Serena `move` | Built-in equivalent | +|---|---|---| +| Calls | 1 | ~8–12 (Read source, Edit source to remove, Edit target to add, Grep for imports, Read+Edit each import site) | +| Import updates | Automatic | Manual — must rewrite `from X import Y` → `from Z import Y` in each file | +| Dependency imports | Automatically adds needed imports to target file | Must manually inspect what the moved function imports and replicate | + +Moving `get_stddev_from_dist` from `collector.py` to `stats.py`: Serena updated 3 files (source, target, test) in 1 call, including adding necessary imports to the target module. + +**Verdict:** Move is Serena's highest-value single operation — it handles import graph updates that would be error-prone manually. + +### 3.11 Move File (Task 12a) + +| Metric | Serena `move` | Built-in equivalent | +|---|---|---| +| Calls | 1 | 1 `git mv` + 1 Grep + N×(Read+Edit) for import updates = ~12 | +| Files updated | 5 import updates automatic | Must manually discover and rewrite | + +**Verdict:** Same pattern as symbol move — Serena collapses an N-step process. + +### 3.12 Safe Delete (Task 12b) + +Serena's `safe_delete` in safe mode (default): +- Reports all usages before deleting — acts as a guard. +- For unused symbols, deletes cleanly in 1 call. +- For used symbols, refuses and lists usages. + +Built-in equivalent: Grep to check usages → if none found, Read + Edit to delete. 2–3 calls. + +**Verdict:** Safe delete adds a safety check that the built-in path must implement manually. + +### 3.13 Inline (Task 13) + +Serena's inline tool did not work for any Python function tested. The JetBrains backend does not support Python function inlining (this is a language-level limitation of the IDE's refactoring engine, not a Serena bug). + +**Verdict:** No candidate inlinable in this codebase; inline capability appears unavailable for Python. + +### 3.14 Scope Precision (Task 14) + +Three methods named `reset_env` exist in `collector.py` (in `BaseCollector`, `Collector`, `AsyncCollector`). + +- Serena: `find_symbol("Collector/reset_env")` returns exactly that override's body. +- Grep: Returns 3 line numbers. Determining which belongs to which class requires reading surrounding context. + +**Verdict:** Serena's name-path addressing eliminates class-level disambiguation that Grep requires. + +### 3.15 Chained Edits (Task 17) + +Three consecutive edits to methods in `CollectStats`: +- Serena: 3 `replace_symbol_body` calls, 0 intermediate reads. Name paths (`CollectStats/refresh_return_stats`, `CollectStats/refresh_len_stats`, `CollectStats/refresh_std_array_stats`) remained valid across all edits because they are structural identifiers, not positions. +- Built-in: 1 Read + 3 Edit calls = 4 calls in the best case. The Read was required upfront (Edit enforces "must read before editing"). The 3 Edits succeeded without re-reads only because the `old_string` anchors happened to remain unique after each edit. But this is fragile: Edit also enforces a "file has been modified since read" check when external tools modify the file, forcing a re-Read. More fundamentally, if I had needed to *find* these methods first (the typical case when you don't already know the line numbers), each Grep result from before the first edit would have been stale after it — line 232 is no longer line 232 after inserting 2 lines above it. + +The core asymmetry: Serena's addressing is *structural* (survives edits by definition), while built-in addressing is *positional* (line numbers from Grep/Read go stale after any insertion or deletion). Edit's text matching partially mitigates this, but only for the final step — the discovery steps (Grep, Read with offset) remain position-dependent. + +**Verdict:** Serena's name-path stability eliminates the re-read/re-grep cycle between edits; built-in tools require re-acquiring positions after each edit that shifts line numbers. + +--- + +## 4. Token-Efficiency Analysis + +### Payload by edit size + +| Edit scale | Serena payload (edit call) | Edit payload (edit call) | Winner | +|---|---|---|---| +| 1-line change in 13-line method | ~550 chars (full body) | ~120 chars | Edit (~4.5×) | +| 19-line medium rewrite | ~550 chars | ~1000 chars | Serena (~1.8×) | +| 55-line full rewrite | ~2200 chars | ~4400 chars | Serena (~2×) | +| Cross-file rename (4 files) | ~100 chars | ~800 chars (4 Edit calls) | Serena (~8×) | + +### Prerequisite reads + +- Serena: `find_symbol(include_body=True)` returns the symbol body. If you've already navigated to it during exploration, no additional call needed. +- Built-in: `Read` with offset/limit. Always required before Edit (enforced by the tool). + +### Stable vs ephemeral addressing + +- Serena's name paths (`CollectStats/refresh_return_stats`) are stable identifiers — they survive any edit to surrounding code. No re-read or re-discovery needed between edits. +- The built-in workflow uses ephemeral, position-based addresses at every stage: Grep returns line numbers, Read takes line offsets, and both are invalidated by any insertion or deletion above the target. Edit's `old_string` matching is content-based and more resilient, but it depends on the upstream position-based steps to know *what* to match. After an edit shifts line numbers, the entire Grep→Read→Edit chain must be re-executed from the top. +- In a session with N edits to the same file, this costs up to N-1 additional Grep/Read round-trips with built-ins. With Serena, the cost is zero — the same name path works on the first and tenth edit. + +**Verdict:** Serena is more token-efficient for medium-to-large edits and cross-file operations; Edit is more efficient for small, localized changes. Across multi-edit sessions, Serena's stable addressing avoids the re-read tax that compounds with each successive built-in edit. + +--- + +## 5. Reliability & Correctness (Under Correct Use) + +### Precision of matching +- Serena: Exact symbol resolution via name paths. `Collector/reset_env` unambiguously selects one method. +- Edit: Text matching. Unique strings match correctly; non-unique strings fail (and Edit reports the error). + +### Scope disambiguation +- Serena distinguishes overrides, overloads (via indices), and nested classes by name path. +- Grep/Edit cannot distinguish methods with the same name in different classes without reading surrounding context. + +### Atomicity +- Serena's cross-file operations (rename, move) are atomic — all files updated or none. +- Built-in multi-file edits are sequential — a failure mid-chain leaves an inconsistent state (recoverable via `git checkout`). + +### Semantic queries vs text search +- `find_referencing_symbols` returns code-level references categorized by type. Grep returns all textual mentions. +- `type_hierarchy` returns transitive sub/supertypes. No built-in equivalent without iterative search. + +### External dependency lookup +- Serena: Available via IDE index, no environment setup needed. Can retrieve symbol docs and location. +- Built-in: Requires working Python environment, correct venv, and manual file discovery. More powerful when available (full source), but higher setup cost. + +**Verdict:** Serena provides stronger correctness guarantees for symbol-level operations (scope, atomicity, semantic precision); built-in tools are reliable for text-level operations with the caveat that scope disambiguation requires manual effort. + +--- + +## 6. Workflow Effects Across a Session + +### Compound advantages +- **Exploration → Edit without position re-acquisition**: Serena's overview tools produce name paths that are directly usable as edit targets. The built-in path produces line numbers (from Grep) that are consumed by Read — but after an edit, those line numbers are stale. In a typical explore-edit-explore-edit cycle, Serena's name paths remain valid throughout while built-in line numbers must be re-acquired after each edit. Over 3 explore+edit cycles, this saves ~3–6 intermediate Grep/Read calls. +- **Chained edits without re-reads**: Name-path stability means no re-reads between edits to the same file. For N edits, this saves up to N-1 Read calls. Edit's text matching partially avoids this (if old_strings stay unique), but the upstream discovery steps (Grep line numbers, Read offsets) still go stale. +- **Cross-file refactoring**: When a rename or move is part of a larger change, doing it atomically avoids the need to manually track "which files still need updating." + +### Diminishing returns +- For purely exploratory sessions (reading code, no edits), Serena's advantage is modest — `Grep` and `Read` are nearly as fast for navigation, and Serena's structured output doesn't save many calls. +- For sessions dominated by small text edits (config changes, log message tweaks), Serena adds no value. + +### Neutral findings +- Both toolsets require similar total calls for single-file work. The difference is ~1 call per operation. +- Output quality for code review (reading diffs, understanding changes) is identical — both require `git diff`. + +**Verdict:** Serena's advantages compound across multi-step refactoring sessions where each operation feeds into the next; for read-heavy or small-edit sessions, the advantage is marginal. + +--- + +## 7. Unique Capabilities (No Practical Built-In Equivalent) + +1. **Atomic cross-file rename** — 1 call, all imports and usages updated, all-or-nothing. No built-in equivalent without scripting a multi-step chain. Frequency: moderate (refactoring sessions). Impact: high (saves 5–10 calls, eliminates partial-update risk). + +2. **Atomic cross-file move** (symbol or file) with import rewriting — includes adding necessary imports to the target module. Frequency: low-to-moderate. Impact: very high per occurrence (saves 8–12 calls, handles import dependency graph). + +3. **Type hierarchy traversal** — transitive supertypes and subtypes in 1 call. Built-in requires iterative Grep with manual transitive closure. Frequency: low. Impact: moderate (saves 3–5 calls). + +4. **Safe delete with usage check** — reports all usages before deleting, with option to propagate. Built-in requires Grep + manual verification. Frequency: low. Impact: moderate (safety check is the value). + +5. **External dependency symbol lookup** via IDE index — no Python environment needed. Frequency: moderate. Impact: moderate (avoids environment setup friction). + +**Verdict:** Serena provides 3–5 capabilities with no practical built-in equivalent, concentrated in cross-file refactoring and semantic navigation. + +--- + +## 8. Tasks Outside Serena's Scope (Built-In Only) + +- **Non-code file operations**: Reading/editing config files, docs, changelogs, notebooks → `Read`/`Edit`/`Write` +- **Free-text search**: Finding log strings, URLs, magic constants → `Grep` +- **Shell operations**: Running tests, builds, git commands, package management → `Bash` +- **File creation**: New files from scratch → `Write` +- **Glob-based file discovery**: Finding files by pattern → `Glob` +- **Broad codebase search**: When you don't know what you're looking for → `Grep` with regex + +Estimated share of daily work: These tasks constitute roughly 40–60% of a typical coding session (reading docs, running tests, searching for patterns, editing config). Serena's augmentation covers the remaining 40–60% where code-level semantic operations apply. + +**Verdict:** Built-in tools handle roughly half of daily work that falls entirely outside Serena's scope; Serena augments the code-centric other half. + +--- + +## 9. Practical Usage Rule + +| Task type | Use | +|---|---| +| Cross-file rename, move, delete | Serena (unique capability) | +| Understanding class hierarchy or symbol relationships | Serena (1 call vs. iterative search) | +| Getting a structural overview of a large file | Serena (richer structure) or Grep (simpler, faster) | +| Reading a specific method body by name | Serena (direct) or Grep+Read (2 calls) | +| Small edit (1–5 lines inside a method) | Edit (less payload) | +| Full method rewrite | Serena `replace_symbol_body` (less payload, no re-read) | +| Single-file rename of a unique identifier | Edit with `replace_all` (equivalent) | +| Non-code files, config, docs | Built-in `Read`/`Edit` | +| Text search, pattern matching | Built-in `Grep` | +| Anything involving shell, git, tests | Built-in `Bash` | +| External dependency inspection | Serena (no env setup needed) | + +**Verdict:** Use Serena for cross-file refactoring and semantic navigation; use built-ins for text-level edits, non-code files, and shell operations; for single-file code edits, choose based on edit size (Edit for small, Serena for full-body rewrites). diff --git a/docs/04-evaluation/030_results/020_codex_on_jbplugin.md b/docs/04-evaluation/030_results/020_codex_on_jbplugin.md new file mode 100644 index 00000000..27fa09f3 --- /dev/null +++ b/docs/04-evaluation/030_results/020_codex_on_jbplugin.md @@ -0,0 +1,665 @@ +:::{admonition} Evaluation Result +:class: note +**Generated by:** GPT-5.4 (high) in Codex +**Codebase:** Serena JetBrains Plugin (Java) +::: + +# Codex (GPT-5.4) + + +> **One-line summary:** As a coding agent, I would ask my owner to add Serena because it turns fragile text-and-line-number work into precise symbol-aware navigation and +refactoring, which makes real code changes feel faster, safer, and far less blind. + +This report compares Serena's JetBrains-backed semantic coding tools with built-in file, shell, search, and patch tools in this repository. The comparison assumes competent use of both toolsets: built-ins are used for text, file, shell, config, and small patch work; Serena is used where symbol identity, language semantics, or IDE refactoring semantics apply. + +Method: I explored code first, avoided repo documentation and prior notes, ran real edits/refactors, and reverted after each experiment. After each edit/refactor experiment I checked `git status --short` and returned the tree to clean before moving on. Measurements are approximate, but call counts, diff sizes, and result shapes are from observed runs. + +## 1. Headline: what Serena changes + +Serena adds a semantic layer over the codebase. Its concrete delta is the ability to address and transform code by symbols: name paths, overload indexes, reference graphs, type hierarchies, external declarations, and JetBrains refactoring operations. + +Tasks where Serena adds capability: + +- Code-reference search: `find_referencing_symbols` for `Symbol/findReferences` returned 6 precise code usages grouped by containing symbols. `rg findReferences` also returned the declaration, the `/findReferences` route string, and comments. +- Type hierarchy: one `type_hierarchy` call for `PostRequestHandler` returned 18 direct endpoint subclasses, a transitive `TypeHierarchyHandler -> GetSupertypesHandler/GetSubtypesHandler` branch, and the external `Object` supertype. +- External dependency lookup: `find_declaration` resolved `ReferencesSearch.search(anchorElement)` to ``, and `find_symbol` returned the selected overload body. Built-in repo/cache grep found the import and call but not the dependency definition. +- Cross-file refactors: semantic rename, symbol move, file move, safe delete, and inline executed real IDE refactors and updated files/usages/imports. +- Stable addressing: a chained edit used `SymbolFinder/findFilesByName`, `SymbolFinder/getProject`, and `SymbolFinder/qualNameMatchesName` after earlier edits shifted line numbers. + +Tasks where Serena applies but offered little or no improvement: + +- Tiny intra-method edits: a 1-line error-message change was smaller with a built-in patch; Serena body replacement required sending the whole method. +- Simple one-file private rename: manual patch and semantic rename produced the same 2-line diff for `qualNameMatchesName -> qualifiedNameMatchesName`. +- Method insertion at a known spot: both workflows produced the same 4-line diff; Serena's benefit was target stability, not smaller payload. +- Whole-method rewrite: Serena targets the method boundary cleanly, but the full replacement body still has to be sent. + +Tasks outside Serena's scope: + +- Reading config/non-code files such as `plugin.xml` and `build.gradle.kts`. +- Free-text searches for URLs, endpoint strings, settings values, comments, and resources. +- File inventory, line counts, shell commands, build/test execution, Git operations, and arbitrary text edits. + +Value-weighted summary: + +| Rank | Difference | Frequency | Value per hit | Delta | +|---|---:|---:|---:|---| +| 1 | Cross-file semantic refactors | Medium-high | High | Often avoids 5-20 manual edit/search/verify steps and reduces partial-update risk. | +| 2 | Code usages vs text mentions | High | Medium-high | Removes noisy grep triage for "who uses this symbol?" | +| 3 | Symbol overview/body retrieval | High | Medium | Avoids large reads and stale line targeting. | +| 4 | Type hierarchy/implementations | Medium | High | Hard to reproduce correctly with text search, especially transitively. | +| 5 | Stable addressing across edit chains | Medium | Medium | Reduces refresh work after line shifts. | +| 6 | External dependency lookup | Low-medium | High | Needs IDE index; built-ins need source/decompiler infrastructure. | +| 7 | Small local edits | High | Neutral/negative for Serena | Built-in patch/edit usually sends less. | + +**Verdict:** Serena materially changes symbol-centric exploration and refactoring; it does not replace built-ins for ordinary text, file, shell, config, or tiny local edits. + +## 2. Added value and differences by area + +- Cross-file semantic refactoring changes both workflow and correctness. Frequency: medium-high. Value per hit: high. The observed class rename updated a file/class plus imports/usages; the nested-record move changed 4 existing files and created 1 new file; the file move updated package/imports across 3 paths. Built-ins can reproduce the result, but only through search, file moves, patches, import cleanup, and verification. + +- Semantic search separates code usage from text mention. Frequency: high. Value per hit: medium-high. Serena returned only code references for `Symbol/findReferences`; `rg` returned code, route strings, comments, and the declaration. Built-ins remain better for "mentioned anywhere." + +- Symbol overview/body retrieval cuts exploration payload. Frequency: high. Value per hit: medium. `get_symbols_overview` on `Symbol.java` returned fields, methods, nested classes, and overload indexes without reading the 1,000+ line file. Regex produced line-oriented candidates and still required manual scope interpretation. + +- Type/dependency queries add capabilities not practically present in built-ins. Frequency: low-medium. Value per hit: high. Serena returned a transitive type graph and an external IntelliJ API method body; text tools needed iterative searches or extra source/decompiler setup. + +- Small localized edits are not improved by symbol-body replacement. Frequency: high. Value per hit: neutral or negative for Serena. A one-line patch was smaller than replacing an 11-line method body. + +- Some JetBrains refactors staged added/deleted files. Frequency: limited to class/file move/delete style operations. Value per hit: neutral operational tradeoff. Semantic value remains, but verification/revert must check the index as well as unstaged files. + +**Verdict:** Serena's highest-value differences are symbol identity, reference graphs, and IDE refactor execution; built-ins remain superior for simple text-local work. + +## 3. Detailed evidence, grouped by capability + +### 3.1 Repository structure and entry points + +Attempted: identify top-level layout, packages, and entry points. + +Built-in call chain: + +1. `git status --short` -> clean. +2. `rg --files -g '!*.md' -g '!docs/**' -g '!CLAUDE.md'` -> code/config/resource inventory. +3. `Get-ChildItem -Force` -> top-level directories. +4. `Get-Content src/main/resources/META-INF/plugin.xml` -> plugin entry points. +5. `Get-Content build.gradle.kts -TotalCount 80` -> Gradle/IntelliJ platform setup. + +Serena call chain: not applicable for repository inventory and non-code config. + +Observed structure: + +- `service`: backend service and request handling. +- `service/endpoint`: endpoint handlers for symbol search, references, formatting, rename, move, safe delete, inline, inspections, and completions. +- `symbol`: symbol model, symbol lookup, hierarchy, path matching, and move processors. +- `util`: IDE/project/editor/UI helpers. +- `ui`: tool window content. +- `plugin.xml`: registers `PluginStartupActivity`, settings service/configurable, and a dummy test action. + +Payloads: built-ins used 5 small shell/read calls and returned several KB of code/config context; Serena had no relevant semantic operation. + +**Verdict:** Repository layout and non-code entry-point discovery are built-in work; Serena starts adding value once the target is code symbols. + +### 3.2 Large file structural overview + +Attempted: compare structural overview on `src/main/java/de/oraios/serena/symbol/Symbol.java`. + +Serena call chain: + +1. `get_symbols_overview(relative_path=Symbol.java, depth=1)` +2. Next step: `find_symbol(Symbol/findReferences, include_body=true)` + +Serena output: one top-level `Symbol` class; fields; nested `ChildrenCollector` and `DocumentationResolver`; methods including overload-indexed names such as `getLocationString[0]`, `getLocationString[1]`, `move[0]`, `move[1]`, `getDocumentation[0]`, and `getDocumentation[1]`. + +Built-in call chain: + +1. `rg` for method-like declaration lines in `Symbol.java`. +2. `rg` for class/interface/enum lines in `Symbol.java`. +3. Next step: line-slice read around the selected method. + +Built-in output: about 70 line-oriented method/class matches, including nested symbols, with no durable symbol identity beyond manual signature inspection. + +Payloads: Serena used 1 compact overview call plus a targeted follow-up. Built-ins used 2 regex searches plus a later line-range read. + +**Verdict:** Serena's overview is more actionable because its output feeds directly into stable symbol-addressed calls; regex outlines are useful but line-based. + +### 3.3 Targeted method body retrieval + +Attempted: retrieve `Symbol/findReferences` without reading surrounding file content. + +Serena call chain: `find_symbol(relative_path=Symbol.java, name_path_pattern=Symbol/findReferences, include_body=true)`. + +Serena output: only the `public ArrayList findReferences()` body. + +Built-in call chain: + +1. `rg "findReferences\\(" src/main/java -n -C 2` +2. `Get-Content Symbol.java` line slice around the declaration. + +Built-in output: search context across multiple files, then the selected method slice. + +Payloads: Serena input was one path/name-path request and returned roughly 25 method lines. Built-ins required search output plus a separate slice read. + +**Verdict:** If the symbol is known, Serena retrieves the body in one precise call; built-ins need search plus line-targeted reading. + +### 3.4 References: code usages vs mentions anywhere + +Attempted: find all references to `Symbol/findReferences`. + +Serena call chain: `find_referencing_symbols(relative_path=Symbol.java, name_path=Symbol/findReferences)`. + +Serena output: + +- `Symbol/formatReferenceLocations/references` +- `Symbol/inline/refCountBefore` +- `Symbol/verifyInlineResult/refCountAfter` +- `SymbolDTO/Builder/buildDTO/references` +- `RunInspectionsOnSymbolsHandler/handleRequest/refs` +- `FindReferencesHandler/buildResponse/references` + +Built-in call chain: `rg "findReferences" src/main/java -n -C 1`. + +Built-in output: same call sites plus the declaration, `/findReferences` route registration, and explanatory comments. + +Payloads: Serena returned a compact grouped usage list of about 500-700 characters. `rg` returned broader line context of about 1-2 KB and mixed code usages with text mentions. + +**Verdict:** Serena has higher precision for "who uses this symbol in code"; built-ins have higher recall for "where is this text mentioned anywhere." + +### 3.5 Type hierarchy + +Attempted: list supertypes and subtypes transitively for `PostRequestHandler`. + +Serena call chain: `type_hierarchy(relative_path=PostRequestHandler.java, name_path=PostRequestHandler, hierarchy_type=both, depth=0)`. + +Serena output: external `Object` supertype; 18 direct endpoint subclasses; transitive nested branch `TypeHierarchyHandler` containing `GetSupertypesHandler` and `GetSubtypesHandler`. + +Built-in call chain: + +1. `rg "extends PostRequestHandler|extends TypeHierarchyHandler|class PostRequestHandler" src/main/java -n` +2. Manually follow any discovered intermediate types. +3. Repeat searches if deeper hierarchy exists. + +Payloads: Serena used 1 hierarchy query and returned structured JSON. Built-ins used pattern search and manual transitive grouping. + +**Verdict:** Serena turns hierarchy discovery into a semantic graph query; text search requires iterative pattern expansion and manual reasoning. + +### 3.6 External dependency symbol lookup + +Attempted: retrieve the IntelliJ dependency symbol behind `ReferencesSearch.search(anchorElement)`. + +Serena call chain: + +1. `find_declaration(relative_path=Symbol.java, regex="ReferencesSearch\\.(search)\\(anchorElement\\)")` +2. `find_symbol(relative_path=, name_path_pattern=ReferencesSearch/search[0], include_body=true, search_deps=true)` + +Serena output: declaration `ReferencesSearch/search[0]` in an external class path and the overload body: + +```java +public static @NotNull Query search(@NotNull PsiElement element) { + return search(element, GlobalSearchScope.allScope(PsiUtilCore.getProjectInReadAction(element)), false); +} +``` + +Built-in call chain: + +1. `rg "ReferencesSearch" src/main/java -n` -> import and local call. +2. `rg` over repo, `.intellijPlatform`, and Gradle caches for `class ReferencesSearch` or matching method signatures -> no useful source hit. +3. `Get-ChildItem` over Gradle/IntelliJ caches -> jar candidates but no direct definition. + +Infrastructure difference: Serena depends on the JetBrains IDE index; built-ins need source jars, decompiler tooling, or classpath-specific jar inspection. + +**Verdict:** External dependency lookup is a genuine Serena capability when IDE indexes are available; ordinary built-ins do not provide it. + +### 3.7 Single-file edits across edit sizes + +Small tweak attempted: change `"No symbol found for "` to `"No matching symbol found for "` in `SymbolFinder/findSymbolByNamePath`. + +Built-in call chain: + +1. Use existing search/body context. +2. `apply_patch` one-line replacement. +3. `git diff`, `git status --short`. +4. Revert and clean check. + +Serena call chain: + +1. `replace_symbol_body(SymbolFinder/findSymbolByNamePath, full method body with changed string)`. +2. `git diff`, `git status --short`. +3. Revert and clean check. + +Observed diff: both produced the same 1-line change. Payload difference: built-in edit input was a tiny hunk; Serena input was the full 11-line method body. + +Medium rewrite attempted: rewrite `SymbolFinder/findFilesByName` from list accumulation to stream collection. + +Built-in call chain: `apply_patch` over the method body, then diff/status/revert/status. + +Serena call chain: `replace_symbol_body(SymbolFinder/findFilesByName, new method body)`, then diff/status/revert/status. + +Observed diff: both produced `10 +++-------`, 3 insertions and 7 deletions. Payload difference was small: a patch hunk versus an 8-line replacement method. + +Large/whole-body rewrite attempted: rewrite `InspectionRunner/collectResults` while preserving signature and behavior shape. + +Built-in call chain: `apply_patch` against the method body, then diff/status/revert/status. + +Serena call chain: `replace_symbol_body(InspectionRunner/collectResults, full symbol text including Javadoc and replacement body)`, then diff/status/revert/status. + +Observed diff: both produced `38 +++++++++++-----------`, 19 insertions and 19 deletions. Payload difference: built-in required a large contextual hunk; Serena required the full symbol text, about 70 lines including Javadoc/signature/body. + +**Verdict:** Serena is not automatically more token-efficient for edits; its edit advantage is stable symbol targeting, while built-ins are smaller for tiny local hunks. + +### 3.8 Insert method at structural location + +Attempted: insert `private Logger getLogger()` immediately after `SymbolFinder/getProject`. + +Built-in call chain: + +1. Search/read surrounding area. +2. `apply_patch` with nearby context. +3. Diff/status/revert/status. + +Serena call chain: + +1. `insert_after_symbol(relative_path=SymbolFinder.java, name_path=SymbolFinder/getProject, body=...)` +2. Diff/status/revert/status. + +Observed diff: both produced the same 4-line insertion. + +Payloads: built-ins sent inserted body plus surrounding text context; Serena sent inserted body plus symbol target. + +**Verdict:** Structural insertion is a modest Serena improvement: same resulting diff, less dependence on line/context stability. + +### 3.9 Rename: private helper in one file + +Attempted: rename `SymbolFinder/qualNameMatchesName` to `qualifiedNameMatchesName`. + +Built-in call chain: + +1. `rg "qualNameMatchesName" SymbolFinder.java -n -C 2` +2. `apply_patch` declaration and one call site. +3. Diff/status/revert/status. + +Serena call chain: + +1. `find_referencing_symbols(SymbolFinder/qualNameMatchesName)` -> one referencing method. +2. `rename(name_path=SymbolFinder/qualNameMatchesName, new_name=qualifiedNameMatchesName, rename_in_comments=true, rename_in_text_occurrences=true)`. +3. Diff/status/revert/status. + +Observed result: Serena returned `"Success"`. Both produced the same 2-line diff. + +**Verdict:** Semantic rename has little advantage for a one-file helper with one call site, but it scales better than text edits as references spread or names become ambiguous. + +### 3.10 Rename: cross-file class including imports + +Attempted: rename `MoveOperationFailedException` to `MoveFailedException`. + +Serena call chain: + +1. `rename(relative_path=MoveOperationFailedException.java, name_path=MoveOperationFailedException, new_name=MoveFailedException, rename_in_comments=true, rename_in_text_occurrences=true)`. +2. `git diff --stat`, `git diff`, `git diff --cached --stat`, `git status --short`. +3. Restore staged rename and modified files; clean check. + +Serena result: `"Success"`. + +Observed edits: + +- Added/staged `MoveFailedException.java`. +- Deleted/staged `MoveOperationFailedException.java`. +- Updated import, Javadoc comment, and thrown class in `MoveProcessor.java`. + +Built-in equivalent chain: + +1. `rg "MoveOperationFailedException" src/main/java -n`. +2. Rename/create/delete file. +3. Patch class name and constructor. +4. Patch imports and usages. +5. Patch comments/text occurrences if desired. +6. Verify no unintended old references remain. +7. Compile/test for a permanent change. + +**Verdict:** Cross-file class renames are high-value Serena cases because file/class coupling, imports, usages, and optional comments are handled as one semantic refactor. + +### 3.11 Move symbol to another file context + +Attempted: move nested record `InspectionRunner/PendingFix` to a top-level file in the same package. + +Serena call chain: + +1. `move(relative_path=InspectionRunner.java, name_path=InspectionRunner/PendingFix, target_relative_path=src/main/java/de/oraios/serena/service/endpoint)`. +2. Diff/status/cached-diff inspection. +3. Restore staged new file and modified files; clean check. + +Serena result: + +```json +{ + "source_relative_path": "src/main/java/de/oraios/serena/service/endpoint/InspectionRunner.java", + "target_relative_path": "src/main/java/de/oraios/serena/service/endpoint", + "moved_symbol": {"name_path": "PendingFix", "type": "CLASS"} +} +``` + +Observed edits: + +- Created/staged `PendingFix.java` with package and imports. +- Removed nested record from `InspectionRunner.java`. +- Updated 3 external references from `InspectionRunner.PendingFix` to `PendingFix`. +- Removed unused imports in 2 files. + +Built-in equivalent chain: read nested record/import needs, add file, delete nested symbol, search usages, patch qualified usages, patch internal references if needed, remove imports, verify, compile/test. + +**Verdict:** Symbol move is a substantial Serena addition because the difficult part is reference/import repair, not copying text. + +### 3.12 Move file/package location + +Attempted: move `PluginUtil.java` from `util` to `service`. + +Serena call chain: + +1. `move(relative_path=src/main/java/de/oraios/serena/util/PluginUtil.java, target_relative_path=src/main/java/de/oraios/serena/service)`. +2. Diff/status/cached-diff inspection. +3. Restore staged rename and modified imports; clean check. + +Serena result: + +```json +{ + "source_relative_path": "src/main/java/de/oraios/serena/util/PluginUtil.java", + "target_relative_path": "src/main/java/de/oraios/serena/service" +} +``` + +Observed edits: + +- Staged rename `util/PluginUtil.java -> service/PluginUtil.java`. +- Changed package declaration to `de.oraios.serena.service`. +- Updated import in `PluginStartupActivity`. +- Removed now-unneeded same-package import in `SerenaBackendService`. + +Built-in equivalent chain: move file, patch package declaration, search `PluginUtil`, patch imports/usages, remove redundant imports, verify no old package import remains, compile/test. + +**Verdict:** File moves are high-value when packages/imports must change; built-ins can do them manually but require dependency repair steps. + +### 3.13 Safe delete and propagated delete + +Attempted: delete unused `DebugUtil`. + +Serena call chain: + +1. `safe_delete(relative_path=DebugUtil.java, name_path=DebugUtil, delete_even_if_used=false, propagate=false)`. +2. `git diff --cached --stat`, `git status --short`. +3. Restore staged deletion; clean check. + +Serena result: + +```json +{ + "deleted_symbol": "DebugUtil", + "relative_path": "src/main/java/de/oraios/serena/util/DebugUtil.java", + "affected_references": [], + "message": "Symbol deleted successfully with no affected references." +} +``` + +Built-in equivalent chain: `rg "DebugUtil"`, inspect declaration vs usages, delete file/symbol, verify no remaining usages. + +Propagated deletion: no suitable correct-use candidate was found where JetBrains safe-delete propagation would meaningfully delete a used symbol and propagate deletion through call sites. For ordinary used-method deletion, correct safe delete reports usages or, if forced, deletes the symbol and reports affected references; that is not the same as arbitrary call-site deletion. + +**Verdict:** Safe delete adds an integrated usage check plus deletion operation; propagated deletion was not evaluated because this codebase did not provide a suitable candidate. + +### 3.14 Inline + +Attempted: inline `ProjectUtil/getAbsolutePath`, a single-expression helper. + +Serena call chain: + +1. `find_symbol(ProjectUtil/getAbsolutePath, include_body=true)`. +2. `find_referencing_symbols(ProjectUtil/getAbsolutePath)` -> 2 references. +3. `inline_symbol(relative_path=ProjectUtil.java, name_path=ProjectUtil/getAbsolutePath, keep_definition=false)`. +4. Diff/status/revert/status. + +Serena result: `{"status": "SUCCESS"}`. + +Observed edits: + +- Removed `getAbsolutePath`. +- Replaced internal call in `ProjectUtil/getVirtualFile`. +- Replaced external call in `RefreshFileHandler`. + +Built-in equivalent chain: read helper body, `rg getAbsolutePath`, patch each call site with substituted expression, delete helper, verify no stale references, compile/test. + +**Verdict:** Inline is a strong Serena refactor for legally substitutable helpers; built-ins can reproduce it but must manually adapt each substitution. + +### 3.15 Scope precision and overload targeting + +Attempted: distinguish overloaded `Symbol/getLocationString` methods. + +Serena call chain: + +1. `find_symbol(Symbol/getLocationString[0], include_body=true)`. +2. `find_symbol(Symbol/getLocationString[1], include_body=true)`. + +Serena output: + +- `[0]`: `public String getLocationString() { return getLocationString(element); }` +- `[1]`: `private static String getLocationString(PsiElement element) { ... }` + +Built-in call chain: `rg "getLocationString" Symbol.java -n -C 1`, then manual signature inspection. + +**Verdict:** Serena's overload-indexed name paths provide precise symbol addressing that text search does not. + +### 3.16 Atomicity and success signals + +Observed Serena success signals: + +- `replace_symbol_body` -> `OK` +- `insert_after_symbol` -> `OK` +- `rename` -> `"Success"` +- `move` -> JSON with source/target/moved symbol or paths +- `safe_delete` -> JSON with deleted symbol, affected references, and message +- `inline_symbol` -> `{"status": "SUCCESS"}` + +Built-in success signals: + +- `apply_patch` -> "Success. Updated the following files" +- Shell/Git commands -> exit codes and textual output +- Cross-file consistency -> user-driven diff/search/build verification + +Atomicity comparison: Serena refactors executed as single IDE operations. Built-in equivalents are chains of searches, file operations, and patches; a competent user can make them correct, but intermediate states can be partial. + +Tradeoff: class/file move/delete refactors staged added/deleted files in the Git index, so clean verification must check both staged and unstaged state. + +**Verdict:** Serena gives clearer semantic success signals and more atomic cross-file edits; built-ins require manual consistency management. + +### 3.17 Workflow effects across multiple edits + +Attempted: chain 3 edits in `SymbolFinder.java` without refreshing between them. + +Serena call chain: + +1. `replace_symbol_body(SymbolFinder/findFilesByName, ...)`. +2. `insert_after_symbol(SymbolFinder/getProject, ...)`. +3. `rename(SymbolFinder/qualNameMatchesName, ...)`. +4. Diff/status/revert/status. + +Observed result: all edits applied after earlier edits shifted file positions. Final diff touched 18 lines with net 9 insertions and 9 deletions. + +Built-in equivalent chain: initial read/search, apply first patch, rely on robust patch context or refresh line numbers, apply second patch, refresh or search again, apply rename patch, verify. + +**Verdict:** Serena's name paths remain useful after line shifts, so its advantage compounds in multi-edit sessions. + +### 3.18 Non-code reads and free-text search + +Attempted: read config and search free text. + +Built-in call chain: + +1. `Get-Content plugin.xml`. +2. `Get-Content build.gradle.kts -TotalCount 80`. +3. `rg "http://|https://|localhost|127\\.0\\.0\\.1|8080|24282|SERENA|serena" src/main/java src/main/resources -n`. + +Serena call chain: not applicable. + +Observed result: built-ins exposed plugin registration, Gradle platform setup, URLs, IDs, settings defaults, and resource/text mentions. + +**Verdict:** Built-ins are natural and sufficient for config reading and free-text search; this is outside Serena's semantic-code scope. + +## 4. Token-efficiency analysis + +Payload differences across edit sizes: + +| Task | Built-in payload | Serena payload | More efficient | +|---|---|---|---| +| 1-line string tweak | Tiny patch hunk, 1 changed line | Full 11-line method body | Built-ins | +| Medium method rewrite | 10-line diff hunk | Full 8-line replacement method | Roughly equal | +| Large body rewrite | Large contextual patch, 38-line diff | Full symbol text, about 70 lines | Roughly equal; Serena has better target | +| Insert method | Inserted body plus surrounding context | Inserted body plus name path | Serena slightly | +| Private one-file rename | `rg` plus 2-line patch | optional refs query plus rename command | Roughly equal | +| Cross-file rename/move/inline | Multiple searches, file operations, patches, verifications | One semantic refactor plus diff/status | Serena | + +Forced reads: + +- Built-ins commonly need a search before a precise read, then a line slice or full context before editing. +- Serena can skip full-file reads when the symbol name path is known. +- When the symbol is not known, Serena's `get_symbols_overview` and shallow `find_symbol` calls provide compact discovery rather than full-file reads. + +Output payload: + +- Serena exploration output is structured and scoped to symbols. +- Built-in grep output can be larger because it includes declarations, comments, strings, and unrelated text mentions. +- Built-ins can be extremely low-output for known-location small edits. + +Stable vs ephemeral addressing: + +- Built-ins address code by file paths, line numbers, text context, or byte positions. Line numbers and slices go stale after edits. +- Serena addresses symbols by `relative_path + name_path`, with overload indexes where needed. The chained-edit experiment showed those targets survived line shifts. +- Path-changing operations still require updated paths afterward, so symbol stability is strongest before moves/renames that alter file locations. + +**Verdict:** Serena saves tokens by avoiding broad reads/search triage and by collapsing multi-file edit chains; built-ins remain more token-efficient for tiny known-location hunks. + +## 5. Reliability & correctness under correct use + +Precision of matching: + +- Serena reference search returned code usages and excluded route strings/comments. +- Built-in text search returned all mentions and therefore mixed true code usage with broader text hits. +- Serena overload indexes targeted `Symbol/getLocationString[0]` and `[1]`; text search required manual signature inspection. + +Scope disambiguation: + +- Serena name paths encode class nesting and overload identity. +- Built-ins can disambiguate with careful patterns and reads, but the disambiguation is manual. +- In simple cases, such as a private helper with one call site, this semantic precision had little practical value. In overloaded, nested, or cross-file cases, it materially reduced ambiguity. + +Atomicity: + +- Serena refactors apply through JetBrains refactoring operations and produce one operation-level success signal. +- Built-in equivalents are decomposed into separate searches, file moves, patches, and verification steps. +- Competent users can verify either path, but Serena reduces the number of intermediate inconsistent states. + +Semantic queries vs text search: + +- `type_hierarchy` and dependency declaration lookup produced semantic information not available from ordinary grep. +- `rg` remained better for broad mention searches and config/resource scans. + +External dependency limitations: + +- Serena's dependency lookup depends on IDE/language indexes and available dependency metadata. +- Built-ins can match or exceed it only if paired with source jars, decompilers, or separate language-server tooling. + +Operational tradeoff: + +- Several Serena refactors staged created/deleted files. This is not a correctness problem, but it adds index-state verification to the cleanup workflow. + +**Verdict:** Serena improves correctness where symbol identity and cross-file semantics matter; built-ins remain reliable and simpler for text/file tasks. + +## 6. Workflow effects across a session + +Where Serena advantages compound: + +- Symbol overview results become inputs to body reads, reference searches, renames, moves, safe deletes, and inline operations. +- Name paths remain useful after line shifts, reducing re-read/re-target work during multi-edit sessions. +- Refactor-heavy sessions benefit because one semantic command replaces search/edit/verify loops across files. + +Where Serena advantages diminish: + +- After a full file has already been read, overview adds less incremental value. +- If the task is a small textual substitution in a known place, symbolic body replacement can cost more payload than a tiny patch. +- For non-code and free-text work, Serena has no role. + +Intermediate result durability: + +- Serena intermediate results such as `SymbolFinder/findFilesByName` or `Symbol/getLocationString[1]` remain meaningful as long as the file/symbol still exists. +- Built-in line numbers became stale after insertions; robust textual patch context remained usable when surrounding lines did not change. + +Verification cost: + +- Both workflows still need `git diff`, build/test, and domain-specific verification for permanent changes. +- Serena success signals reduce the need to inspect every call site manually, but not the need to inspect the intended diff. + +**Verdict:** Serena's advantages compound inside symbol-centric sessions and diminish once the work becomes plain text, config, shell, or already-loaded single-file editing. + +## 7. Unique capabilities + +Unique here means no practical built-in equivalent without adding separate language-server, IDE, source-index, or decompiler infrastructure. + +- Transitive type hierarchy. Frequency: medium. Impact: high when changing base classes, interfaces, or handler contracts. Built-ins can grep explicit `extends` clauses but do not produce a semantic transitive graph or external supertypes. + +- External dependency declaration/body lookup from a call site. Frequency: low-medium. Impact: high for API behavior questions. Serena resolved `ReferencesSearch.search` into an external class method body; ordinary built-ins did not. + +- IDE semantic move of a nested symbol to a top-level file with usage/import repair. Frequency: low-medium. Impact: high during refactors. Built-ins can reproduce this only as a manual multi-step refactor. + +- Safe delete with semantic usage check. Frequency: medium. Impact: medium-high. Built-ins can search and delete, but they do not provide an integrated safe-delete operation over code usages. + +- Inline refactor with call-site substitution. Frequency: low-medium. Impact: medium-high. Built-ins can patch substitutions manually, but expression adaptation and helper deletion are not primitive text operations. + +Not unique: reading files, listing files, grep, small patches, simple one-file renames, config/resource understanding, and shell/Git operations. + +**Verdict:** Serena's unique practical capabilities are hierarchy/dependency queries and IDE refactors; its non-unique areas are ordinary text and file operations. + +## 8. Tasks outside Serena's scope + +Built-in-only or built-in-natural tasks observed: + +- Repository inventory and top-level layout. +- Reading `plugin.xml`, `build.gradle.kts`, scripts, resources, and generated artifacts. +- Free-text search for URLs, settings strings, endpoint strings, magic constants, resource IDs, and comments. +- Running shell commands, builds, tests, Git operations, and filesystem cleanup. +- Small text edits where a one-line patch is the whole task. + +Estimated share of daily coding work: + +- About 30-50% of a normal coding session is built-in-natural: shell, tests, grep, config reads, small text patches, and Git checks. +- Serena applies to much of the remaining 50-70% when the work is code navigation, symbol understanding, references, hierarchy, or refactoring. +- Serena coverage rises in refactor-heavy sessions and falls in config/debug/test-log sessions. + +**Verdict:** Serena augments the semantic-code portion of development; built-ins still carry a large and necessary share of routine work. + +## 9. Practical usage rule + +Use Serena when the task can be stated as a code-symbol operation: + +- "What methods/classes are in this file?" +- "Give me this method body." +- "Who uses this symbol?" +- "What implements/subclasses this?" +- "Rename, move, delete, or inline this symbol and update usages." +- "Find the declaration of this call, including dependency symbols." +- "Apply several edits where line locations may shift." + +Use built-ins when the task is text, file, shell, config, or tiny local editing: + +- "List files." +- "Read config." +- "Search for this string anywhere." +- "Change this one known line." +- "Run tests/build." +- "Inspect Git state." +- "Edit markdown/resources/scripts." + +For mixed workflows: + +1. Use built-ins to find files/config and run verification. +2. Use Serena to identify and transform code symbols. +3. Use built-ins to inspect final diffs and run tests. +4. Prefer built-ins for tiny local hunks after the symbol has already been located. +5. Prefer Serena for any refactor that crosses file boundaries or depends on type/reference semantics. + +**Verdict:** Choose Serena for symbol identity and semantic refactoring; choose built-ins for text, shell, config, verification, and minimal local edits. diff --git a/docs/_config.yml b/docs/_config.yml index 999c6a62..42e89034 100644 --- a/docs/_config.yml +++ b/docs/_config.yml @@ -102,7 +102,7 @@ sphinx: local_extensions : # A list of local extensions to load by sphinx specified by "name: path" items recursive_update : false # A boolean indicating whether to overwrite the Sphinx config (true) or recursively update (false) config : # key-value pairs to directly over-ride the Sphinx configuration - master_doc: "01-about/000_intro.md" + master_doc: "01-about/000_evaluation-results.md" html_theme_options: logo: image_light: ../resources/serena-logo.svg diff --git a/docs/autogen_docs.py b/docs/autogen_docs.py index d20eaa92..6f574ab5 100644 --- a/docs/autogen_docs.py +++ b/docs/autogen_docs.py @@ -190,7 +190,7 @@ def autogen_about_intro_features(): autogen_info = f"\n\n" - with open(Path(__file__).parent / "01-about" / "000_intro.md", "w", encoding="utf-8") as f: + with open(Path(__file__).parent / "01-about" / "000_evaluation-results.md", "w", encoding="utf-8") as f: f.write(autogen_info) f.write("# About Serena\n\n") f.write(f"**{tagline}**\n\n") diff --git a/src/serena/tools/jetbrains_tools.py b/src/serena/tools/jetbrains_tools.py index a94d5c99..9e217090 100644 --- a/src/serena/tools/jetbrains_tools.py +++ b/src/serena/tools/jetbrains_tools.py @@ -58,6 +58,8 @@ class JetBrainsFindSymbolTool(Tool, ToolMarkerSymbolicRead, ToolMarkerOptional): for the case where the symbol is a class, this will return its methods). Ignored if `include_body=True`. Default 0. :param relative_path: Optional. Restrict search to this file or directory. If not specified, searches entire codebase. + Note: for external dependencies, this must be an identifier starting with ` str: + # some clients replace < and > with their escaped html versions, we need to counteract this + return raw_param.replace("<", "<").replace(">", ">") + @classmethod def set_last_tool_call_client_str(cls, client_str: str | None) -> None: cls._last_tool_call_client_str = client_str