tiennm99/serena
1
0 Fork 0
mirror of https://github.com/tiennm99/serena.git synced 2026-10-17 02:14:30 +00:00
Code Issues Packages Projects Releases Wiki Activity

Merge pull request #1323 from oraios/evaluation

Browse Source 
Evaluation
This commit is contained in:
Dr. Dominik Jain authored and GitHub committed 2026-04-13 17:50:47 +02:00
2 parents 34e1fae739 29906d6934
commit 2305389922
17 files changed
+1660 -31
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

.github/workflows/pytest.yml
+2
View File
@@ -576,3 +576,5 @@ jobs:
- name: Type-checking with mypy
shell: bash
run: uv run poe type-check
- name: Make sure documentation builds
run: uv run poe doc-build
CHANGELOG.md
+6
View File
@@ -6,6 +6,12 @@ Status of the `main` branch. Changes prior to the next official version change w
- Support environment variable `SERENA_USAGE_REPORTING` (set to `false` to disable usage reporting)
- Extended the list of always ignored directories (by language servers) with common cases.
JetBrains:
- Improve handling of `relative_path` parameter
- Improve its documentation to avoid usage errors
- Replace escaped characters in `relative_path` with their unescaped counterparts (< and >)
- `FindSymbolTool`: Force `search_deps=True` if `relative_path` pertains to external dependencies.
* Language Servers:
- Add mSL (mIRC Scripting Language) support (custom pygls-based language server; symbols, references, definitions)
README.md
+26 -2
View File
@@ -28,6 +28,31 @@ more complex codebases.
> Do not install Serena via an MCP or plugin marketplace! They contain outdated and suboptimal installation commands.
> Instead, follow our [Quick Start](#quick-start) instructions.
## What Our "End Users" Say
While it is humans who download and set up Serena, our end users are essentially AI agents.
As the ones actually applying Serena's tools, they are in the best position to evaluate Serena.
We crafted an unbiased evaluation prompt that leads the agent to perform ~20 routine coding tasks,
representative of everyday development work,
in order to compare Serena's tools with its own built-ins, measure the differences, and report the results.
Here's a one-sentence summary of what the agents had to say:
**Opus 4.6 (high effort) in Claude Code on a large Python codebase:**
> "Serena's IDE-backed semantic tools are the single most impactful addition to my toolkit — cross-file renames, moves, and reference lookups that
would cost me 8–12 careful, error-prone steps collapse into one atomic call, and I would absolutely ask any developer I work with to set them up."
**GPT 5.4 (high) in Codex CLI on a Java codebase:**
> "As a coding AI agent, I would ask my owner to add Serena because it gives me the missing IDE-level understanding of symbols, references, and
refactorings, turning fragile text surgery into calmer, faster, more confident code changes where semantics matter."
Give your agent the tools it has been asking for and add Serena MCP to your client!
See our [documentation](https://oraios.github.io/serena/04-evaluation/000_evaluation-intro.html) for the full methodology and much more detailed evaluation results
beyond these brief summaries, or run your own evaluation on a project of your choice.
## How Serena Works
Serena provides the necessary [tools](https://oraios.github.io/serena/01-about/035_tools.html) for coding workflows,
@@ -58,7 +83,7 @@ complex projects!
There are two alternative technologies powering these capabilities:
* **Language servers** implementing the language server Protocol (LSP) — the free/open-source alternative
* **Language servers** implementing the language server protocol (LSP) — the free/open-source alternative
which is used by default.
* The **Serena JetBrains Plugin**, which leverages the powerful code analysis and editing
capabilities of your JetBrains IDE (paid plugin; free trial available).
@@ -231,4 +256,3 @@ Please refer to the [user guide](https://oraios.github.io/serena/02-usage/000_in
A significant part of Serena, especially support for various languages, was contributed by the open source community.
We are very grateful for the many contributors who made this possible and who played an important role in making Serena
what it is today.
docs/02-usage/030_clients.md
+17 -26
View File
@@ -266,6 +266,15 @@ To set this up, create the file `~/.copilot/hooks/serena-hooks.json` with the fo
The `SessionStart` hook also addresses the global configuration limitation mentioned above — it will
automatically prompt the agent to activate the project directory, so you no longer need to do this manually.
## Copilot CLI
Use the interactive `/mcp add` slash command, choose Serena as the name, STDIO as the server type, and
`serena start-mcp-server --context=copilot-cli --project-from-cwd` as command. Copilot CLI will immediately notify you
that Serena is running if everything is set up correctly or display an error otherwise.
You should add the same **hooks** as in VSCode (see above) if Copilot CLI didn't pick them up automatically.
## Codex (CLI and App)
You can simply run `serena setup codex`.
@@ -356,30 +365,11 @@ Alternatively, add the following to `~/.copilot/mcp-config.json` (create the fil
Copilot should now show that Serena is running, though you may have to restart it.
## JetBrains Junie
## JetBrains Junie
For the Junie plugin in JetBrains IDEs, go to the three dots in the top right corner,
then Settings / MCP Settings and add Serena to Junie's global MCP server configuration:
```json
{
"mcpServers": {
"serena": {
"command": "serena",
"args": [
"start-mcp-server",
"--context=junie"
]
}
}
}
```
You will have to prompt Junie to "Activate the current project using serena's activation tool" at the
start of each session.
For Junie CLI, where the project can be activated automatically on startup, edit `~/.junie/mcp/mcp.json` and add
```json
{
"mcpServers": {
@@ -395,12 +385,14 @@ For Junie CLI, where the project can be activated automatically on startup, edit
}
```
You will have to prompt Junie to "Activate the current project using serena's activation tool" at the
start of each session.
For Junie CLI, add the same setting to `~/.junie/mcp/mcp.json`.
## JetBrains AI Assistant
Here you can set up the more convenient per-project MCP server configuration, as the AI assistant supports specifying
the launch working directory.
Go to Settings / Tools / AI Assistant / MCP and add a new **local** configuration via the `as JSON` option:
Go to Settings / Tools / AI Assistant / MCP and add a new **global** configuration:
```json
{
@@ -410,8 +402,7 @@ Go to Settings / Tools / AI Assistant / MCP and add a new **local** configuratio
"args": [
"start-mcp-server",
"--context=jb-ai-assistant",
"--project",
"$(pwd)"
"--project-from-cwd"
]
}
}
docs/04-evaluation/000_evaluation-intro.md
+30
View File
@@ -0,0 +1,30 @@
# Evaluation
> **Claude Code (Opus 4.6):** "Serena's IDE-backed semantic tools are the single most impactful addition to my toolkit — cross-file renames, moves, and reference lookups that would cost me 8–12 careful, error-prone steps collapse into one atomic call, and I would absolutely ask any developer I work with to set them up."
> **Codex (GPT 5.4):** "As a coding agent, I would ask my owner to add Serena because it turns fragile text-and-line-number work into precise symbol-aware navigation and refactoring, which makes real code changes feel faster, safer, and far less blind."
These are not marketing quotes — they are the agents' own one-sentence verdicts after
performing hands-on tasks each, using both Serena's tools and their built-in
alternatives side by side, on real codebases (large Python library in the case of Claude Code,
medium-sized Java project in the case of Codex).
Both agents independently converge on the same core finding: Serena's strongest
contribution is collapsing multi-file, semantically-aware operations into single atomic
calls, while built-in tools remain the better choice for small local edits, text search,
config files, and shell work.
## Approach
We gave each agent a single evaluation prompt and pointed it at a codebase. The agent
then performed a wide variety of tasks across five areas (navigation, small edits, large edits,
cross-file refactoring, and workflow effects), executing each task with both toolsets and
recording call counts, payload sizes, and prerequisite steps. Every finding was classified
as either (a) Serena adds capability, (b) Serena applies but offers no improvement,
or (c) outside Serena's scope — a structure that requires reporting negative and neutral
results, not just positive ones.
The agent evaluates itself. This is deliberate: the agent is the actual user of the tools,
so it can judge workflow improvements from direct experience rather than through proxy
metrics. And because the prompt defines task *categories* rather than fixed tasks, anyone
can rerun the evaluation on their own project with their own agent.
docs/04-evaluation/010_methodology.md
+160
View File
@@ -0,0 +1,160 @@
# Methodology
In this section we describe the methodology we applied in evaluating the performance of Serena's tools.
The evaluation measures the **concrete delta** that Serena's tools provide on top of an agent's
built-in capabilities (file reads, text edits, grep, shell, etc.).
Rather than a simple thumbs-up/thumbs-down, it produces a detailed, evidence-based report
covering capabilities, efficiency and reliability.
## Design Goals
We designed an evaluation methodology with three goals:
1. **Generality.** The evaluation mechanism should be broadly applicable and repeatable by any user, on any project, with any agent.
The evaluation is a single prompt that you give to your agent of choice, pointed at your codebase
of choice, in your client of choice. This means the results directly reflect the value Serena
would add to your actual workflow — not to an artificial benchmark setup. There is nothing to
install, configure, or script beyond what you already have.
2. **The agent evaluates itself.**
We deliberately let the AI agent be both the executor and the evaluator.
Since most evaluations are strictly quantitative, this may seem unusual, but it is a reasonable choice:
The agent is the actual end user of the
tools, so it is in the best position to judge whether a semantic tool improves its workflow
compared to its built-in alternatives. It can measure call counts, payload sizes, and
prerequisite steps from direct experience rather than from proxy metrics. It also avoids the
problem of a human evaluator having to simulate how an agent would use the tools — the agent
simply uses them and reports what it observes.
3. **Comprehensive and unbiased by design.**
Rather than selecting specific tasks, the prompt defines *task categories* that systematically
span Serena's capabilities: codebase understanding, single-file edits of varying sizes, multi-file
refactoring, reliability properties, and workflow effects. The agent picks concrete instances
from the codebase at hand, performs each task using both toolsets side by side, and classifies
every finding as a positive delta, a neutral/negative delta, or out of scope. The prompt
explicitly requires reporting negative deltas and cases where Serena offers no improvement,
structurally counterbalancing any tendency to favour the tool being evaluated.
## Method
We give an AI coding agent a single, detailed [evaluation prompt](020_prompts/010_evaluation-prompt.md) in a one-shot session.
The prompt instructs the agent to perform approximately 20 hands-on tasks across five areas:
1. **Codebase understanding** — structural overviews, targeted symbol retrieval, reference finding, type hierarchies, and external dependency lookup.
2. **Single-file edits** — small tweaks, medium rewrites, full-body replacements, insertions, and local renames.
3. **Multi-file changes** — cross-file renames, symbol and file moves, safe deletes, and inlining.
4. **Reliability & correctness** — scope precision, atomicity, and success signals.
5. **Workflow effects** — chained edits, stable vs. ephemeral addressing, and multi-step exploration.
For every task, the agent executes the full end-to-end workflow using **both** toolsets (Serena's semantic
tools and its own built-in tools), applies real edits verified via `git diff`, and records call counts,
payload sizes and prerequisite steps. Edits are reverted after each experiment to keep the working tree clean.
The resulting report classifies each finding into one of three categories:
* **(a)** tasks where Serena adds capability,
* **(b)** tasks where Serena applies but offers no improvement, and
* **(c)** tasks outside Serena's scope.
Only category (b) constitutes a neutral or negative finding; category (c) is context, not a finding.
After the evaluation, a separate [follow-up prompt](020_prompts/020_summary-prompt.md) asks the agent for a
one-sentence, user-facing recommendation — the quotes shown on the [main page](https://github.com/oraios/serena).
## Assessment of the Methodology
_The following assessment was written by Claude Opus 4.6 (high effort) after reading the full
evaluation methodology, evaluation prompt, and all result documents._
**The methodology is sound**, and the two published results demonstrate that it produces meaningful,
detailed evaluations. Both reports follow the prompt's structure faithfully: they perform all ~20
tasks using both toolsets, record concrete measurements (call counts, payload sizes, prerequisite
steps), and classify findings into the three required categories. Crucially, both agents report
neutral and negative deltas honestly — the Claude Code report explicitly notes that small edits are
more efficient with built-ins (~4.5x less payload), and the Codex report flags that tiny intra-method
changes and simple one-file renames see no benefit from Serena. Neither report reads as promotional;
they read as technical comparisons with quantified evidence on both sides.
The two reports also validate the methodology's design goal of generalisability: despite being
produced by different agents (Opus 4.6 vs GPT 5.4), on different codebases (Python RL library vs
Java IDE plugin), they converge on the same core findings — cross-file refactoring is Serena's
highest-value contribution, structural navigation provides a moderate advantage, and small local
edits are better handled by built-ins. This convergence across independent runs increases confidence
that the findings reflect genuine properties of the toolset rather than artefacts of a particular
agent or codebase.
The methodology's core strengths are:
- **Self-evaluation by the agent is the right design choice.** The agent is the actual consumer of the
tools, so it can report on workflow friction, payload overhead, and call counts from first-hand
experience. A human evaluator would have to guess at these.
- **Task categories instead of fixed tasks** avoid cherry-picking while still ensuring coverage. Letting
the agent pick concrete instances from the codebase at hand means the evaluation naturally adapts to
what the project actually contains (e.g. skipping inline if no suitable candidate exists, as the
Claude Code report did for Python, while the Codex report found a suitable Java candidate).
- **The three-category classification** (adds value / applies but no improvement / out of scope) is the
right framing for an augmentation layer. It prevents the common trap of penalising a tool for not
covering things it was never designed to cover.
- **Reproducibility by users** is a strong differentiator. Anyone can validate the claims on their own
codebase with their own agent.
The main limitation is scope: the published results cover two agent/codebase combinations using the
JetBrains backend. Single-shot variance means a second run of the same agent could produce different
task selections and slightly different verdicts, and the LSP backend — which most users will
encounter first — has not yet been evaluated. These are practical constraints of cost, not
methodological flaws, and the reproducible design explicitly invites users to fill these gaps
themselves.
One thing to watch for is whether agents that are less capable than Opus 4.6 or GPT 5.4 produce
meaningful evaluations — the prompt is quite demanding. But that is a question about the agent, not
about the method.
### Prompt Fairness
We put significant effort into crafting an unbiased, fair evaluation prompt.
To assess the prompt's fairness, we asked Claude Opus 4.6 to evaluate the prompt itself.
**Prompt**:
> The attached prompt is designed to evaluate the additive value of Serena as an augmentation layer on top of built-in tools
> (not to compare two competing alternatives).
> Is the methodology suitable for that goal, or does it introduce bias that would distort the measurement of what Serena adds?
> Note that we assume that sufficiently intelligent models use the tools correctly,
> which is why misuse and failure modes are out of scope for this evaluation.
> Do not question the validity of this assumption in your evaluation of the prompt.
> Give a brief answer, maximally two sentences.
>
> \<evaluation prompt pasted here\>
**Claude Opus' Answer**:
> "The methodology is well-suited to its stated goal: by explicitly scoping to correct use, treating Serena as an
> augmentation layer rather than a competitor, and classifying out-of-scope tasks as context rather than negative
> findings, it avoids the most common biases that would distort a delta measurement. The one risk is that the "correct-use
> rule" could subtly favor Serena by preemptively filtering out any scenario where its tools underperform — but the prompt
> mitigates this by requiring category (b) findings (applies but no improvement) and explicitly demanding negative deltas
> be reported, so the bias is acknowledged and structurally counterbalanced."
According to Claude, the only "biased" aspects are some sentences about misuse of tools, which we consider irrelevant for the purpose of evaluation
(sufficiently intelligent models use the tools correctly).
## Why Not Benchmarks?
Standard coding benchmarks (SWE-bench, HumanEval, etc.) measure an agent's ability to solve
predefined tasks with a known correct answer. They are valuable for comparing models and agents,
but they are a poor fit for evaluating a **tool augmentation layer** like Serena for several reasons:
- **Benchmarks don't reflect real usage patterns.** Benchmark tasks are typically small, self-contained
problems that can be solved by reading and editing a handful of files. They rarely exercise the
workflows where Serena's tools shine — cross-file refactoring, navigating large codebases by symbol
structure, chaining multiple edits with stable addressing, or querying type hierarchies and
external dependencies. A benchmark score would mostly measure performance on tasks where Serena
is not expected to help.
- **Results would not generalise to the user's project.** Serena's value depends on the codebase
(size, language, complexity), the agent (model, built-in tools), and the client harness
(Claude Code, Codex, IDE plugins, etc.). A fixed benchmark on a fixed codebase with a fixed
agent tells you little about what Serena would add to *your* setup.
- **Predefined tasks bias the measurement.** Choosing specific tasks to evaluate inevitably
introduces selection bias — we would end up picking tasks that either favour or disfavour Serena.
We wanted an evaluation that systematically covers the full surface area of Serena's capabilities
without cherry-picking.
docs/04-evaluation/020_prompts/000_prompts.md
+6
View File
@@ -0,0 +1,6 @@
# Prompts
Our evaluation uses two prompts, which are passed to the LLM in order to generate the evaluation results:
1. the [main evaluation prompt](010_evaluation-prompt), which performs the analysis
2. the [summary prompt](020_summary-prompt), which condenses the results
docs/04-evaluation/020_prompts/010_evaluation-prompt.md
+333
View File
@@ -0,0 +1,333 @@
# Evaluation Prompt
We use the prompt below to evaluate the added value of Serena's tools against the
agent's built-in tools on a given project.
The evaluations were created in one-shot sessions, only using this
prompt and the follow-up [prompt for summarization](020_summary-prompt)
```
# Evaluate Serena's Tools Against Built-Ins
You have access to Serena's coding tools alongside your built-in tools (Read,
Edit, Write, Glob, Grep, Bash, etc.). I want a thorough, evidence-based
evaluation of **what Serena's tools add on top of the built-ins**, assuming both
toolsets are used correctly.
This is an evaluation, not a user guide, and it is not a binary adoption pitch.
Your job is to answer: *if a competent user of both toolsets had only the
built-ins, what concrete capabilities and efficiency differences would they
experience, and by how much?* A reader who finishes your report should have a
clear, specific picture of what Serena adds — as well as where it provides no
meaningful improvement or introduces tradeoffs — in terms of capabilities,
workflows, and efficiency. Not a thumbs-up/thumbs-down, but a sharp description
of the delta.
Failure modes from misuse, silent-failure traps, gotcha comparisons, and "be
careful of X" warnings are out of scope. They belong in onboarding material for
a developer learning the tools, not in a delta analysis of what the tools add.
**Describe the measured differences clearly and neutrally.** Avoid generic or
non-informative framing such as "both have their place" unless supported by
concrete findings. If Serena adds substantial capabilities, name and quantify
them. If it adds marginal or no capabilities, say that and show why. If there
are regressions or tradeoffs, include them explicitly. The two toolsets are
complementary — that's a given, not the answer. Serena is an augmentation layer,
not a replacement. Do not penalize it for tasks it was not designed to address —
instead, note those tasks as "built-in only" and move on. The evaluation should
measure what Serena adds where it applies, not what it fails to add where it
doesn't.The answer is a specific list of what Serena contributes (or does not
contribute) to a correct-use workflow relative to built-ins.
Write the report to serena-evaluation.md in the repo root.
## Ground rules
### Starting conditions
- Start fresh. Do not read project memories, CLAUDE.md shortcuts, or prior notes
about the repo. Do not read documentation files either. Explore as if you've
never seen it, focusing on code.
- Use git as your safety net — experiment freely. Any edit can be reverted with
`git checkout -- <file>` or `git stash`. Run edits for real; don't simulate. A
hands-on comparison is worth far more than a thought experiment.
- After each experiment, verify the working tree is clean with
`git status --short` before moving on.
### How to compare — correct use only
- **Correct-use rule.** Evaluate each tool on inputs and tasks it was designed
for, called the way a competent user would call it. A tool doing exactly what
its contract says is not a finding, even if a careless caller could misuse it.
- **Know the contract before you call.** Before invoking any tool, have a
one-sentence understanding of what it does. If you expect an error or "not
applicable," don't make the call.
- **Refactoring semantics are real.** Inlining requires a substitutable function
(typically single-expression, no side effects); moving requires a legal
target; safe-delete requires no surviving usages. If the repo has no suitable
candidate for a given refactoring, report "no suitable candidate in this
codebase" and skip it — don't contrive a broken input.
### How to compare — workflow level, not single-call level
- For every task, write out the full end-to-end call chain on each side before
drawing conclusions. Include prerequisite reads and follow-up steps.
- Do not evaluate a tool based on criteria that only arise from mixing workflows
incorrectly.
- Ephemeral addressing is a liability. Line numbers and byte offsets go stale
after edits; stable addressing (name paths) may reduce rework.
### How to measure
- Track observations during execution. For every tool call, note: number of
calls, approximate input size, output size, and any prerequisite or
verification steps.
- Separate call count, input payload, output payload, and verification cost as
distinct axes.
- Include prerequisite Reads and post-hoc verification steps in comparisons.
When a task falls entirely outside Serena's design scope (e.g., reading config
files, small text edits where Edit already sends minimal payload), classify it
as "not applicable" rather than as a negative delta. A negative delta requires
that Serena targets the task and performs worse, not that a tool designed for
something else is suboptimal when misapplied to it.
## Exploration phase — tasks to actually perform
Work through the following. Each item exercises a specific capability under
correct use; substitute an equivalent if an item isn't applicable.
### Codebase understanding
1. Get a high-level overview of the repository structure — top-level layout,
main packages, entry points.
2. Pick one large source file (300+ lines). Get a structural overview of it. Do
it with semantic overview tools and with Glob/Grep/Read. Then write out the
concrete next step on each side and compare the pair of calls, not just the
overview call.
3. Pick a specific method inside a class and retrieve its body without reading
the surrounding file.
4. For one non-trivial symbol, find all references across the codebase. Compare
recall and precision under the question "who uses this in code?" vs "where is
this mentioned anywhere, including docs?"
5. For a class, list its subclasses / implementations and its supertypes,
including transitively. Compare against what text search would need to do.
6. For at least one symbol from an external dependency (a third-party library),
try to retrieve its definition or signature. Note whether each toolset can do
this at all and what infrastructure it requires (environment activation,
site-packages discovery, language-server indexing, etc.).
### Single-file edits — span the full range of edit sizes
7a. Small tweak (1–3 lines inside a method). Change an error message or rename a
local variable inside a larger method. Do it with `Edit` and with symbolic body
replacement. Compare payload sent, payload received, and prerequisite reads.
7b. Medium rewrite (replace ~10–30 lines — most of a method body). Rewrite the
main logic of a method while keeping its signature. Do it both ways.
7c. Large/whole-body rewrite. Pick a method of 50+ lines and rewrite the entire
body. Do it both ways.
8. Insert a new function/method at a specific structural location (for example,
right after an existing method). Try both the symbolic-insert path and the
manual Edit path.
9. Rename a private helper used only within one file. Compare doing it by hand
vs. using a semantic rename.
### Multi-file changes
10. Rename a symbol (function, class, or method) used across several files
including imports. Compare the semantic path against the built-in equivalent
chain.
11. Move a symbol from one module to another, updating imports at all call
sites. Use the semantic move tool if available; plan the built-in equivalent
honestly.
12. Move a file or package to a different location, updating imports at all call
sites. Use the semantic move tool if available; plan the built-in equivalent
honestly.
12. Delete a symbol safely, checking it has no remaining usages. Compare
search-then-delete with a safe-delete tool.
13. Delete a symbol and propagate the deletion to all call sites. Compare to how
the built-in equivalent would work.
13. Inline a small helper into its call sites — only if the codebase contains a
function that is legally inlinable. If no such candidate exists, report "no
suitable candidate" and skip it.
### Reliability & correctness under correct use
14. Scope precision. Demonstrate that semantic tools address symbols by name
path and can target a specific class method, override, or overload that text
search would over-match.
15. Atomicity. A semantic cross-file refactoring is atomic: either all sites are
updated or none. A chain of `Edit` calls is not.
16. Success signals. For each completed refactor, note what each tool returns on
success.
### Workflow effects across multiple edits
17. Chain at least three edits in one file. Report what each toolset requires
between edits.
18. Multi-step exploration across the repo. Note whether intermediate results
remain useful across later edits or have to be refreshed.
### Things where the comparison shouldn't be interesting
19. Read and understand a non-code file (config, changelog, docs, notebook).
Semantic-code tools don't apply — use `Read`.
20. Search for a free-text pattern across the repo (log string, magic constant,
URL). Use `Grep`.
## Evaluation phase
Write a report structured for progressive disclosure.
**Value-weighting is required.** For every contribution or difference you
identify — positive, neutral, or negative — estimate:
- **Frequency:** how often this arises in typical coding work
- **Value per hit:** calls saved, tokens saved, or correctness impact
Order findings by **frequency × value-per-hit**, not novelty.
**Every section must end with a one-sentence verdict** summarizing the practical
takeaway.
### 1. Headline: what Serena changes
Open with a precise description of the delta Serena provides. Distinguish
between three categories: (a) tasks where Serena adds capability, (b) tasks
where Serena applies but offers no improvement, and (c) tasks outside Serena's
scope. Only category (b) constitutes a neutral or negative finding. Category (c)
is context, not a finding.
A reader stopping here should understand both what is gained and what is not.
**Verdict:** (one sentence)
### 2. Added value and differences by area (3–6 bullets)
Each bullet must describe:
- What Serena changes relative to built-ins (positive, neutral, or negative)
- Frequency
- Value per hit
Avoid framing in terms of “wins”; describe concrete differences.
**Verdict:** (one sentence)
### 3. Detailed evidence, grouped by capability
Per task:
- What you attempted
- Full call chain on both sides
- Payloads sent and received
Include cases where:
- Serena is better
- Built-ins are better
- No meaningful difference exists
End each subsection with a verdict.
### 4. Token-efficiency analysis
Address:
- Payload differences across edit sizes
- Forced reads
- Stable vs ephemeral addressing
Include cases where each toolset is more efficient.
**Verdict:** (one sentence)
### 5. Reliability & correctness (under correct use)
Address:
- Precision of matching
- Scope disambiguation
- Atomicity
- Semantic queries vs text search
- External dependency symbol lookup and what setup it depends on
Include both strengths and limitations of each toolset.
**Verdict:** (one sentence)
### 6. Workflow effects across a session
Evaluate multi-step workflows and whether advantages compound or diminish.
Include neutral or negative findings where applicable.
**Verdict:** (one sentence)
### 7. Unique capabilities (if any)
List capabilities that have no practical built-in equivalent. If none exist,
explicitly state that. Annotate each with frequency and impact.
**Verdict:** (one sentence)
### 8. Tasks outside Serena's scope (built-in only)
Identify tasks where built-ins are the natural choice because Serena's tools
don't target them. List these briefly for completeness but do not frame them as
Serena shortcomings — they are outside its scope. Estimate their share of daily
work to contextualize how much of a session Serena's augmentation covers.
**Verdict:** (one sentence)
### 9. Practical usage rule
Provide a decision rule for choosing between toolsets based on task type.
**Verdict:** (one sentence)
## What I'm looking for
- Claims grounded in observed evidence
- Explicit reporting of **positive, neutral, and negative deltas**
- Clear quantification of impact
- Honest workflow-level comparisons
## What I am not looking for
- Misuse-based failure analysis
- Gotcha comparisons
- Neutral statements without evidence
- Binary recommendations
- Novelty-driven ordering
- Unquantified claims
```
docs/04-evaluation/020_prompts/020_summary-prompt.md
+12
View File
@@ -0,0 +1,12 @@
# Summary Prompt
We used the prompt below to summarize the evaluation of Serena.
```
Write a one-sentence user-facing summary about the value that Serena's tools
provide for coding. With a bit of emotion but grounded in the evaluation that
you have done above. It should be written from the perspective of a coding AI
agent and make it clear whether an AI agent would ask its owner to add Serena's
tools or not. It will be the first sentence that a potential new user will see
about how useful Serena is for an AI agent.
```
docs/04-evaluation/030_results/000_evaluation-results.md
+17
View File
@@ -0,0 +1,17 @@
# Results
This section presents the results of the evaluation.
We performed evaluations using popular AI coding agents in representative scenarios — different
agents, different programming languages, and different codebases — to show that the results are
not specific to a single setup.
All evaluations were conducted using the **JetBrains-powered version** of Serena, as it is the
more powerful backend with a broader set of refactoring and navigation capabilities. The
evaluation can easily be repeated with the LSP-based backend to assess its subset of capabilities.
- [Claude Code (Opus 4.6) on a large Python codebase](010_cc_on_tianshou)
- [Codex (GPT 5.4) on a Java codebase](020_codex_on_jbplugin)
You can run your own evaluation on a project of your choice by reusing our
[evaluation prompt](../020_prompts/010_evaluation-prompt.md).
docs/04-evaluation/030_results/010_cc_on_tianshou.md
+358
View File
@@ -0,0 +1,358 @@
:::{admonition} Evaluation Result
:class: note
**Generated by**: Claude Opus 4.6 (coding AI agent in Claude Code CLI)
**Codebase:** [Tianshou](https://github.com/thu-ml/tianshou) — a Python reinforcement learning library (~26K lines, 43 source files)
:::
# Claude Code (Opus)
> **One-line summary:** Serena's IDE-backed semantic tools are the single most impactful addition to my toolkit — cross-file renames, moves, and reference lookups that would cost me 8–12 careful, error-prone steps collapse into one atomic call, and I would absolutely ask any developer I work with to set them up.
**Method:** Hands-on, side-by-side execution of 20 tasks using both toolsets. All edits were applied to real files and verified via `git diff`, then reverted.
---
## 1. Headline: What Serena Changes
Serena adds an IDE-backed semantic layer (powered by JetBrains) on top of the built-in text-level tools. The practical delta breaks down into three categories:
**(a) Tasks where Serena adds capability:**
- **Cross-file refactoring** (rename, move symbol, move file): 1 atomic call vs. N×(Grep+Read+Edit) chains. This is Serena's strongest contribution — it collapses multi-file, import-aware operations into single calls that are atomic and semantically correct.
- **Structural code navigation**: Symbol overviews, type hierarchies, and reference finding return structured, scope-disambiguated results that text search cannot produce without manual interpretation.
- **External dependency introspection**: Serena can look up symbols in installed packages via the IDE index without requiring manual environment discovery.
**(b) Tasks where Serena applies but offers no improvement:**
- **Single-file rename of a unique string**: `Edit` with `replace_all=true` achieves the same result in 1 call (after a Read), with comparable effort.
- **Small edits (1–3 lines inside a method)**: Edit's substring matching is more efficient — less payload sent, no requirement to supply the full symbol body.
- **Inserting new code at a known location**: Both approaches require ~1–2 calls. Edit can target by surrounding text; Serena targets by name path. The effort is comparable.
**(c) Tasks outside Serena's scope (built-in only):**
- Non-code files (config, docs, notebooks, changelogs)
- Free-text search across the repo
- Shell commands, git operations, test execution
- File creation from scratch
**Verdict:** Serena's primary contribution is collapsing multi-file, semantically-aware operations (rename, move, reference-finding, type hierarchy) from multi-step manual processes into single atomic calls; for single-file text edits, it is comparable or slightly less efficient.
---
## 2. Added Value and Differences by Area
**1. Cross-file rename/move (Serena: strong positive)**
- Serena: 1 call updates definition + all imports + all usages across N files atomically.
- Built-in: 1 Grep + N×(Read+Edit) = 2N+1 calls, non-atomic.
- Frequency: Moderate (a few times per working session during refactoring work). Value per hit: High — saves 6–10+ calls and eliminates the risk of partial updates.
**2. Structural overview / targeted symbol reading (Serena: moderate positive)**
- `get_symbols_overview(depth=1)`: 1 call returns all classes, methods, and attributes in structured JSON, via the IDE's parser — language-agnostic and always correct. Built-in equivalent: `Grep` for a language-specific heuristic pattern like `^(class |def )`. This works for simple Python files but is inherently fragile: the pattern must be hand-tuned per language, misses decorated or multi-line declarations, and can match inside strings or comments. For non-Python languages, an entirely different regex would be needed.
- `find_symbol(include_body=True)`: Retrieves a specific method body by name path without reading surrounding code. Built-in: requires knowing the line number (via Grep) then Read with offset — 2 calls.
- Frequency: High (many times per session). Value per hit: Low-to-moderate — saves ~1 call and some context window tokens per navigation. The reliability advantage over heuristic Grep patterns is consistent across all languages.
**3. Reference finding with structural context (Serena: moderate positive)**
- `find_referencing_symbols`: Returns which *symbols* reference a target, with file grouping and usage type (import, call, type annotation). Built-in `Grep` returns all text matches including docs, comments, and string literals — same recall but lower precision.
- Frequency: Moderate. Value per hit: Moderate — precision matters when planning refactors of widely-used symbols.
**4. Type hierarchy (Serena: positive, niche)**
- 1 call returns full super/sub-type chains transitively. Built-in: requires iterative Grep for `class X(Y)` patterns, manual transitive closure.
- Frequency: Low (occasional during architecture exploration). Value per hit: High per occurrence — several calls saved.
**5. Stable addressing across edits (Serena: moderate positive)**
- Serena addresses symbols by name path (`CollectStats/refresh_return_stats`), which is invariant under edits elsewhere in the file. The built-in workflow is fundamentally line-number-mediated: Grep returns line numbers, Read takes line offsets — and both go stale after any edit to the file. Edit's text matching (`old_string`) is more resilient than line numbers, but it sits at the end of a chain that starts with position-based lookups. After an edit shifts lines, previously noted Grep results and Read offsets are invalid and must be re-acquired.
- This means a multi-edit session with built-ins requires re-Grep or re-Read between edits to re-establish positions, while Serena's name paths remain valid throughout. In Task 17 (three chained edits), Serena needed 3 calls total; the built-in path needed 4 (1 Read + 3 Edits) only because the text anchors happened to be unique — but had any Edit failed uniqueness, a re-Read would have been required, pushing the count to 5–7.
- Frequency: High (any session with multiple edits to the same file). Value per hit: Moderate — saves 1–2 re-read calls per edit chain, and eliminates the class of errors where stale line numbers cause edits to land in the wrong place.
**6. Single-file small/medium edits (Serena: neutral to slight negative)**
- For a 1-line change inside a 13-line method, Serena's `replace_symbol_body` requires sending the entire 13-line body. Edit sends just the changed line. The prerequisite cost differs: Serena needs `find_symbol(include_body=True)` to get the body; Edit needs `Read` of the relevant lines. Both are 1 prerequisite call + 1 edit call.
- Frequency: Very high. Value per hit: Slightly negative for Serena — more payload for small changes.
**Verdict:** Serena's strongest contributions are in cross-file refactoring (high value, moderate frequency) and structured navigation (moderate value, high frequency); for small text edits, built-ins are slightly more efficient.
---
## 3. Detailed Evidence, Grouped by Capability
### 3.1 Structural Overview (Task 2)
| Metric | Serena `get_symbols_overview(depth=1)` | Built-in `Grep` for class/def |
|---|---|---|
| Calls | 1 | 1 |
| Output structure | JSON tree: 14 classes with nested methods + attributes | Flat list: 54 `class`/`def` lines with line numbers |
| Nesting visible? | Yes — methods grouped under classes | No — indentation implies nesting but no grouping |
| Attributes visible? | Yes | No |
| Output size | ~2.5KB structured JSON | ~3KB flat text |
| Correctness | Always correct — uses IDE parser, language-agnostic | Heuristic — the regex `^(class \|def \| def )` is Python-specific and can miss decorated methods, multi-line signatures, or match inside strings/comments |
| Language portability | Works unchanged for any language the IDE supports | Requires a new hand-tuned regex per language |
| Next step | `find_symbol("ClassName/method", include_body=True)` — 1 call | `Read(file, offset=line, limit=N)` — 1 call |
Both paths need 1 call for overview + 1 call for drill-down. Serena's output is more structured and reliably correct; Grep's is simpler but fragile. The Grep pattern used here happened to work well for this Python codebase, but would need to be rewritten for Java (`class|interface|enum`, brace-delimited), Rust (`fn|struct|impl|trait`), TypeScript (`class|function|interface`), etc. — and even within Python, it misses cases like `@overload`-decorated methods or methods whose `def` line is preceded by a long decorator stack that pushes `def` to a non-standard indentation.
**Verdict:** Serena provides a reliably correct structural overview across languages; Grep-based overviews are heuristic approximations that work in simple cases but degrade for complex declarations or non-Python codebases.
### 3.2 Targeted Method Retrieval (Task 3)
| Metric | Serena | Built-in |
|---|---|---|
| Calls | 1 (`find_symbol` with `include_body=True`) | 2 (Grep to find line + Read with offset) |
| Prerequisite | Know the name path | Know the method name |
| Output | Method body only (no surrounding code) | Surrounding code included (must estimate limit) |
For `Collector/_collect` (330 lines): Serena returned exactly 330 lines. Built-in Read returned 332 lines (including the next method's signature).
**Verdict:** Serena saves 1 call and returns precisely the requested body; built-in is adequate but requires line-number discovery.
### 3.3 Cross-File References (Task 4)
| Metric | Serena `find_referencing_symbols` | Built-in `Grep` |
|---|---|---|
| Calls | 1 | 1 |
| Files found | 63 (code files only, structured by symbol type) | 83 (includes docs, notebooks, changelogs, README) |
| Output structure | Grouped by file, categorized (import, call, type annotation) | Flat file list |
| Precision | Code-only references | All textual mentions |
Grep finds 83 files including `README.md`, `CHANGELOG.md`, and `.ipynb` notebooks. Serena finds 63 code files. For the question "who uses this in code?", Serena's result is directly usable. For "where is this mentioned anywhere?", Grep is the right tool.
**Verdict:** Serena provides higher-precision code-usage results; Grep provides broader text-level coverage. Different tools for different questions.
### 3.4 Type Hierarchy (Task 5)
| Metric | Serena `type_hierarchy` | Built-in |
|---|---|---|
| Calls | 1 | 2+ (Grep for `class X(BaseCollector` + Grep for `class Collector(` + potential transitive search) |
| Result | `BaseCollector → ABC → object` (supers), `BaseCollector → Collector → AsyncCollector` (subs), with file locations | `class Collector(BaseCollector[TCollectStats], ...)` at line 551 — requires further reads for AsyncCollector's superclass |
**Verdict:** Serena produces the full transitive hierarchy in 1 call; built-in requires iterative searches.
### 3.5 External Dependency Lookup (Task 6)
| Metric | Serena | Built-in |
|---|---|---|
| Can retrieve? | Yes — `find_declaration` + `search_deps=True` | Requires environment discovery: `python -c "import X; print(inspect.getfile(X))"` then `Read` |
| Infrastructure | IDE index (pre-built) | Working Python environment, correct venv activated |
| Result | Symbol location + documentation | Full source file (if env is set up) |
In this session, the Python environment wasn't directly accessible from bash, so built-in lookup would have required additional setup. Serena retrieved `torch.distributions.Distribution`'s location and docstring via the IDE index.
**Verdict:** Serena provides dependency introspection without environment setup; built-in requires a working interpreter.
### 3.6 Small Edit — 1 Line Change (Task 7a)
| Metric | Serena `replace_symbol_body` | Built-in `Edit` |
|---|---|---|
| Prerequisite calls | 1 (`find_symbol` with `include_body=True`) — already done in Task 3 | 1 (`Read` of ~15 lines) |
| Edit call | 1 (send full 13-line body) | 1 (send 1-line old + 1-line new) |
| Payload sent (edit call) | ~550 chars (full body) | ~120 chars (changed line only) |
| Total calls | 1–2 | 2 |
**Verdict:** For small changes, Edit sends ~4.5× less payload in the edit call. Total call count is the same.
### 3.7 Medium Rewrite — ~19 Lines (Task 7b)
| Metric | Serena | Built-in |
|---|---|---|
| Prerequisite | 1 `find_symbol` (already done) | 1 `Read` (~22 lines) |
| Edit payload | ~550 chars (full body) | ~1000 chars (old 19 lines + new 20 lines) |
| Total calls | 1–2 | 2 |
At this scale, Serena's payload is actually *smaller* because Edit must send both old and new text, while Serena sends only the new body. The crossover point is approximately where the changed region exceeds half the method body.
**Verdict:** At medium scale, payloads converge; Serena's is slightly smaller due to only sending the replacement.
### 3.8 Large Rewrite — 55 Lines (Task 7c)
| Metric | Serena | Built-in |
|---|---|---|
| Prerequisite | 1 `find_symbol` (already done) | 1 `Read` (~67 lines) |
| Edit payload | ~2200 chars (new body) | ~4400 chars (old 63 lines + new 62 lines) |
| Total calls | 1–2 | 2 |
**Verdict:** For full-method rewrites, Serena sends ~50% less payload because it only sends the replacement, not the original.
### 3.9 Cross-File Rename (Task 10)
| Metric | Serena `rename` | Built-in chain |
|---|---|---|
| Calls | 1 | 1 Grep + 4 Read + 4 Edit = 9 |
| Files affected | 4 (automatically discovered) | 4 (manually discovered via Grep) |
| Import handling | Automatic | Manual — must identify and rewrite each import statement |
| Atomicity | Atomic — all-or-nothing | Sequential — intermediate states are inconsistent |
**Verdict:** Serena reduces a 9-call chain to 1 call for cross-file rename, with atomicity.
### 3.10 Move Symbol Across Modules (Task 11)
| Metric | Serena `move` | Built-in equivalent |
|---|---|---|
| Calls | 1 | ~8–12 (Read source, Edit source to remove, Edit target to add, Grep for imports, Read+Edit each import site) |
| Import updates | Automatic | Manual — must rewrite `from X import Y` → `from Z import Y` in each file |
| Dependency imports | Automatically adds needed imports to target file | Must manually inspect what the moved function imports and replicate |
Moving `get_stddev_from_dist` from `collector.py` to `stats.py`: Serena updated 3 files (source, target, test) in 1 call, including adding necessary imports to the target module.
**Verdict:** Move is Serena's highest-value single operation — it handles import graph updates that would be error-prone manually.
### 3.11 Move File (Task 12a)
| Metric | Serena `move` | Built-in equivalent |
|---|---|---|
| Calls | 1 | 1 `git mv` + 1 Grep + N×(Read+Edit) for import updates = ~12 |
| Files updated | 5 import updates automatic | Must manually discover and rewrite |
**Verdict:** Same pattern as symbol move — Serena collapses an N-step process.
### 3.12 Safe Delete (Task 12b)
Serena's `safe_delete` in safe mode (default):
- Reports all usages before deleting — acts as a guard.
- For unused symbols, deletes cleanly in 1 call.
- For used symbols, refuses and lists usages.
Built-in equivalent: Grep to check usages → if none found, Read + Edit to delete. 2–3 calls.
**Verdict:** Safe delete adds a safety check that the built-in path must implement manually.
### 3.13 Inline (Task 13)
Serena's inline tool did not work for any Python function tested. The JetBrains backend does not support Python function inlining (this is a language-level limitation of the IDE's refactoring engine, not a Serena bug).
**Verdict:** No candidate inlinable in this codebase; inline capability appears unavailable for Python.
### 3.14 Scope Precision (Task 14)
Three methods named `reset_env` exist in `collector.py` (in `BaseCollector`, `Collector`, `AsyncCollector`).
- Serena: `find_symbol("Collector/reset_env")` returns exactly that override's body.
- Grep: Returns 3 line numbers. Determining which belongs to which class requires reading surrounding context.
**Verdict:** Serena's name-path addressing eliminates class-level disambiguation that Grep requires.
### 3.15 Chained Edits (Task 17)
Three consecutive edits to methods in `CollectStats`:
- Serena: 3 `replace_symbol_body` calls, 0 intermediate reads. Name paths (`CollectStats/refresh_return_stats`, `CollectStats/refresh_len_stats`, `CollectStats/refresh_std_array_stats`) remained valid across all edits because they are structural identifiers, not positions.
- Built-in: 1 Read + 3 Edit calls = 4 calls in the best case. The Read was required upfront (Edit enforces "must read before editing"). The 3 Edits succeeded without re-reads only because the `old_string` anchors happened to remain unique after each edit. But this is fragile: Edit also enforces a "file has been modified since read" check when external tools modify the file, forcing a re-Read. More fundamentally, if I had needed to *find* these methods first (the typical case when you don't already know the line numbers), each Grep result from before the first edit would have been stale after it — line 232 is no longer line 232 after inserting 2 lines above it.
The core asymmetry: Serena's addressing is *structural* (survives edits by definition), while built-in addressing is *positional* (line numbers from Grep/Read go stale after any insertion or deletion). Edit's text matching partially mitigates this, but only for the final step — the discovery steps (Grep, Read with offset) remain position-dependent.
**Verdict:** Serena's name-path stability eliminates the re-read/re-grep cycle between edits; built-in tools require re-acquiring positions after each edit that shifts line numbers.
---
## 4. Token-Efficiency Analysis
### Payload by edit size
| Edit scale | Serena payload (edit call) | Edit payload (edit call) | Winner |
|---|---|---|---|
| 1-line change in 13-line method | ~550 chars (full body) | ~120 chars | Edit (~4.5×) |
| 19-line medium rewrite | ~550 chars | ~1000 chars | Serena (~1.8×) |
| 55-line full rewrite | ~2200 chars | ~4400 chars | Serena (~2×) |
| Cross-file rename (4 files) | ~100 chars | ~800 chars (4 Edit calls) | Serena (~8×) |
### Prerequisite reads
- Serena: `find_symbol(include_body=True)` returns the symbol body. If you've already navigated to it during exploration, no additional call needed.
- Built-in: `Read` with offset/limit. Always required before Edit (enforced by the tool).
### Stable vs ephemeral addressing
- Serena's name paths (`CollectStats/refresh_return_stats`) are stable identifiers — they survive any edit to surrounding code. No re-read or re-discovery needed between edits.
- The built-in workflow uses ephemeral, position-based addresses at every stage: Grep returns line numbers, Read takes line offsets, and both are invalidated by any insertion or deletion above the target. Edit's `old_string` matching is content-based and more resilient, but it depends on the upstream position-based steps to know *what* to match. After an edit shifts line numbers, the entire Grep→Read→Edit chain must be re-executed from the top.
- In a session with N edits to the same file, this costs up to N-1 additional Grep/Read round-trips with built-ins. With Serena, the cost is zero — the same name path works on the first and tenth edit.
**Verdict:** Serena is more token-efficient for medium-to-large edits and cross-file operations; Edit is more efficient for small, localized changes. Across multi-edit sessions, Serena's stable addressing avoids the re-read tax that compounds with each successive built-in edit.
---
## 5. Reliability & Correctness (Under Correct Use)
### Precision of matching
- Serena: Exact symbol resolution via name paths. `Collector/reset_env` unambiguously selects one method.
- Edit: Text matching. Unique strings match correctly; non-unique strings fail (and Edit reports the error).
### Scope disambiguation
- Serena distinguishes overrides, overloads (via indices), and nested classes by name path.
- Grep/Edit cannot distinguish methods with the same name in different classes without reading surrounding context.
### Atomicity
- Serena's cross-file operations (rename, move) are atomic — all files updated or none.
- Built-in multi-file edits are sequential — a failure mid-chain leaves an inconsistent state (recoverable via `git checkout`).
### Semantic queries vs text search
- `find_referencing_symbols` returns code-level references categorized by type. Grep returns all textual mentions.
- `type_hierarchy` returns transitive sub/supertypes. No built-in equivalent without iterative search.
### External dependency lookup
- Serena: Available via IDE index, no environment setup needed. Can retrieve symbol docs and location.
- Built-in: Requires working Python environment, correct venv, and manual file discovery. More powerful when available (full source), but higher setup cost.
**Verdict:** Serena provides stronger correctness guarantees for symbol-level operations (scope, atomicity, semantic precision); built-in tools are reliable for text-level operations with the caveat that scope disambiguation requires manual effort.
---
## 6. Workflow Effects Across a Session
### Compound advantages
- **Exploration → Edit without position re-acquisition**: Serena's overview tools produce name paths that are directly usable as edit targets. The built-in path produces line numbers (from Grep) that are consumed by Read — but after an edit, those line numbers are stale. In a typical explore-edit-explore-edit cycle, Serena's name paths remain valid throughout while built-in line numbers must be re-acquired after each edit. Over 3 explore+edit cycles, this saves ~3–6 intermediate Grep/Read calls.
- **Chained edits without re-reads**: Name-path stability means no re-reads between edits to the same file. For N edits, this saves up to N-1 Read calls. Edit's text matching partially avoids this (if old_strings stay unique), but the upstream discovery steps (Grep line numbers, Read offsets) still go stale.
- **Cross-file refactoring**: When a rename or move is part of a larger change, doing it atomically avoids the need to manually track "which files still need updating."
### Diminishing returns
- For purely exploratory sessions (reading code, no edits), Serena's advantage is modest — `Grep` and `Read` are nearly as fast for navigation, and Serena's structured output doesn't save many calls.
- For sessions dominated by small text edits (config changes, log message tweaks), Serena adds no value.
### Neutral findings
- Both toolsets require similar total calls for single-file work. The difference is ~1 call per operation.
- Output quality for code review (reading diffs, understanding changes) is identical — both require `git diff`.
**Verdict:** Serena's advantages compound across multi-step refactoring sessions where each operation feeds into the next; for read-heavy or small-edit sessions, the advantage is marginal.
---
## 7. Unique Capabilities (No Practical Built-In Equivalent)
1. **Atomic cross-file rename** — 1 call, all imports and usages updated, all-or-nothing. No built-in equivalent without scripting a multi-step chain. Frequency: moderate (refactoring sessions). Impact: high (saves 5–10 calls, eliminates partial-update risk).
2. **Atomic cross-file move** (symbol or file) with import rewriting — includes adding necessary imports to the target module. Frequency: low-to-moderate. Impact: very high per occurrence (saves 8–12 calls, handles import dependency graph).
3. **Type hierarchy traversal** — transitive supertypes and subtypes in 1 call. Built-in requires iterative Grep with manual transitive closure. Frequency: low. Impact: moderate (saves 3–5 calls).
4. **Safe delete with usage check** — reports all usages before deleting, with option to propagate. Built-in requires Grep + manual verification. Frequency: low. Impact: moderate (safety check is the value).
5. **External dependency symbol lookup** via IDE index — no Python environment needed. Frequency: moderate. Impact: moderate (avoids environment setup friction).
**Verdict:** Serena provides 3–5 capabilities with no practical built-in equivalent, concentrated in cross-file refactoring and semantic navigation.
---
## 8. Tasks Outside Serena's Scope (Built-In Only)
- **Non-code file operations**: Reading/editing config files, docs, changelogs, notebooks → `Read`/`Edit`/`Write`
- **Free-text search**: Finding log strings, URLs, magic constants → `Grep`
- **Shell operations**: Running tests, builds, git commands, package management → `Bash`
- **File creation**: New files from scratch → `Write`
- **Glob-based file discovery**: Finding files by pattern → `Glob`
- **Broad codebase search**: When you don't know what you're looking for → `Grep` with regex
Estimated share of daily work: These tasks constitute roughly 40–60% of a typical coding session (reading docs, running tests, searching for patterns, editing config). Serena's augmentation covers the remaining 40–60% where code-level semantic operations apply.
**Verdict:** Built-in tools handle roughly half of daily work that falls entirely outside Serena's scope; Serena augments the code-centric other half.
---
## 9. Practical Usage Rule
| Task type | Use |
|---|---|
| Cross-file rename, move, delete | Serena (unique capability) |
| Understanding class hierarchy or symbol relationships | Serena (1 call vs. iterative search) |
| Getting a structural overview of a large file | Serena (richer structure) or Grep (simpler, faster) |
| Reading a specific method body by name | Serena (direct) or Grep+Read (2 calls) |
| Small edit (1–5 lines inside a method) | Edit (less payload) |
| Full method rewrite | Serena `replace_symbol_body` (less payload, no re-read) |
| Single-file rename of a unique identifier | Edit with `replace_all` (equivalent) |
| Non-code files, config, docs | Built-in `Read`/`Edit` |
| Text search, pattern matching | Built-in `Grep` |
| Anything involving shell, git, tests | Built-in `Bash` |
| External dependency inspection | Serena (no env setup needed) |
**Verdict:** Use Serena for cross-file refactoring and semantic navigation; use built-ins for text-level edits, non-code files, and shell operations; for single-file code edits, choose based on edit size (Edit for small, Serena for full-body rewrites).
docs/04-evaluation/030_results/020_codex_on_jbplugin.md
+665
View File
@@ -0,0 +1,665 @@
:::{admonition} Evaluation Result
:class: note
**Generated by:** GPT-5.4 (high) in Codex
**Codebase:** Serena JetBrains Plugin (Java)
:::
# Codex (GPT-5.4)
> **One-line summary:** As a coding agent, I would ask my owner to add Serena because it turns fragile text-and-line-number work into precise symbol-aware navigation and
refactoring, which makes real code changes feel faster, safer, and far less blind.
This report compares Serena's JetBrains-backed semantic coding tools with built-in file, shell, search, and patch tools in this repository. The comparison assumes competent use of both toolsets: built-ins are used for text, file, shell, config, and small patch work; Serena is used where symbol identity, language semantics, or IDE refactoring semantics apply.
Method: I explored code first, avoided repo documentation and prior notes, ran real edits/refactors, and reverted after each experiment. After each edit/refactor experiment I checked `git status --short` and returned the tree to clean before moving on. Measurements are approximate, but call counts, diff sizes, and result shapes are from observed runs.
## 1. Headline: what Serena changes
Serena adds a semantic layer over the codebase. Its concrete delta is the ability to address and transform code by symbols: name paths, overload indexes, reference graphs, type hierarchies, external declarations, and JetBrains refactoring operations.
Tasks where Serena adds capability:
- Code-reference search: `find_referencing_symbols` for `Symbol/findReferences` returned 6 precise code usages grouped by containing symbols. `rg findReferences` also returned the declaration, the `/findReferences` route string, and comments.
- Type hierarchy: one `type_hierarchy` call for `PostRequestHandler` returned 18 direct endpoint subclasses, a transitive `TypeHierarchyHandler -> GetSupertypesHandler/GetSubtypesHandler` branch, and the external `Object` supertype.
- External dependency lookup: `find_declaration` resolved `ReferencesSearch.search(anchorElement)` to `<ext:ReferencesSearch.class|466808a0>`, and `find_symbol` returned the selected overload body. Built-in repo/cache grep found the import and call but not the dependency definition.
- Cross-file refactors: semantic rename, symbol move, file move, safe delete, and inline executed real IDE refactors and updated files/usages/imports.
- Stable addressing: a chained edit used `SymbolFinder/findFilesByName`, `SymbolFinder/getProject`, and `SymbolFinder/qualNameMatchesName` after earlier edits shifted line numbers.
Tasks where Serena applies but offered little or no improvement:
- Tiny intra-method edits: a 1-line error-message change was smaller with a built-in patch; Serena body replacement required sending the whole method.
- Simple one-file private rename: manual patch and semantic rename produced the same 2-line diff for `qualNameMatchesName -> qualifiedNameMatchesName`.
- Method insertion at a known spot: both workflows produced the same 4-line diff; Serena's benefit was target stability, not smaller payload.
- Whole-method rewrite: Serena targets the method boundary cleanly, but the full replacement body still has to be sent.
Tasks outside Serena's scope:
- Reading config/non-code files such as `plugin.xml` and `build.gradle.kts`.
- Free-text searches for URLs, endpoint strings, settings values, comments, and resources.
- File inventory, line counts, shell commands, build/test execution, Git operations, and arbitrary text edits.
Value-weighted summary:
| Rank | Difference | Frequency | Value per hit | Delta |
|---|---:|---:|---:|---|
| 1 | Cross-file semantic refactors | Medium-high | High | Often avoids 5-20 manual edit/search/verify steps and reduces partial-update risk. |
| 2 | Code usages vs text mentions | High | Medium-high | Removes noisy grep triage for "who uses this symbol?" |
| 3 | Symbol overview/body retrieval | High | Medium | Avoids large reads and stale line targeting. |
| 4 | Type hierarchy/implementations | Medium | High | Hard to reproduce correctly with text search, especially transitively. |
| 5 | Stable addressing across edit chains | Medium | Medium | Reduces refresh work after line shifts. |
| 6 | External dependency lookup | Low-medium | High | Needs IDE index; built-ins need source/decompiler infrastructure. |
| 7 | Small local edits | High | Neutral/negative for Serena | Built-in patch/edit usually sends less. |
**Verdict:** Serena materially changes symbol-centric exploration and refactoring; it does not replace built-ins for ordinary text, file, shell, config, or tiny local edits.
## 2. Added value and differences by area
- Cross-file semantic refactoring changes both workflow and correctness. Frequency: medium-high. Value per hit: high. The observed class rename updated a file/class plus imports/usages; the nested-record move changed 4 existing files and created 1 new file; the file move updated package/imports across 3 paths. Built-ins can reproduce the result, but only through search, file moves, patches, import cleanup, and verification.
- Semantic search separates code usage from text mention. Frequency: high. Value per hit: medium-high. Serena returned only code references for `Symbol/findReferences`; `rg` returned code, route strings, comments, and the declaration. Built-ins remain better for "mentioned anywhere."
- Symbol overview/body retrieval cuts exploration payload. Frequency: high. Value per hit: medium. `get_symbols_overview` on `Symbol.java` returned fields, methods, nested classes, and overload indexes without reading the 1,000+ line file. Regex produced line-oriented candidates and still required manual scope interpretation.
- Type/dependency queries add capabilities not practically present in built-ins. Frequency: low-medium. Value per hit: high. Serena returned a transitive type graph and an external IntelliJ API method body; text tools needed iterative searches or extra source/decompiler setup.
- Small localized edits are not improved by symbol-body replacement. Frequency: high. Value per hit: neutral or negative for Serena. A one-line patch was smaller than replacing an 11-line method body.
- Some JetBrains refactors staged added/deleted files. Frequency: limited to class/file move/delete style operations. Value per hit: neutral operational tradeoff. Semantic value remains, but verification/revert must check the index as well as unstaged files.
**Verdict:** Serena's highest-value differences are symbol identity, reference graphs, and IDE refactor execution; built-ins remain superior for simple text-local work.
## 3. Detailed evidence, grouped by capability
### 3.1 Repository structure and entry points
Attempted: identify top-level layout, packages, and entry points.
Built-in call chain:
1. `git status --short` -> clean.
2. `rg --files -g '!*.md' -g '!docs/**' -g '!CLAUDE.md'` -> code/config/resource inventory.
3. `Get-ChildItem -Force` -> top-level directories.
4. `Get-Content src/main/resources/META-INF/plugin.xml` -> plugin entry points.
5. `Get-Content build.gradle.kts -TotalCount 80` -> Gradle/IntelliJ platform setup.
Serena call chain: not applicable for repository inventory and non-code config.
Observed structure:
- `service`: backend service and request handling.
- `service/endpoint`: endpoint handlers for symbol search, references, formatting, rename, move, safe delete, inline, inspections, and completions.
- `symbol`: symbol model, symbol lookup, hierarchy, path matching, and move processors.
- `util`: IDE/project/editor/UI helpers.
- `ui`: tool window content.
- `plugin.xml`: registers `PluginStartupActivity`, settings service/configurable, and a dummy test action.
Payloads: built-ins used 5 small shell/read calls and returned several KB of code/config context; Serena had no relevant semantic operation.
**Verdict:** Repository layout and non-code entry-point discovery are built-in work; Serena starts adding value once the target is code symbols.
### 3.2 Large file structural overview
Attempted: compare structural overview on `src/main/java/de/oraios/serena/symbol/Symbol.java`.
Serena call chain:
1. `get_symbols_overview(relative_path=Symbol.java, depth=1)`
2. Next step: `find_symbol(Symbol/findReferences, include_body=true)`
Serena output: one top-level `Symbol` class; fields; nested `ChildrenCollector` and `DocumentationResolver`; methods including overload-indexed names such as `getLocationString[0]`, `getLocationString[1]`, `move[0]`, `move[1]`, `getDocumentation[0]`, and `getDocumentation[1]`.
Built-in call chain:
1. `rg` for method-like declaration lines in `Symbol.java`.
2. `rg` for class/interface/enum lines in `Symbol.java`.
3. Next step: line-slice read around the selected method.
Built-in output: about 70 line-oriented method/class matches, including nested symbols, with no durable symbol identity beyond manual signature inspection.
Payloads: Serena used 1 compact overview call plus a targeted follow-up. Built-ins used 2 regex searches plus a later line-range read.
**Verdict:** Serena's overview is more actionable because its output feeds directly into stable symbol-addressed calls; regex outlines are useful but line-based.
### 3.3 Targeted method body retrieval
Attempted: retrieve `Symbol/findReferences` without reading surrounding file content.
Serena call chain: `find_symbol(relative_path=Symbol.java, name_path_pattern=Symbol/findReferences, include_body=true)`.
Serena output: only the `public ArrayList<SymbolReference> findReferences()` body.
Built-in call chain:
1. `rg "findReferences\\(" src/main/java -n -C 2`
2. `Get-Content Symbol.java` line slice around the declaration.
Built-in output: search context across multiple files, then the selected method slice.
Payloads: Serena input was one path/name-path request and returned roughly 25 method lines. Built-ins required search output plus a separate slice read.
**Verdict:** If the symbol is known, Serena retrieves the body in one precise call; built-ins need search plus line-targeted reading.
### 3.4 References: code usages vs mentions anywhere
Attempted: find all references to `Symbol/findReferences`.
Serena call chain: `find_referencing_symbols(relative_path=Symbol.java, name_path=Symbol/findReferences)`.
Serena output:
- `Symbol/formatReferenceLocations/references`
- `Symbol/inline/refCountBefore`
- `Symbol/verifyInlineResult/refCountAfter`
- `SymbolDTO/Builder/buildDTO/references`
- `RunInspectionsOnSymbolsHandler/handleRequest/refs`
- `FindReferencesHandler/buildResponse/references`
Built-in call chain: `rg "findReferences" src/main/java -n -C 1`.
Built-in output: same call sites plus the declaration, `/findReferences` route registration, and explanatory comments.
Payloads: Serena returned a compact grouped usage list of about 500-700 characters. `rg` returned broader line context of about 1-2 KB and mixed code usages with text mentions.
**Verdict:** Serena has higher precision for "who uses this symbol in code"; built-ins have higher recall for "where is this text mentioned anywhere."
### 3.5 Type hierarchy
Attempted: list supertypes and subtypes transitively for `PostRequestHandler`.
Serena call chain: `type_hierarchy(relative_path=PostRequestHandler.java, name_path=PostRequestHandler, hierarchy_type=both, depth=0)`.
Serena output: external `Object` supertype; 18 direct endpoint subclasses; transitive nested branch `TypeHierarchyHandler` containing `GetSupertypesHandler` and `GetSubtypesHandler`.
Built-in call chain:
1. `rg "extends PostRequestHandler|extends TypeHierarchyHandler|class PostRequestHandler" src/main/java -n`
2. Manually follow any discovered intermediate types.
3. Repeat searches if deeper hierarchy exists.
Payloads: Serena used 1 hierarchy query and returned structured JSON. Built-ins used pattern search and manual transitive grouping.
**Verdict:** Serena turns hierarchy discovery into a semantic graph query; text search requires iterative pattern expansion and manual reasoning.
### 3.6 External dependency symbol lookup
Attempted: retrieve the IntelliJ dependency symbol behind `ReferencesSearch.search(anchorElement)`.
Serena call chain:
1. `find_declaration(relative_path=Symbol.java, regex="ReferencesSearch\\.(search)\\(anchorElement\\)")`
2. `find_symbol(relative_path=<ext:ReferencesSearch.class|466808a0>, name_path_pattern=ReferencesSearch/search[0], include_body=true, search_deps=true)`
Serena output: declaration `ReferencesSearch/search[0]` in an external class path and the overload body:
```java
public static @NotNull Query<PsiReference> search(@NotNull PsiElement element) {
return search(element, GlobalSearchScope.allScope(PsiUtilCore.getProjectInReadAction(element)), false);
}
```
Built-in call chain:
1. `rg "ReferencesSearch" src/main/java -n` -> import and local call.
2. `rg` over repo, `.intellijPlatform`, and Gradle caches for `class ReferencesSearch` or matching method signatures -> no useful source hit.
3. `Get-ChildItem` over Gradle/IntelliJ caches -> jar candidates but no direct definition.
Infrastructure difference: Serena depends on the JetBrains IDE index; built-ins need source jars, decompiler tooling, or classpath-specific jar inspection.
**Verdict:** External dependency lookup is a genuine Serena capability when IDE indexes are available; ordinary built-ins do not provide it.
### 3.7 Single-file edits across edit sizes
Small tweak attempted: change `"No symbol found for "` to `"No matching symbol found for "` in `SymbolFinder/findSymbolByNamePath`.
Built-in call chain:
1. Use existing search/body context.
2. `apply_patch` one-line replacement.
3. `git diff`, `git status --short`.
4. Revert and clean check.
Serena call chain:
1. `replace_symbol_body(SymbolFinder/findSymbolByNamePath, full method body with changed string)`.
2. `git diff`, `git status --short`.
3. Revert and clean check.
Observed diff: both produced the same 1-line change. Payload difference: built-in edit input was a tiny hunk; Serena input was the full 11-line method body.
Medium rewrite attempted: rewrite `SymbolFinder/findFilesByName` from list accumulation to stream collection.
Built-in call chain: `apply_patch` over the method body, then diff/status/revert/status.
Serena call chain: `replace_symbol_body(SymbolFinder/findFilesByName, new method body)`, then diff/status/revert/status.
Observed diff: both produced `10 +++-------`, 3 insertions and 7 deletions. Payload difference was small: a patch hunk versus an 8-line replacement method.
Large/whole-body rewrite attempted: rewrite `InspectionRunner/collectResults` while preserving signature and behavior shape.
Built-in call chain: `apply_patch` against the method body, then diff/status/revert/status.
Serena call chain: `replace_symbol_body(InspectionRunner/collectResults, full symbol text including Javadoc and replacement body)`, then diff/status/revert/status.
Observed diff: both produced `38 +++++++++++-----------`, 19 insertions and 19 deletions. Payload difference: built-in required a large contextual hunk; Serena required the full symbol text, about 70 lines including Javadoc/signature/body.
**Verdict:** Serena is not automatically more token-efficient for edits; its edit advantage is stable symbol targeting, while built-ins are smaller for tiny local hunks.
### 3.8 Insert method at structural location
Attempted: insert `private Logger getLogger()` immediately after `SymbolFinder/getProject`.
Built-in call chain:
1. Search/read surrounding area.
2. `apply_patch` with nearby context.
3. Diff/status/revert/status.
Serena call chain:
1. `insert_after_symbol(relative_path=SymbolFinder.java, name_path=SymbolFinder/getProject, body=...)`
2. Diff/status/revert/status.
Observed diff: both produced the same 4-line insertion.
Payloads: built-ins sent inserted body plus surrounding text context; Serena sent inserted body plus symbol target.
**Verdict:** Structural insertion is a modest Serena improvement: same resulting diff, less dependence on line/context stability.
### 3.9 Rename: private helper in one file
Attempted: rename `SymbolFinder/qualNameMatchesName` to `qualifiedNameMatchesName`.
Built-in call chain:
1. `rg "qualNameMatchesName" SymbolFinder.java -n -C 2`
2. `apply_patch` declaration and one call site.
3. Diff/status/revert/status.
Serena call chain:
1. `find_referencing_symbols(SymbolFinder/qualNameMatchesName)` -> one referencing method.
2. `rename(name_path=SymbolFinder/qualNameMatchesName, new_name=qualifiedNameMatchesName, rename_in_comments=true, rename_in_text_occurrences=true)`.
3. Diff/status/revert/status.
Observed result: Serena returned `"Success"`. Both produced the same 2-line diff.
**Verdict:** Semantic rename has little advantage for a one-file helper with one call site, but it scales better than text edits as references spread or names become ambiguous.
### 3.10 Rename: cross-file class including imports
Attempted: rename `MoveOperationFailedException` to `MoveFailedException`.
Serena call chain:
1. `rename(relative_path=MoveOperationFailedException.java, name_path=MoveOperationFailedException, new_name=MoveFailedException, rename_in_comments=true, rename_in_text_occurrences=true)`.
2. `git diff --stat`, `git diff`, `git diff --cached --stat`, `git status --short`.
3. Restore staged rename and modified files; clean check.
Serena result: `"Success"`.
Observed edits:
- Added/staged `MoveFailedException.java`.
- Deleted/staged `MoveOperationFailedException.java`.
- Updated import, Javadoc comment, and thrown class in `MoveProcessor.java`.
Built-in equivalent chain:
1. `rg "MoveOperationFailedException" src/main/java -n`.
2. Rename/create/delete file.
3. Patch class name and constructor.
4. Patch imports and usages.
5. Patch comments/text occurrences if desired.
6. Verify no unintended old references remain.
7. Compile/test for a permanent change.
**Verdict:** Cross-file class renames are high-value Serena cases because file/class coupling, imports, usages, and optional comments are handled as one semantic refactor.
### 3.11 Move symbol to another file context
Attempted: move nested record `InspectionRunner/PendingFix` to a top-level file in the same package.
Serena call chain:
1. `move(relative_path=InspectionRunner.java, name_path=InspectionRunner/PendingFix, target_relative_path=src/main/java/de/oraios/serena/service/endpoint)`.
2. Diff/status/cached-diff inspection.
3. Restore staged new file and modified files; clean check.
Serena result:
```json
{
"source_relative_path": "src/main/java/de/oraios/serena/service/endpoint/InspectionRunner.java",
"target_relative_path": "src/main/java/de/oraios/serena/service/endpoint",
"moved_symbol": {"name_path": "PendingFix", "type": "CLASS"}
}
```
Observed edits:
- Created/staged `PendingFix.java` with package and imports.
- Removed nested record from `InspectionRunner.java`.
- Updated 3 external references from `InspectionRunner.PendingFix` to `PendingFix`.
- Removed unused imports in 2 files.
Built-in equivalent chain: read nested record/import needs, add file, delete nested symbol, search usages, patch qualified usages, patch internal references if needed, remove imports, verify, compile/test.
**Verdict:** Symbol move is a substantial Serena addition because the difficult part is reference/import repair, not copying text.
### 3.12 Move file/package location
Attempted: move `PluginUtil.java` from `util` to `service`.
Serena call chain:
1. `move(relative_path=src/main/java/de/oraios/serena/util/PluginUtil.java, target_relative_path=src/main/java/de/oraios/serena/service)`.
2. Diff/status/cached-diff inspection.
3. Restore staged rename and modified imports; clean check.
Serena result:
```json
{
"source_relative_path": "src/main/java/de/oraios/serena/util/PluginUtil.java",
"target_relative_path": "src/main/java/de/oraios/serena/service"
}
```
Observed edits:
- Staged rename `util/PluginUtil.java -> service/PluginUtil.java`.
- Changed package declaration to `de.oraios.serena.service`.
- Updated import in `PluginStartupActivity`.
- Removed now-unneeded same-package import in `SerenaBackendService`.
Built-in equivalent chain: move file, patch package declaration, search `PluginUtil`, patch imports/usages, remove redundant imports, verify no old package import remains, compile/test.
**Verdict:** File moves are high-value when packages/imports must change; built-ins can do them manually but require dependency repair steps.
### 3.13 Safe delete and propagated delete
Attempted: delete unused `DebugUtil`.
Serena call chain:
1. `safe_delete(relative_path=DebugUtil.java, name_path=DebugUtil, delete_even_if_used=false, propagate=false)`.
2. `git diff --cached --stat`, `git status --short`.
3. Restore staged deletion; clean check.
Serena result:
```json
{
"deleted_symbol": "DebugUtil",
"relative_path": "src/main/java/de/oraios/serena/util/DebugUtil.java",
"affected_references": [],
"message": "Symbol deleted successfully with no affected references."
}
```
Built-in equivalent chain: `rg "DebugUtil"`, inspect declaration vs usages, delete file/symbol, verify no remaining usages.
Propagated deletion: no suitable correct-use candidate was found where JetBrains safe-delete propagation would meaningfully delete a used symbol and propagate deletion through call sites. For ordinary used-method deletion, correct safe delete reports usages or, if forced, deletes the symbol and reports affected references; that is not the same as arbitrary call-site deletion.
**Verdict:** Safe delete adds an integrated usage check plus deletion operation; propagated deletion was not evaluated because this codebase did not provide a suitable candidate.
### 3.14 Inline
Attempted: inline `ProjectUtil/getAbsolutePath`, a single-expression helper.
Serena call chain:
1. `find_symbol(ProjectUtil/getAbsolutePath, include_body=true)`.
2. `find_referencing_symbols(ProjectUtil/getAbsolutePath)` -> 2 references.
3. `inline_symbol(relative_path=ProjectUtil.java, name_path=ProjectUtil/getAbsolutePath, keep_definition=false)`.
4. Diff/status/revert/status.
Serena result: `{"status": "SUCCESS"}`.
Observed edits:
- Removed `getAbsolutePath`.
- Replaced internal call in `ProjectUtil/getVirtualFile`.
- Replaced external call in `RefreshFileHandler`.
Built-in equivalent chain: read helper body, `rg getAbsolutePath`, patch each call site with substituted expression, delete helper, verify no stale references, compile/test.
**Verdict:** Inline is a strong Serena refactor for legally substitutable helpers; built-ins can reproduce it but must manually adapt each substitution.
### 3.15 Scope precision and overload targeting
Attempted: distinguish overloaded `Symbol/getLocationString` methods.
Serena call chain:
1. `find_symbol(Symbol/getLocationString[0], include_body=true)`.
2. `find_symbol(Symbol/getLocationString[1], include_body=true)`.
Serena output:
- `[0]`: `public String getLocationString() { return getLocationString(element); }`
- `[1]`: `private static String getLocationString(PsiElement element) { ... }`
Built-in call chain: `rg "getLocationString" Symbol.java -n -C 1`, then manual signature inspection.
**Verdict:** Serena's overload-indexed name paths provide precise symbol addressing that text search does not.
### 3.16 Atomicity and success signals
Observed Serena success signals:
- `replace_symbol_body` -> `OK`
- `insert_after_symbol` -> `OK`
- `rename` -> `"Success"`
- `move` -> JSON with source/target/moved symbol or paths
- `safe_delete` -> JSON with deleted symbol, affected references, and message
- `inline_symbol` -> `{"status": "SUCCESS"}`
Built-in success signals:
- `apply_patch` -> "Success. Updated the following files"
- Shell/Git commands -> exit codes and textual output
- Cross-file consistency -> user-driven diff/search/build verification
Atomicity comparison: Serena refactors executed as single IDE operations. Built-in equivalents are chains of searches, file operations, and patches; a competent user can make them correct, but intermediate states can be partial.
Tradeoff: class/file move/delete refactors staged added/deleted files in the Git index, so clean verification must check both staged and unstaged state.
**Verdict:** Serena gives clearer semantic success signals and more atomic cross-file edits; built-ins require manual consistency management.
### 3.17 Workflow effects across multiple edits
Attempted: chain 3 edits in `SymbolFinder.java` without refreshing between them.
Serena call chain:
1. `replace_symbol_body(SymbolFinder/findFilesByName, ...)`.
2. `insert_after_symbol(SymbolFinder/getProject, ...)`.
3. `rename(SymbolFinder/qualNameMatchesName, ...)`.
4. Diff/status/revert/status.
Observed result: all edits applied after earlier edits shifted file positions. Final diff touched 18 lines with net 9 insertions and 9 deletions.
Built-in equivalent chain: initial read/search, apply first patch, rely on robust patch context or refresh line numbers, apply second patch, refresh or search again, apply rename patch, verify.
**Verdict:** Serena's name paths remain useful after line shifts, so its advantage compounds in multi-edit sessions.
### 3.18 Non-code reads and free-text search
Attempted: read config and search free text.
Built-in call chain:
1. `Get-Content plugin.xml`.
2. `Get-Content build.gradle.kts -TotalCount 80`.
3. `rg "http://|https://|localhost|127\\.0\\.0\\.1|8080|24282|SERENA|serena" src/main/java src/main/resources -n`.
Serena call chain: not applicable.
Observed result: built-ins exposed plugin registration, Gradle platform setup, URLs, IDs, settings defaults, and resource/text mentions.
**Verdict:** Built-ins are natural and sufficient for config reading and free-text search; this is outside Serena's semantic-code scope.
## 4. Token-efficiency analysis
Payload differences across edit sizes:
| Task | Built-in payload | Serena payload | More efficient |
|---|---|---|---|
| 1-line string tweak | Tiny patch hunk, 1 changed line | Full 11-line method body | Built-ins |
| Medium method rewrite | 10-line diff hunk | Full 8-line replacement method | Roughly equal |
| Large body rewrite | Large contextual patch, 38-line diff | Full symbol text, about 70 lines | Roughly equal; Serena has better target |
| Insert method | Inserted body plus surrounding context | Inserted body plus name path | Serena slightly |
| Private one-file rename | `rg` plus 2-line patch | optional refs query plus rename command | Roughly equal |
| Cross-file rename/move/inline | Multiple searches, file operations, patches, verifications | One semantic refactor plus diff/status | Serena |
Forced reads:
- Built-ins commonly need a search before a precise read, then a line slice or full context before editing.
- Serena can skip full-file reads when the symbol name path is known.
- When the symbol is not known, Serena's `get_symbols_overview` and shallow `find_symbol` calls provide compact discovery rather than full-file reads.
Output payload:
- Serena exploration output is structured and scoped to symbols.
- Built-in grep output can be larger because it includes declarations, comments, strings, and unrelated text mentions.
- Built-ins can be extremely low-output for known-location small edits.
Stable vs ephemeral addressing:
- Built-ins address code by file paths, line numbers, text context, or byte positions. Line numbers and slices go stale after edits.
- Serena addresses symbols by `relative_path + name_path`, with overload indexes where needed. The chained-edit experiment showed those targets survived line shifts.
- Path-changing operations still require updated paths afterward, so symbol stability is strongest before moves/renames that alter file locations.
**Verdict:** Serena saves tokens by avoiding broad reads/search triage and by collapsing multi-file edit chains; built-ins remain more token-efficient for tiny known-location hunks.
## 5. Reliability & correctness under correct use
Precision of matching:
- Serena reference search returned code usages and excluded route strings/comments.
- Built-in text search returned all mentions and therefore mixed true code usage with broader text hits.
- Serena overload indexes targeted `Symbol/getLocationString[0]` and `[1]`; text search required manual signature inspection.
Scope disambiguation:
- Serena name paths encode class nesting and overload identity.
- Built-ins can disambiguate with careful patterns and reads, but the disambiguation is manual.
- In simple cases, such as a private helper with one call site, this semantic precision had little practical value. In overloaded, nested, or cross-file cases, it materially reduced ambiguity.
Atomicity:
- Serena refactors apply through JetBrains refactoring operations and produce one operation-level success signal.
- Built-in equivalents are decomposed into separate searches, file moves, patches, and verification steps.
- Competent users can verify either path, but Serena reduces the number of intermediate inconsistent states.
Semantic queries vs text search:
- `type_hierarchy` and dependency declaration lookup produced semantic information not available from ordinary grep.
- `rg` remained better for broad mention searches and config/resource scans.
External dependency limitations:
- Serena's dependency lookup depends on IDE/language indexes and available dependency metadata.
- Built-ins can match or exceed it only if paired with source jars, decompilers, or separate language-server tooling.
Operational tradeoff:
- Several Serena refactors staged created/deleted files. This is not a correctness problem, but it adds index-state verification to the cleanup workflow.
**Verdict:** Serena improves correctness where symbol identity and cross-file semantics matter; built-ins remain reliable and simpler for text/file tasks.
## 6. Workflow effects across a session
Where Serena advantages compound:
- Symbol overview results become inputs to body reads, reference searches, renames, moves, safe deletes, and inline operations.
- Name paths remain useful after line shifts, reducing re-read/re-target work during multi-edit sessions.
- Refactor-heavy sessions benefit because one semantic command replaces search/edit/verify loops across files.
Where Serena advantages diminish:
- After a full file has already been read, overview adds less incremental value.
- If the task is a small textual substitution in a known place, symbolic body replacement can cost more payload than a tiny patch.
- For non-code and free-text work, Serena has no role.
Intermediate result durability:
- Serena intermediate results such as `SymbolFinder/findFilesByName` or `Symbol/getLocationString[1]` remain meaningful as long as the file/symbol still exists.
- Built-in line numbers became stale after insertions; robust textual patch context remained usable when surrounding lines did not change.
Verification cost:
- Both workflows still need `git diff`, build/test, and domain-specific verification for permanent changes.
- Serena success signals reduce the need to inspect every call site manually, but not the need to inspect the intended diff.
**Verdict:** Serena's advantages compound inside symbol-centric sessions and diminish once the work becomes plain text, config, shell, or already-loaded single-file editing.
## 7. Unique capabilities
Unique here means no practical built-in equivalent without adding separate language-server, IDE, source-index, or decompiler infrastructure.
- Transitive type hierarchy. Frequency: medium. Impact: high when changing base classes, interfaces, or handler contracts. Built-ins can grep explicit `extends` clauses but do not produce a semantic transitive graph or external supertypes.
- External dependency declaration/body lookup from a call site. Frequency: low-medium. Impact: high for API behavior questions. Serena resolved `ReferencesSearch.search` into an external class method body; ordinary built-ins did not.
- IDE semantic move of a nested symbol to a top-level file with usage/import repair. Frequency: low-medium. Impact: high during refactors. Built-ins can reproduce this only as a manual multi-step refactor.
- Safe delete with semantic usage check. Frequency: medium. Impact: medium-high. Built-ins can search and delete, but they do not provide an integrated safe-delete operation over code usages.
- Inline refactor with call-site substitution. Frequency: low-medium. Impact: medium-high. Built-ins can patch substitutions manually, but expression adaptation and helper deletion are not primitive text operations.
Not unique: reading files, listing files, grep, small patches, simple one-file renames, config/resource understanding, and shell/Git operations.
**Verdict:** Serena's unique practical capabilities are hierarchy/dependency queries and IDE refactors; its non-unique areas are ordinary text and file operations.
## 8. Tasks outside Serena's scope
Built-in-only or built-in-natural tasks observed:
- Repository inventory and top-level layout.
- Reading `plugin.xml`, `build.gradle.kts`, scripts, resources, and generated artifacts.
- Free-text search for URLs, settings strings, endpoint strings, magic constants, resource IDs, and comments.
- Running shell commands, builds, tests, Git operations, and filesystem cleanup.
- Small text edits where a one-line patch is the whole task.
Estimated share of daily coding work:
- About 30-50% of a normal coding session is built-in-natural: shell, tests, grep, config reads, small text patches, and Git checks.
- Serena applies to much of the remaining 50-70% when the work is code navigation, symbol understanding, references, hierarchy, or refactoring.
- Serena coverage rises in refactor-heavy sessions and falls in config/debug/test-log sessions.
**Verdict:** Serena augments the semantic-code portion of development; built-ins still carry a large and necessary share of routine work.
## 9. Practical usage rule
Use Serena when the task can be stated as a code-symbol operation:
- "What methods/classes are in this file?"
- "Give me this method body."
- "Who uses this symbol?"
- "What implements/subclasses this?"
- "Rename, move, delete, or inline this symbol and update usages."
- "Find the declaration of this call, including dependency symbols."
- "Apply several edits where line locations may shift."
Use built-ins when the task is text, file, shell, config, or tiny local editing:
- "List files."
- "Read config."
- "Search for this string anywhere."
- "Change this one known line."
- "Run tests/build."
- "Inspect Git state."
- "Edit markdown/resources/scripts."
For mixed workflows:
1. Use built-ins to find files/config and run verification.
2. Use Serena to identify and transform code symbols.
3. Use built-ins to inspect final diffs and run tests.
4. Prefer built-ins for tiny local hunks after the symbol has already been located.
5. Prefer Serena for any refactor that crosses file boundaries or depends on type/reference semantics.
**Verdict:** Choose Serena for symbol identity and semantic refactoring; choose built-ins for text, shell, config, verification, and minimal local edits.
docs/_config.yml
+1 -1
View File
@@ -102,7 +102,7 @@ sphinx:
local_extensions : # A list of local extensions to load by sphinx specified by "name: path" items
recursive_update : false # A boolean indicating whether to overwrite the Sphinx config (true) or recursively update (false)
config : # key-value pairs to directly over-ride the Sphinx configuration
master_doc: "01-about/000_intro.md"
master_doc: "01-about/000_evaluation-results.md"
html_theme_options:
logo:
image_light: ../resources/serena-logo.svg
docs/autogen_docs.py
+1 -1
View File
@@ -190,7 +190,7 @@ def autogen_about_intro_features():
autogen_info = f"<!-- This section is auto-generated by {__file__} from the root README.md; do not edit. -->\n\n"
with open(Path(__file__).parent / "01-about" / "000_intro.md", "w", encoding="utf-8") as f:
with open(Path(__file__).parent / "01-about" / "000_evaluation-results.md", "w", encoding="utf-8") as f:
f.write(autogen_info)
f.write("# About Serena\n\n")
f.write(f"**{tagline}**\n\n")
src/serena/tools/jetbrains_tools.py
+19
View File
@@ -58,6 +58,8 @@ class JetBrainsFindSymbolTool(Tool, ToolMarkerSymbolicRead, ToolMarkerOptional):
for the case where the symbol is a class, this will return its methods).
Ignored if `include_body=True`. Default 0.
:param relative_path: Optional. Restrict search to this file or directory. If not specified, searches entire codebase.
Note: for external dependencies, this must be an identifier starting with `<ext` that you have received
earlier (don't try to guess!).
:param include_body: If True, include the symbol's source code. Use judiciously.
:param include_info: whether to include additional info (hover-like, typically including docstring and signature),
about the symbol.
@@ -71,8 +73,16 @@ class JetBrainsFindSymbolTool(Tool, ToolMarkerSymbolicRead, ToolMarkerOptional):
if include_body:
depth = 0 # ignore user-specified depth if body is requested
name_path_pattern = self._sanitize_input_param(name_path_pattern)
if relative_path:
relative_path = self._sanitize_input_param(relative_path)
if relative_path == ".":
relative_path = None
if relative_path is not None and relative_path.startswith(jb.JB_EXTERNAL_FILE_PREFIX):
search_deps = True
with JetBrainsPluginClient.from_project(self.project) as client:
if include_body:
include_quick_info = False
@@ -149,6 +159,7 @@ class JetBrainsMoveTool(Tool, ToolMarkerSymbolicEdit, ToolMarkerOptional, ToolMa
:param target_relative_path: the relative path of the target directory or file.
:param target_parent_name_path: the name path of the target parent symbol.
"""
relative_path = self._sanitize_input_param(relative_path)
with JetBrainsPluginClient.from_project(self.project) as client:
response_dict = client.move(
name_path=name_path,
@@ -187,6 +198,7 @@ class JetBrainsSafeDeleteTool(Tool, ToolMarkerSymbolicEdit, ToolMarkerOptional,
:param propagate: whether to propagate the deletion to usages of the symbol and also
remove symbols that become unused after the deletion. Default is False.
"""
relative_path = self._sanitize_input_param(relative_path)
with JetBrainsPluginClient.from_project(self.project) as client:
response_dict = client.safe_delete(
name_path=name_path,
@@ -220,6 +232,7 @@ class JetBrainsInlineSymbol(Tool, ToolMarkerSymbolicEdit, ToolMarkerOptional, To
:param keep_definition: whether to keep the original method definition after inlining all call sites.
May be ignored in some cases (e.g. when inlining a class).
"""
relative_path = self._sanitize_input_param(relative_path)
with JetBrainsPluginClient.from_project(self.project) as client:
response_dict = client.inline_symbol(
name_path=name_path,
@@ -251,6 +264,7 @@ class JetBrainsFindReferencingSymbolsTool(Tool, ToolMarkerSymbolicRead, ToolMark
:param relative_path: the relative path to the file containing the symbol (must be a file, not a directory)
:param max_answer_chars: max characters for the result (-1 for default). If exceeded, no content/a shortened result is returned.
"""
relative_path = self._sanitize_input_param(relative_path)
with JetBrainsPluginClient.from_project(self.project) as client:
response_dict = client.find_references(
name_path=name_path,
@@ -317,6 +331,7 @@ class JetBrainsGetSymbolsOverviewTool(Tool, ToolMarkerSymbolicRead, ToolMarkerOp
:param max_answer_chars: max characters for the result (-1 for default). If exceeded, no content/a shortened result is returned.
:param include_file_documentation: whether to include the file's docstring. Default False.
"""
relative_path = self._sanitize_input_param(relative_path)
with JetBrainsPluginClient.from_project(self.project) as client:
symbol_overview = client.get_symbols_overview(
relative_path=relative_path, depth=depth, include_file_documentation=include_file_documentation
@@ -419,6 +434,7 @@ class JetBrainsTypeHierarchyTool(Tool, ToolMarkerSymbolicRead, ToolMarkerOptiona
-1 means the default value from the config will be used.
:return: Compact JSON with file-grouped hierarchy. Error string if not applicable.
"""
relative_path = self._sanitize_input_param(relative_path)
with JetBrainsPluginClient.from_project(self.project) as client:
subtypes = None
supertypes = None
@@ -473,6 +489,9 @@ class JetBrainsFindDeclarationTool(Tool, ToolMarkerSymbolicRead, ToolMarkerOptio
Uses Python syntax with MULTILINE and DOTALL flags enabled.
:param include_body: whether to include the symbol's body in the result. Default False.
"""
relative_path = self._sanitize_input_param(relative_path)
regex = self._sanitize_input_param(regex)
editor = self.create_code_editor()
content = editor.read_file(relative_path)
coords = find_text_coordinates(content, regex, require_unique=True)
src/serena/tools/symbol_tools.py
+2 -1
View File
@@ -258,7 +258,8 @@ class FindReferencingSymbolsTool(Tool, ToolMarkerSymbolicRead):
:param name_path: for finding the symbol to find references for, same logic as in the `find_symbol` tool.
:param relative_path: the relative path to the file containing the symbol for which to find references.
Note that here you can't pass a directory but must pass a file.
Note: for external dependencies, this must be an identifier starting with `<ext` that you have received
earlier (don't try to guess!).
:param include_kinds: same as in the `find_symbol` tool.
:param exclude_kinds: same as in the `find_symbol` tool.
:param max_answer_chars: same as in the `find_symbol` tool.
src/serena/tools/tools_base.py
+5
View File
@@ -137,6 +137,11 @@ class Tool(Component):
_last_tool_call_client_str: str | None = None
"""We can only get the client info from within a tool call. Each tool call will update this variable."""
@staticmethod
def _sanitize_input_param(raw_param: str) -> str:
# some clients replace < and > with their escaped html versions, we need to counteract this
return raw_param.replace("&lt;", "<").replace("&gt;", ">")
@classmethod
def set_last_tool_call_client_str(cls, client_str: str | None) -> None:
cls._last_tool_call_client_str = client_str