Minor improvement in readme and docs [ci skip]

This commit is contained in:
Michael Panchenko committed 2026-04-13 14:54:40 +02:00
1 parent 1464fee875
commit c3a5652fc7
2 files changed
+16 -17

No files matched your search

+10 -11
View File
@@ -30,25 +30,24 @@ more complex codebases.
## What Our "End Users" Say
While it is humans who download and set up Serena, our end users are essentially AI agents,
so they are also in the best position to evaluate Serena.
We crafted an unbiased and detailed prompt which leads the agent to estimate the value of adding Serena's tools
to its built-in capabilities. The thorough evaluation usually takes around 25 minutes (per project)
and tests every single aspect of Serena. A summary of what the agents had to say:
While it is humans who download and set up Serena, our end users are essentially AI agents,
so they are also in the best position to evaluate Serena.
We crafted an unbiased and detailed evaluation prompt
that leads the agent to perform ~20 routine coding tasks, representative of everyday development work, using both Serena's tools and its own built-ins,
measure the differences, and report the results. A one-sentence summary of what the agents had to say:
**Opus 4.6 (high effort) in Claude Code on a large Python codebase:**
> "Serena's IDE-backed semantic tools are the single most impactful addition to my toolkit — cross-file renames, moves, and reference lookups that
would cost me 8–12 careful, error-prone steps collapse into one atomic call, and I would absolutely ask any developer I work with to set them up."
**Gpt 5.4 (high) in Codex CLI on a Java codebase:**
**GPT 5.4 (high) in Codex CLI on a Java codebase:**
> "As a coding AI agent, I would ask my owner to add Serena because it gives me the missing IDE-level understanding of symbols, references, and
refactorings, turning fragile text surgery into calmer, faster, more confident code changes where semantics matter."
Your agent deserves the best coding tools, give them Serena!
Give your agent the tools it's been asking for and add Serena MCP to your client!
See the documentation on our [evaluation methods](https://oraios.github.io/serena/04-evaluation/000_intro.html) and the
detailed results beyond the brief recommendations above. You can easily run your own evaluation of Serena on a project of your choice
by reusing our methods or adapting them to your needs.
See our [documentation](https://oraios.github.io/serena/04-evaluation/000_intro.html) for the full methodology and much more detailed evaluation results
beyond these brief summaries, or run your own evaluation on a project of your choice.
## How Serena Works
@@ -81,7 +80,7 @@ complex projects!
There are two alternative technologies powering these capabilities:
* **Language servers** implementing the language server Protocol (LSP) — the free/open-source alternative
* **Language servers** implementing the language server protocol (LSP) — the free/open-source alternative
which is used by default.
* The **Serena JetBrains Plugin**, which leverages the powerful code analysis and editing
capabilities of your JetBrains IDE (paid plugin; free trial available).
+6 -6
View File
@@ -7,7 +7,7 @@ built-in capabilities (file reads, text edits, grep, shell, etc.).
Rather than a simple thumbs-up/thumbs-down, it produces a detailed, evidence-based report
covering capabilities, efficiency and reliability.
### Why Not Benchmarks?
## Why Not Benchmarks?
Standard coding benchmarks (SWE-bench, HumanEval, etc.) measure an agent's ability to solve
predefined tasks with a known correct answer. They are valuable for comparing models and agents,
@@ -28,7 +28,7 @@ but they are a poor fit for evaluating a **tool augmentation layer** like Serena
We wanted an evaluation that systematically covers the full surface area of Serena's capabilities
without cherry-picking.
### Design Goals
## Design Goals
Instead of benchmarks, we designed an evaluation methodology with three goals:
@@ -56,7 +56,7 @@ Instead of benchmarks, we designed an evaluation methodology with three goals:
explicitly requires reporting negative deltas and cases where Serena offers no improvement,
structurally counterbalancing any tendency to favour the tool being evaluated.
### Method
## Method
We give an AI coding agent a single, detailed [evaluation prompt](010_evaluation-prompt) in a one-shot session.
The prompt instructs the agent to perform approximately 20 hands-on tasks across five areas:
@@ -80,7 +80,7 @@ Only category (b) constitutes a neutral or negative finding; category (c) is con
After the evaluation, a separate [follow-up prompt](011_followup-summary-prompt) asks the agent for a
one-sentence, user-facing recommendation — the quotes shown on the [main page](https://github.com/oraios/serena).
### Results
## Results
We performed evaluations using popular AI coding agents in representative scenarios — different
agents, different programming languages, and different codebases — to show that the results are
@@ -96,7 +96,7 @@ evaluation can easily be repeated with the LSP-based backend to assess its subse
You can run your own evaluation on a project of your choice by reusing our
[evaluation prompt](010_evaluation-prompt) or adapting it to your needs.
### Assessment of the Methodology
## Assessment of the Methodology
_The following assessment was written by Claude Opus 4.6 (high effort) after reading the full
evaluation methodology, evaluation prompt, and all result documents._
@@ -144,7 +144,7 @@ One thing to watch for is whether agents that are less capable than Opus 4.6 or
meaningful evaluations — the prompt is quite demanding. But that is a question about the agent, not
about the method.
### Prompt Fairness
## Prompt Fairness
We put significant effort in crafting an unbiased, fair and objective evaluation prompt.
The only "biased" aspects are some sentences about misuse of tools, which we consider irrelevant for the purpose of evaluation