diff --git a/README.md b/README.md index 72e3e467..81198d80 100644 --- a/README.md +++ b/README.md @@ -30,25 +30,24 @@ more complex codebases. ## What Our "End Users" Say -While it is humans who download and set up Serena, our end users are essentially AI agents, -so they are also in the best position to evaluate Serena. -We crafted an unbiased and detailed prompt which leads the agent to estimate the value of adding Serena's tools -to its built-in capabilities. The thorough evaluation usually takes around 25 minutes (per project) -and tests every single aspect of Serena. A summary of what the agents had to say: +While it is humans who download and set up Serena, our end users are essentially AI agents, +so they are also in the best position to evaluate Serena. +We crafted an unbiased and detailed evaluation prompt +that leads the agent to perform ~20 routine coding tasks, representative of everyday development work, using both Serena's tools and its own built-ins, +measure the differences, and report the results. A one-sentence summary of what the agents had to say: **Opus 4.6 (high effort) in Claude Code on a large Python codebase:** > "Serena's IDE-backed semantic tools are the single most impactful addition to my toolkit — cross-file renames, moves, and reference lookups that would cost me 8–12 careful, error-prone steps collapse into one atomic call, and I would absolutely ask any developer I work with to set them up." -**Gpt 5.4 (high) in Codex CLI on a Java codebase:** +**GPT 5.4 (high) in Codex CLI on a Java codebase:** > "As a coding AI agent, I would ask my owner to add Serena because it gives me the missing IDE-level understanding of symbols, references, and refactorings, turning fragile text surgery into calmer, faster, more confident code changes where semantics matter." -Your agent deserves the best coding tools, give them Serena! +Give your agent the tools it's been asking for and add Serena MCP to your client! -See the documentation on our [evaluation methods](https://oraios.github.io/serena/04-evaluation/000_intro.html) and the -detailed results beyond the brief recommendations above. You can easily run your own evaluation of Serena on a project of your choice -by reusing our methods or adapting them to your needs. +See our [documentation](https://oraios.github.io/serena/04-evaluation/000_intro.html) for the full methodology and much more detailed evaluation results +beyond these brief summaries, or run your own evaluation on a project of your choice. ## How Serena Works @@ -81,7 +80,7 @@ complex projects! There are two alternative technologies powering these capabilities: -* **Language servers** implementing the language server Protocol (LSP) — the free/open-source alternative +* **Language servers** implementing the language server protocol (LSP) — the free/open-source alternative which is used by default. * The **Serena JetBrains Plugin**, which leverages the powerful code analysis and editing capabilities of your JetBrains IDE (paid plugin; free trial available). diff --git a/docs/04-evaluation/000_intro.md b/docs/04-evaluation/000_intro.md index 74b300af..1b91a316 100644 --- a/docs/04-evaluation/000_intro.md +++ b/docs/04-evaluation/000_intro.md @@ -7,7 +7,7 @@ built-in capabilities (file reads, text edits, grep, shell, etc.). Rather than a simple thumbs-up/thumbs-down, it produces a detailed, evidence-based report covering capabilities, efficiency and reliability. -### Why Not Benchmarks? +## Why Not Benchmarks? Standard coding benchmarks (SWE-bench, HumanEval, etc.) measure an agent's ability to solve predefined tasks with a known correct answer. They are valuable for comparing models and agents, @@ -28,7 +28,7 @@ but they are a poor fit for evaluating a **tool augmentation layer** like Serena We wanted an evaluation that systematically covers the full surface area of Serena's capabilities without cherry-picking. -### Design Goals +## Design Goals Instead of benchmarks, we designed an evaluation methodology with three goals: @@ -56,7 +56,7 @@ Instead of benchmarks, we designed an evaluation methodology with three goals: explicitly requires reporting negative deltas and cases where Serena offers no improvement, structurally counterbalancing any tendency to favour the tool being evaluated. -### Method +## Method We give an AI coding agent a single, detailed [evaluation prompt](010_evaluation-prompt) in a one-shot session. The prompt instructs the agent to perform approximately 20 hands-on tasks across five areas: @@ -80,7 +80,7 @@ Only category (b) constitutes a neutral or negative finding; category (c) is con After the evaluation, a separate [follow-up prompt](011_followup-summary-prompt) asks the agent for a one-sentence, user-facing recommendation — the quotes shown on the [main page](https://github.com/oraios/serena). -### Results +## Results We performed evaluations using popular AI coding agents in representative scenarios — different agents, different programming languages, and different codebases — to show that the results are @@ -96,7 +96,7 @@ evaluation can easily be repeated with the LSP-based backend to assess its subse You can run your own evaluation on a project of your choice by reusing our [evaluation prompt](010_evaluation-prompt) or adapting it to your needs. -### Assessment of the Methodology +## Assessment of the Methodology _The following assessment was written by Claude Opus 4.6 (high effort) after reading the full evaluation methodology, evaluation prompt, and all result documents._ @@ -144,7 +144,7 @@ One thing to watch for is whether agents that are less capable than Opus 4.6 or meaningful evaluations — the prompt is quite demanding. But that is a question about the agent, not about the method. -### Prompt Fairness +## Prompt Fairness We put significant effort in crafting an unbiased, fair and objective evaluation prompt. The only "biased" aspects are some sentences about misuse of tools, which we consider irrelevant for the purpose of evaluation