mirror of
https://github.com/tiennm99/serena.git
synced 2026-10-12 16:15:33 +00:00
Minor improvement in readme and docs [ci skip]
This commit is contained in:
1 parent
1464fee875
commit
c3a5652fc7
2 files changed
+16
-17
No files matched your search
@@ -30,25 +30,24 @@ more complex codebases.
|
||||
|
||||
## What Our "End Users" Say
|
||||
|
||||
While it is humans who download and set up Serena, our end users are essentially AI agents,
|
||||
so they are also in the best position to evaluate Serena.
|
||||
We crafted an unbiased and detailed prompt which leads the agent to estimate the value of adding Serena's tools
|
||||
to its built-in capabilities. The thorough evaluation usually takes around 25 minutes (per project)
|
||||
and tests every single aspect of Serena. A summary of what the agents had to say:
|
||||
While it is humans who download and set up Serena, our end users are essentially AI agents,
|
||||
so they are also in the best position to evaluate Serena.
|
||||
We crafted an unbiased and detailed evaluation prompt
|
||||
that leads the agent to perform ~20 routine coding tasks, representative of everyday development work, using both Serena's tools and its own built-ins,
|
||||
measure the differences, and report the results. A one-sentence summary of what the agents had to say:
|
||||
|
||||
**Opus 4.6 (high effort) in Claude Code on a large Python codebase:**
|
||||
> "Serena's IDE-backed semantic tools are the single most impactful addition to my toolkit — cross-file renames, moves, and reference lookups that
|
||||
would cost me 8–12 careful, error-prone steps collapse into one atomic call, and I would absolutely ask any developer I work with to set them up."
|
||||
|
||||
**Gpt 5.4 (high) in Codex CLI on a Java codebase:**
|
||||
**GPT 5.4 (high) in Codex CLI on a Java codebase:**
|
||||
> "As a coding AI agent, I would ask my owner to add Serena because it gives me the missing IDE-level understanding of symbols, references, and
|
||||
refactorings, turning fragile text surgery into calmer, faster, more confident code changes where semantics matter."
|
||||
|
||||
Your agent deserves the best coding tools, give them Serena!
|
||||
Give your agent the tools it's been asking for and add Serena MCP to your client!
|
||||
|
||||
See the documentation on our [evaluation methods](https://oraios.github.io/serena/04-evaluation/000_intro.html) and the
|
||||
detailed results beyond the brief recommendations above. You can easily run your own evaluation of Serena on a project of your choice
|
||||
by reusing our methods or adapting them to your needs.
|
||||
See our [documentation](https://oraios.github.io/serena/04-evaluation/000_intro.html) for the full methodology and much more detailed evaluation results
|
||||
beyond these brief summaries, or run your own evaluation on a project of your choice.
|
||||
|
||||
|
||||
## How Serena Works
|
||||
@@ -81,7 +80,7 @@ complex projects!
|
||||
|
||||
There are two alternative technologies powering these capabilities:
|
||||
|
||||
* **Language servers** implementing the language server Protocol (LSP) — the free/open-source alternative
|
||||
* **Language servers** implementing the language server protocol (LSP) — the free/open-source alternative
|
||||
which is used by default.
|
||||
* The **Serena JetBrains Plugin**, which leverages the powerful code analysis and editing
|
||||
capabilities of your JetBrains IDE (paid plugin; free trial available).
|
||||
|
||||
@@ -7,7 +7,7 @@ built-in capabilities (file reads, text edits, grep, shell, etc.).
|
||||
Rather than a simple thumbs-up/thumbs-down, it produces a detailed, evidence-based report
|
||||
covering capabilities, efficiency and reliability.
|
||||
|
||||
### Why Not Benchmarks?
|
||||
## Why Not Benchmarks?
|
||||
|
||||
Standard coding benchmarks (SWE-bench, HumanEval, etc.) measure an agent's ability to solve
|
||||
predefined tasks with a known correct answer. They are valuable for comparing models and agents,
|
||||
@@ -28,7 +28,7 @@ but they are a poor fit for evaluating a **tool augmentation layer** like Serena
|
||||
We wanted an evaluation that systematically covers the full surface area of Serena's capabilities
|
||||
without cherry-picking.
|
||||
|
||||
### Design Goals
|
||||
## Design Goals
|
||||
|
||||
Instead of benchmarks, we designed an evaluation methodology with three goals:
|
||||
|
||||
@@ -56,7 +56,7 @@ Instead of benchmarks, we designed an evaluation methodology with three goals:
|
||||
explicitly requires reporting negative deltas and cases where Serena offers no improvement,
|
||||
structurally counterbalancing any tendency to favour the tool being evaluated.
|
||||
|
||||
### Method
|
||||
## Method
|
||||
|
||||
We give an AI coding agent a single, detailed [evaluation prompt](010_evaluation-prompt) in a one-shot session.
|
||||
The prompt instructs the agent to perform approximately 20 hands-on tasks across five areas:
|
||||
@@ -80,7 +80,7 @@ Only category (b) constitutes a neutral or negative finding; category (c) is con
|
||||
After the evaluation, a separate [follow-up prompt](011_followup-summary-prompt) asks the agent for a
|
||||
one-sentence, user-facing recommendation — the quotes shown on the [main page](https://github.com/oraios/serena).
|
||||
|
||||
### Results
|
||||
## Results
|
||||
|
||||
We performed evaluations using popular AI coding agents in representative scenarios — different
|
||||
agents, different programming languages, and different codebases — to show that the results are
|
||||
@@ -96,7 +96,7 @@ evaluation can easily be repeated with the LSP-based backend to assess its subse
|
||||
You can run your own evaluation on a project of your choice by reusing our
|
||||
[evaluation prompt](010_evaluation-prompt) or adapting it to your needs.
|
||||
|
||||
### Assessment of the Methodology
|
||||
## Assessment of the Methodology
|
||||
|
||||
_The following assessment was written by Claude Opus 4.6 (high effort) after reading the full
|
||||
evaluation methodology, evaluation prompt, and all result documents._
|
||||
@@ -144,7 +144,7 @@ One thing to watch for is whether agents that are less capable than Opus 4.6 or
|
||||
meaningful evaluations — the prompt is quite demanding. But that is a question about the agent, not
|
||||
about the method.
|
||||
|
||||
### Prompt Fairness
|
||||
## Prompt Fairness
|
||||
|
||||
We put significant effort in crafting an unbiased, fair and objective evaluation prompt.
|
||||
The only "biased" aspects are some sentences about misuse of tools, which we consider irrelevant for the purpose of evaluation
|
||||
|
||||
Reference in new issue
Block a user