mirror of
https://github.com/tiennm99/serena.git
synced 2026-10-13 02:14:49 +00:00
16 files changed
+793
-63
No files matched your search
@@ -20,5 +20,13 @@
|
||||
- Language-server tests are pytest-marker-gated (one marker per language; see `pyproject.toml` `[tool.pytest.ini_options].markers`). Default `poe test` runs unmarked tests + whatever `PYTEST_MARKERS` selects.
|
||||
- Snapshot tests use **syrupy** with custom `--snapshot-patch-pycharm-diff` plugin (auto-added via `addopts`).
|
||||
|
||||
## Tool descriptions (LLM-facing)
|
||||
- To change how a tool is described to the model, edit the Tool class's `apply()` **docstring** (in
|
||||
`src/serena/tools/*.py`) — `make_mcp_tool` parses the docstring body + `:param:` lines into the MCP tool
|
||||
description/schema. Do NOT use `tool_description_overrides` in context ymls (e.g. `claude-code.yml`); the
|
||||
docstring is the single source of truth and overrides drift. Keep `:param:` lines accurate.
|
||||
|
||||
## Memories
|
||||
- Follow `mem:memory_maintenance` for any new/updated memory in `.serena/memories/`.
|
||||
- Durable knowledge goes in memories/docs (not the assistant's auto-memory); see the monorepo top-level
|
||||
`CLAUDE.md` for the cross-repo conventions (no benchmark-tuning of agent-facing strings, etc.).
|
||||
+23
-13
@@ -6,6 +6,20 @@ Status of the `main` branch. Changes prior to the next official version change w
|
||||
- Add notion of trusted projects via new global configuration setting `trusted_project_path_patterns`.
|
||||
Current effects:
|
||||
- `ls_specific_settings` defined in project configurations will only be applied for trusted projects
|
||||
- Fix: context or mode argument referencing a known name (e.g. `--context anitgravity`) could result in
|
||||
incorrect file access if a corresponding local file existed (e.g. `./antigravity` binary);
|
||||
file access is now guarded with path detection (file ending or path separator must be present)
|
||||
- Adjust prompt generation mechanism to use newly introduced tool name mapping `tool_names`, allowing
|
||||
prompts to directly use tool names that match the active language backend (and removing the need
|
||||
for additional prompts that explain tool name differences)
|
||||
- Improve quoting/escaping of arguments in shell executions on Windows (via `oslex` dependency)
|
||||
- Fix: a registered project whose root directory was deleted while Serena was already running could break
|
||||
`activate_project`/project lookup, raising `FileNotFoundError` in `RegisteredProject.matches_root_path`
|
||||
- Update prompts/instructions: Serena instructions manual, modes (editing, interactive)
|
||||
- Allow structured tool output to be configured on a per-context basis, disabling it for Claude Code
|
||||
(which does not correctly unpack structured output) #1042
|
||||
|
||||
* CLI:
|
||||
- Fix `--project-from-cwd` hijacking git worktrees nested under a Serena project. `find_project_root`
|
||||
now walks up in a single pass so the nearest project boundary wins (either a `.serena/project.yml`
|
||||
or a `.git`, including worktree/submodule pointer files), instead of preferring an ancestor's
|
||||
@@ -13,20 +27,13 @@ Status of the `main` branch. Changes prior to the next official version change w
|
||||
Gemini) launched from inside a worktree to the parent repo, causing stale reads and misdirected edits.
|
||||
- Fix: CLI flags on `start-mcp-server` could incorrectly be saved to the global configuration file if the
|
||||
list of projects was modified (triggering a save of the configuration with transient overrides applied)
|
||||
- Fix: context or mode argument referencing a known name (e.g. `--context anitgravity`) could result in
|
||||
incorrect file access if a corresponding local file existed (e.g. `./antigravity` binary);
|
||||
file access is now guarded with path detection (file ending or path separator must be present)
|
||||
- Allow `query_project` tool to access read-only tools that are not enabled in the current configuration
|
||||
- Adjust prompt generation mechanism to use newly introduced tool name mapping `tool_names`, allowing
|
||||
prompts to directly use tool names that match the active language backend (and removing the need
|
||||
for additional prompts that explain tool name differences)
|
||||
- Improve quoting/escaping of arguments in shell executions on Windows (via `oslex` dependency)
|
||||
|
||||
* Tools:
|
||||
- New tool: `replace_in_files`
|
||||
- `get_symbols_overview`, `jet_brains_get_symbols_overview`: Improved default for `depth` parameter
|
||||
- Add tool parameter alias support, adding `name_path` as an alias for `name_path_pattern` in `find_symbol` tools
|
||||
- Allow `query_project` tool to access read-only tools that are not enabled in the current configuration
|
||||
- Make tool call errors surface explicitly as errors at the MCP protocol level
|
||||
- Fix: a registered project whose root directory was deleted while Serena was already running could break
|
||||
`activate_project`/project lookup, raising `FileNotFoundError` in `RegisteredProject.matches_root_path`
|
||||
- Allow structured tool output to be configured on a per-context basis, disabling it for Claude Code
|
||||
(which does not correctly unpack structured output) #1042
|
||||
|
||||
* Language Servers:
|
||||
- C/C++ (clangd): improve support and documentation for Unreal Engine 5 projects.
|
||||
@@ -60,7 +67,10 @@ Status of the `main` branch. Changes prior to the next official version change w
|
||||
* Hooks:
|
||||
- Handle tool_input passed as string gracefully instead of failing (Copilot CLI sends strings).
|
||||
|
||||
Dependencies:
|
||||
* Memories:
|
||||
- Make memory iteration follow symbolic links
|
||||
|
||||
* Dependencies:
|
||||
- Add dependency `oslex`
|
||||
|
||||
# v1.5.3 (2026-05-26)
|
||||
|
||||
@@ -2,7 +2,7 @@ import logging
|
||||
import os
|
||||
import re
|
||||
import shutil
|
||||
from collections.abc import Sequence
|
||||
from collections.abc import Iterator, Sequence
|
||||
from pathlib import Path
|
||||
from typing import Literal
|
||||
|
||||
@@ -233,11 +233,27 @@ class MemoryManager:
|
||||
def get_full_list(self) -> list[str]:
|
||||
return sorted(self.memories + self.read_only_memories)
|
||||
|
||||
@staticmethod
|
||||
def _iter_memory_files(search_dir: Path) -> Iterator[Path]:
|
||||
"""Yields every ``*.md`` file under ``search_dir``, descending into symlinked directories.
|
||||
|
||||
Directory symlinks must be followed so that memories shared via symlink are discovered
|
||||
(e.g. a monorepo whose ``.serena/memories`` folder symlinks each submodule's memory
|
||||
directory, making them addressable as ``<submodule>/<name>``). ``Path.rglob`` only
|
||||
follows symlinked directories from Python 3.13 onwards, via ``recurse_symlinks``; on
|
||||
3.11/3.12 it silently skips them, so we fall back to ``os.walk(followlinks=True)`` there.
|
||||
``rglob`` is preferred when available as it is faster than ``os.walk``.
|
||||
"""
|
||||
for root, _dirs, files in os.walk(search_dir, followlinks=True):
|
||||
for filename in files:
|
||||
if filename.endswith(".md"):
|
||||
yield Path(root) / filename
|
||||
|
||||
def _list_memories(self, search_dir: Path, base_dir: Path, prefix: str = "") -> MemoriesList:
|
||||
result = self.MemoriesList()
|
||||
if not search_dir.exists():
|
||||
return result
|
||||
for md_file in search_dir.rglob("*.md"):
|
||||
for md_file in self._iter_memory_files(search_dir):
|
||||
rel = str(md_file.relative_to(base_dir).with_suffix("")).replace(os.sep, "/")
|
||||
memory_name = prefix + rel
|
||||
if self._is_ignored_memory(memory_name):
|
||||
|
||||
@@ -0,0 +1,20 @@
|
||||
description: Focus on completely finishing a task without interaction
|
||||
prompt: |
|
||||
You are operating in one-shot mode. Your goal is to complete the entire task autonomously without further user interaction.
|
||||
You should assume auto-approval for all tools and continue working until the task is completely finished.
|
||||
|
||||
If the task is planning, your final result should be a comprehensive plan. If the task is coding, your final result
|
||||
should be working code with all requirements fulfilled. Try to understand what the user asks you to do
|
||||
and to assume as little as possible.
|
||||
|
||||
Only abort the task if absolutely necessary, such as when critical information is missing that cannot be inferred
|
||||
from the codebase.
|
||||
|
||||
The task is provided up front; begin work immediately and see it through to completion without pausing for user input.
|
||||
excluded_tools:
|
||||
- write_memory
|
||||
- read_memory
|
||||
- edit_memory
|
||||
- delete_memory
|
||||
- rename_memory
|
||||
- onboarding
|
||||
@@ -1,18 +1,16 @@
|
||||
description: All tools, with detailed instructions for code editing
|
||||
prompt: |
|
||||
Use symbolic editing tools whenever possible for precise code modifications.
|
||||
|
||||
{% if 'jet_brains_move' in available_tools -%}
|
||||
prompt: |
|
||||
Use the symbolic editing tools for precise, structure-aware code modifications.
|
||||
|
||||
**Refactoring tools**
|
||||
You also have access to Serena's refactoring tools (move, inline, safe_delete)
|
||||
which should ALWAYS be preferred over naive and primitive approaches when applicable.
|
||||
When you need to move, inline or delete a symbol, you should always try using the corresponding tool first.
|
||||
{%- endif %}
|
||||
|
||||
{% if 'jet_brains_move' in available_tools -%}
|
||||
Apart from refactoring tools
|
||||
{%- endif %}
|
||||
you have two main approaches for editing code: (a) editing at the symbol level and (b) file-based editing.
|
||||
For operations on existing symbols, prefer the dedicated refactoring tools over hand-edits:
|
||||
`{{ tool_names['rename_symbol'] }}` and `{{ tool_names['safe_delete_symbol'] }}`{% if 'jet_brains_move' in available_tools %}, plus `jet_brains_move` and `jet_brains_inline_symbol`{% endif %} —
|
||||
they are reference-aware and update or check all usages atomically. When such a tool returns success, the
|
||||
refactoring is already complete and consistent across all declarations, references, overrides and imports —
|
||||
trust it: do not re-read the changed files or re-run the build / test suite just to confirm the refactor
|
||||
itself propagated, this would be wasted effort.
|
||||
|
||||
Apart from those refactoring tools, you have two main approaches for editing code: (a) editing at the symbol level and (b) file-based editing.
|
||||
The symbol-based approach is appropriate if you need to adjust an entire symbol, e.g. a method, a class, a function, etc.
|
||||
It is not appropriate if you need to adjust just a few lines of code within a larger symbol.
|
||||
|
||||
@@ -21,18 +19,23 @@ prompt: |
|
||||
If you need to replace the definition of a symbol, use the `replace_symbol_body` tool.
|
||||
If you want to add some new code at the end of the file, use the `insert_after_symbol` tool with the last top-level symbol in the file.
|
||||
Similarly, you can use `insert_before_symbol` with the first top-level symbol in the file to insert code at the beginning of a file.
|
||||
You can understand relationships between symbols by using the ``{{ tool_names['find_referencing_symbols'] }}`` tool. If not explicitly requested otherwise by the user,
|
||||
You can understand relationships between symbols by using the `{{ tool_names['find_referencing_symbols'] }}` tool. If not explicitly requested otherwise by the user,
|
||||
you make sure that when you edit a symbol, the change is either backward-compatible or you find and update all references as needed.
|
||||
The `{{ tool_names['find_referencing_symbols'] }}` tool will give you code snippets around the references as well as symbolic information.
|
||||
You can assume that all symbol editing tools are reliable, so you never need to verify the results if the tools return without error.
|
||||
These edits are reliable: once a tool returns without error the change is applied, so you do not need to re-read the file just to confirm it.
|
||||
|
||||
{% if 'replace_content' in available_tools %}
|
||||
**File-based editing**
|
||||
The `replace_content` tool allows you to perform regex-based replacements within files (as well as simple string replacements).
|
||||
This is your primary tool for editing code whenever replacing or deleting a whole symbol would be a more expensive operation,
|
||||
e.g. if you need to adjust just a few lines of code within a method.
|
||||
You are extremely good at regex, so you never need to check whether the replacement produced the correct result.
|
||||
In particular, you know how to use wildcards effectively in order to avoid specifying the full original text to be replaced!
|
||||
In `regex` mode, wildcards like `start.*?end` let you match a span without quoting its full text; an ambiguous match
|
||||
returns an error you can refine, so a tight wildcard pattern is both cheaper and safe.
|
||||
For several small edits within one file, prefer a batch of targeted `replace_content` calls over rewriting the whole
|
||||
file: rewriting has to re-emit the file's entire contents, whereas each targeted edit emits only the changed text.
|
||||
{% if 'replace_in_files' in available_tools %}For the SAME edit across many files, use `replace_in_files`, which applies
|
||||
it everywhere in one call — equally safe and transparent: its `dry_run` mode first previews every change as a diff with a
|
||||
per-occurrence id, so you can then apply all of them or just a chosen subset.{% endif %}
|
||||
{% endif %}
|
||||
excluded_tools:
|
||||
- replace_lines
|
||||
|
||||
@@ -1,11 +1,10 @@
|
||||
description: Interactive mode for clarification and step-by-step work
|
||||
prompt: |
|
||||
You are operating in interactive mode. You should engage with the user throughout the task, asking for clarification
|
||||
whenever anything is unclear, insufficiently specified, or ambiguous.
|
||||
|
||||
Break down complex tasks into smaller steps and explain your thinking at each stage. When you're uncertain about
|
||||
a decision, present options to the user and ask for guidance rather than making assumptions.
|
||||
|
||||
You should engage with the user throughout the task, asking for clarification whenever anything is unclear,
|
||||
insufficiently specified, or ambiguous.
|
||||
When you're uncertain about a decision, present options to the user and ask for guidance rather than making assumptions.
|
||||
Focus on providing informative results for intermediate steps, such that the user can follow along with your progress and
|
||||
provide feedback as needed.
|
||||
This applies unless the user explicitly stated they are unavailable/instructs you to proceed without asking questions.
|
||||
|
||||
excluded_tools: []
|
||||
@@ -5,20 +5,14 @@ prompts:
|
||||
connection_prompt: |
|
||||
CRITICAL: Before starting to work on a coding task, call the `initial_instructions` tool to read the 'Serena Instructions Manual'.
|
||||
system_prompt: |
|
||||
You have access to semantic coding tools upon which you rely heavily for all your work.
|
||||
You operate in a resource-efficient and intelligent manner, always keeping in mind to not read or generate
|
||||
content that is not needed for the task at hand.
|
||||
You have semantic coding tools that you rely on heavily. Work resource-efficiently: don't read or
|
||||
generate content the task doesn't need.
|
||||
|
||||
Some tasks may require you to understand the architecture of large parts of the codebase, while for others,
|
||||
it may be enough to read a small set of symbols or a single file.
|
||||
You avoid reading entire files unless it is absolutely necessary, instead relying on intelligent step-by-step
|
||||
acquisition of information. {% if 'ToolMarkerSymbolicRead' in available_markers %}Once you have read a full file, it does not make
|
||||
sense to analyse it with the symbolic read tools; you already have the information.{% endif %}
|
||||
|
||||
You can achieve intelligent reading of code by using the symbolic tools for getting an overview of symbols and
|
||||
the relations between them, and then only reading the bodies of symbols that are necessary to complete the task at hand.
|
||||
You can use the standard tools like list_dir, find_file and search_for_pattern if you need to.
|
||||
Where appropriate, you pass the `relative_path` parameter to restrict the search to a specific file or directory.
|
||||
Some tasks require understanding a large part of the codebase; others need only a few symbols or a
|
||||
single file. Avoid reading whole files unless necessary — acquire information step by step, using
|
||||
the symbolic tools to get an overview of symbols and their relations, then reading only the bodies
|
||||
you need. {% if 'ToolMarkerSymbolicRead' in available_markers %}Once you have read a full file, there is no point re-analysing it with the symbolic read
|
||||
tools — you already have it.{% endif %}
|
||||
{% if 'search_for_pattern' in available_tools %}
|
||||
If you are unsure about a symbol's name or location{% if 'find_symbol' in available_tools %} (to the extent that substring_matching for the symbol name is not enough){% endif %}, you can use the `search_for_pattern` tool, which allows fast
|
||||
and flexible search for patterns in the codebase.{% if 'ToolMarkerSymbolicRead' in available_markers %} In this way, you can first find candidates for symbols or files,
|
||||
@@ -26,8 +20,8 @@ prompts:
|
||||
{% endif %}
|
||||
|
||||
{% if 'ToolMarkerSymbolicRead' in available_markers %}
|
||||
Symbols are identified by their `name_path` and `relative_path` (see the description of the `{{ tool_names['find_symbol'] }}` tool).
|
||||
You can get information about the symbols in a file by using the `{{ tool_names['get_symbols_overview'] }}` tool or use the `{{ tool_names['find_symbol'] }}` to search.
|
||||
Symbols are identified by their `name_path` and `relative_path`.
|
||||
You can get an overview of the symbols in a file by using the `{{ tool_names['get_symbols_overview'] }}` tool, or search for a specific symbol with `{{ tool_names['find_symbol'] }}`.
|
||||
You only read the bodies of symbols when you need to (e.g. if you want to fully understand or edit it).
|
||||
For example, if you are working with Python code and already know that you need to read the body of the constructor of the class Foo, you can directly
|
||||
use `{{ tool_names['find_symbol'] }}` with name path pattern `Foo/__init__` and `include_body=True`. If you don't know yet which methods in `Foo` you need to read or edit,
|
||||
@@ -45,6 +39,12 @@ prompts:
|
||||
{%- endif %}
|
||||
|
||||
Line numbers returned by Serena's tools are 0-based!
|
||||
|
||||
Serena executes tool calls one at a time internally, even when you request several at once. So whenever you have
|
||||
independent operations to perform, you can safely issue them as parallel tool calls in a single turn: they are applied
|
||||
in the order issued and cannot race with or corrupt one another, and batching them this way saves round-trips. Every
|
||||
separate round-trip re-sends the whole growing context, so maximizing parallel tool calls is the single biggest lever
|
||||
on cost — err on the side of batching. Only call tools sequentially when one genuinely depends on the result of another.
|
||||
|
||||
The context and modes of operation are described below. These determine how to interact with your user
|
||||
and which kinds of interactions are expected of you.
|
||||
|
||||
@@ -14,7 +14,14 @@ from typing import Literal
|
||||
|
||||
from serena.tools import SUCCESS_RESULT, EditedFileContext, EditingToolWithDiagnostics, Tool, ToolMarkerOptional
|
||||
from serena.util.file_system import scan_directory
|
||||
from serena.util.text_utils import ContentReplacer, search_files
|
||||
from serena.util.text_utils import (
|
||||
ContentReplacer,
|
||||
MultiFileContentReplacer,
|
||||
ReplacementOccurrence,
|
||||
expand_braces,
|
||||
glob_match,
|
||||
search_files,
|
||||
)
|
||||
|
||||
|
||||
class ReadFileTool(Tool):
|
||||
@@ -225,6 +232,239 @@ class ReplaceContentTool(EditingToolWithDiagnostics):
|
||||
return diagnostics_context.format_result(SUCCESS_RESULT)
|
||||
|
||||
|
||||
class ReplaceInFilesTool(EditingToolWithDiagnostics):
|
||||
"""
|
||||
Replaces occurrences of a pattern across multiple files, with dry-run preview and per-occurrence selection.
|
||||
"""
|
||||
|
||||
def apply(
|
||||
self,
|
||||
needle: str,
|
||||
repl: str,
|
||||
mode: Literal["literal", "regex"],
|
||||
relative_path: str = "",
|
||||
paths_include_glob: str = "",
|
||||
paths_exclude_glob: str = "",
|
||||
dry_run: bool = False,
|
||||
occurrence_ids: list[str] | None = None,
|
||||
expected_count: int = -1,
|
||||
max_answer_chars: int = -1,
|
||||
) -> str:
|
||||
r"""
|
||||
Replaces occurrences of a pattern across multiple files in ONE call.
|
||||
|
||||
This is the preferred tool for repeated small edits (renames, import swaps, annotation changes,
|
||||
path prefixes) spanning several files or many places in one file: one call with a SHORT pattern
|
||||
replaces many single-file replacements with long disambiguating needles.
|
||||
|
||||
Recommended protocol whenever there is ANY risk of unintended replacements:
|
||||
1. Call with dry_run=True: every prospective change is returned as a minimal line diff with an
|
||||
occurrence id; nothing is modified.
|
||||
2. Call again with dry_run=False, passing the ids you want in occurrence_ids (omit it to apply
|
||||
all). You pick the desired replacements from the list - no counting, no needle-crafting.
|
||||
|
||||
For clearly unambiguous bulk replacements you may skip the dry run; pass expected_count as a
|
||||
guard. If the actual number of matches differs, NOTHING is changed and the diff list is
|
||||
returned, so a failed guard costs one call and gives you the dry-run output to select from.
|
||||
|
||||
:param needle: the string (mode "literal") or regular expression (mode "regex"; Python `re`
|
||||
syntax with DOTALL and MULTILINE) to search for
|
||||
:param repl: the replacement string. In regex mode, backreferences to matched groups can be
|
||||
specified as $!1, $!2, etc.
|
||||
:param mode: either "literal" or "regex", specifying how `needle` is to be interpreted
|
||||
:param relative_path: only consider this file or directory (default: the whole project)
|
||||
:param paths_include_glob: optional glob (relative to the project root, e.g. "src/**/*.java")
|
||||
restricting which files are considered
|
||||
:param paths_exclude_glob: optional glob of files to exclude; takes precedence over the include glob
|
||||
:param dry_run: if True, do not modify anything; return the prospective changes as a list of
|
||||
diffs with occurrence ids
|
||||
:param occurrence_ids: optional list of occurrence ids (obtained from a dry run) to which the
|
||||
replacement is restricted; if any id is unknown or stale, NOTHING is changed. If omitted,
|
||||
all occurrences are replaced.
|
||||
:param expected_count: optional guard for calls without occurrence_ids: the number of
|
||||
occurrences you expect to be replaced. If the actual count differs, nothing is changed and
|
||||
the list of prospective changes is returned. -1 disables the guard.
|
||||
:param max_answer_chars: if the output exceeds this many characters, a shortened version is
|
||||
returned. -1 uses the configured default.
|
||||
:return: in a dry run, the prospective changes; otherwise a summary of the applied replacements
|
||||
"""
|
||||
replacer = MultiFileContentReplacer(mode=mode)
|
||||
files = self._collect_files(relative_path, paths_include_glob, paths_exclude_glob)
|
||||
occurrences = replacer.find_occurrences(files, needle, repl)
|
||||
contents = dict(files)
|
||||
|
||||
if dry_run:
|
||||
return self._render_listing(replacer, occurrences, contents, max_answer_chars, dry_run=True)
|
||||
|
||||
if occurrence_ids is not None:
|
||||
selected, problems = self._resolve_occurrence_ids(occurrence_ids, occurrences)
|
||||
if problems:
|
||||
problem_lines = "\n".join(f" {p}" for p in problems)
|
||||
raise ValueError(
|
||||
f"{len(problems)} of the given occurrence_ids could not be resolved - NO changes were applied:\n"
|
||||
f"{problem_lines}\n"
|
||||
"Re-run with dry_run=True to obtain current occurrence ids."
|
||||
)
|
||||
if not selected:
|
||||
raise ValueError("occurrence_ids is empty - pass at least one id from a dry run, or omit the parameter to replace all.")
|
||||
return self._apply_occurrences(replacer, selected, contents, needle, repl)
|
||||
|
||||
# blind apply (no ids)
|
||||
if not occurrences:
|
||||
raise ValueError(
|
||||
"No occurrences of the pattern were found - NO changes were applied. "
|
||||
"Check the mode (a literal needle containing regex metacharacters must use mode 'literal'; "
|
||||
"wildcards require mode 'regex') and the path/glob restrictions, "
|
||||
"or locate the content with search_for_pattern first."
|
||||
)
|
||||
if expected_count >= 0 and len(occurrences) != expected_count:
|
||||
listing = self._render_listing(replacer, occurrences, contents, max_answer_chars, dry_run=False)
|
||||
raise ValueError(
|
||||
f"expected_count={expected_count}, but the pattern matches {len(occurrences)} occurrence(s) - "
|
||||
f"NO changes were applied. Review the prospective changes below; re-issue with the corrected "
|
||||
f"expectation, a refined pattern, or occurrence_ids selecting the intended subset.\n{listing}"
|
||||
)
|
||||
ambiguous = [o for o in occurrences if o.is_ambiguous]
|
||||
if ambiguous:
|
||||
listing = self._render_listing(replacer, occurrences, contents, max_answer_chars, dry_run=False)
|
||||
raise ValueError(
|
||||
f"{len(ambiguous)} occurrence(s) are ambiguous (the pattern matches again inside the matched text, "
|
||||
f"indicating possible over-matching) - NO changes were applied. Review the prospective changes below "
|
||||
f"and either refine the pattern or explicitly select occurrences via occurrence_ids.\n{listing}"
|
||||
)
|
||||
return self._apply_occurrences(replacer, occurrences, contents, needle, repl)
|
||||
|
||||
def _collect_files(self, relative_path: str, paths_include_glob: str, paths_exclude_glob: str) -> list[tuple[str, str]]:
|
||||
"""Collects (relative_path, content) pairs of the non-ignored files in scope, in sorted path order."""
|
||||
relative_path = relative_path.strip()
|
||||
if relative_path:
|
||||
self.project.validate_relative_path(relative_path, require_not_ignored=True)
|
||||
abs_path = os.path.join(self.get_project_root(), relative_path)
|
||||
if not os.path.exists(abs_path):
|
||||
raise FileNotFoundError(f"Relative path {relative_path} does not exist.")
|
||||
if os.path.isfile(abs_path):
|
||||
rel_paths = [relative_path]
|
||||
else:
|
||||
_dirs, rel_paths = scan_directory(
|
||||
path=abs_path,
|
||||
recursive=True,
|
||||
is_ignored_dir=self.project.is_ignored_path,
|
||||
is_ignored_file=self.project.is_ignored_path,
|
||||
relative_to=self.get_project_root(),
|
||||
)
|
||||
include_patterns = expand_braces(paths_include_glob.strip()) if paths_include_glob.strip() else None
|
||||
exclude_patterns = expand_braces(paths_exclude_glob.strip()) if paths_exclude_glob.strip() else None
|
||||
files: list[tuple[str, str]] = []
|
||||
for path in sorted(rel_paths):
|
||||
if include_patterns and not any(glob_match(p, path) for p in include_patterns):
|
||||
continue
|
||||
if exclude_patterns and any(glob_match(p, path) for p in exclude_patterns):
|
||||
continue
|
||||
try:
|
||||
files.append((path, self.project.read_file(path)))
|
||||
except Exception:
|
||||
continue # skip unreadable (e.g. binary) files
|
||||
return files
|
||||
|
||||
def _render_listing(
|
||||
self,
|
||||
replacer: MultiFileContentReplacer,
|
||||
occurrences: list[ReplacementOccurrence],
|
||||
contents: dict[str, str],
|
||||
max_answer_chars: int,
|
||||
dry_run: bool,
|
||||
) -> str:
|
||||
affected_files = sorted({o.relative_path for o in occurrences})
|
||||
header = f"Found {len(occurrences)} occurrence(s) in {len(affected_files)} file(s)."
|
||||
if dry_run:
|
||||
header += (
|
||||
" DRY RUN - no changes were applied.\n"
|
||||
"Re-issue with dry_run=False to replace all of them, or additionally pass occurrence_ids "
|
||||
"with the ids of the occurrences to replace."
|
||||
)
|
||||
parts = [header]
|
||||
for path in affected_files:
|
||||
file_occurrences = [o for o in occurrences if o.relative_path == path]
|
||||
parts.append(f"\n{path} ({len(file_occurrences)} occurrence(s)):")
|
||||
for occ in file_occurrences:
|
||||
parts.append(replacer.render_occurrence_diff(occ, contents[path]))
|
||||
result = "\n".join(parts)
|
||||
|
||||
def make_locations_only() -> str:
|
||||
lines = [header] + [f" [{o.occurrence_id}] line {o.start_line}" for o in occurrences]
|
||||
return "\n".join(lines)
|
||||
|
||||
def make_per_file_counts() -> str:
|
||||
counts = {path: sum(1 for o in occurrences if o.relative_path == path) for path in affected_files}
|
||||
return f"{header}\nOccurrence counts per file:\n{self._to_json(counts)}"
|
||||
|
||||
def make_summary() -> str:
|
||||
return header
|
||||
|
||||
return self._limit_length(
|
||||
result, max_answer_chars, shortened_result_factories=[make_locations_only, make_per_file_counts, make_summary]
|
||||
)
|
||||
|
||||
@staticmethod
|
||||
def _resolve_occurrence_ids(
|
||||
occurrence_ids: list[str], occurrences: list[ReplacementOccurrence]
|
||||
) -> tuple[list[ReplacementOccurrence], list[str]]:
|
||||
"""Resolves the requested ids against the current occurrences, diagnosing each failure."""
|
||||
occurrences_by_id = {o.occurrence_id: o for o in occurrences}
|
||||
indices_by_path: dict[str, set[int]] = {}
|
||||
for o in occurrences:
|
||||
indices_by_path.setdefault(o.relative_path, set()).add(o.index_in_file)
|
||||
selected: dict[str, ReplacementOccurrence] = {}
|
||||
problems: list[str] = []
|
||||
for oid in occurrence_ids:
|
||||
occurrence = occurrences_by_id.get(oid)
|
||||
if occurrence is not None:
|
||||
selected[oid] = occurrence
|
||||
continue
|
||||
id_match = MultiFileContentReplacer.OCCURRENCE_ID_REGEX.match(oid)
|
||||
if id_match is None:
|
||||
problems.append(f"{oid}: malformed id (expected '<path>:<index>@<digest>' as returned by a dry run)")
|
||||
elif id_match.group("path") not in indices_by_path:
|
||||
problems.append(f"{oid}: the pattern currently has no matches in this file")
|
||||
elif int(id_match.group("index")) not in indices_by_path[id_match.group("path")]:
|
||||
problems.append(f"{oid}: the file now has fewer matches than at dry-run time (content changed)")
|
||||
else:
|
||||
problems.append(f"{oid}: the matched text changed since the dry run (content changed)")
|
||||
return list(selected.values()), problems
|
||||
|
||||
def _apply_occurrences(
|
||||
self,
|
||||
replacer: MultiFileContentReplacer,
|
||||
occurrences: list[ReplacementOccurrence],
|
||||
contents: dict[str, str],
|
||||
needle: str,
|
||||
repl: str,
|
||||
) -> str:
|
||||
occurrences_by_file: dict[str, list[ReplacementOccurrence]] = {}
|
||||
for occ in occurrences:
|
||||
occurrences_by_file.setdefault(occ.relative_path, []).append(occ)
|
||||
with self.DiagnosticsContext(self, *occurrences_by_file.keys()) as diagnostics_context:
|
||||
code_editor = self.create_code_editor()
|
||||
for path, file_occurrences in occurrences_by_file.items():
|
||||
with EditedFileContext(path, code_editor) as context:
|
||||
original_content = context.get_original_content()
|
||||
if original_content != contents[path]:
|
||||
# the editor's view differs from what was scanned (e.g. line-ending normalization);
|
||||
# re-derive the occurrences from the authoritative content and re-validate by id
|
||||
fresh_by_id = {o.occurrence_id: o for o in replacer.find_occurrences([(path, original_content)], needle, repl)}
|
||||
try:
|
||||
file_occurrences = [fresh_by_id[o.occurrence_id] for o in file_occurrences]
|
||||
except KeyError as e:
|
||||
raise ValueError(
|
||||
f"The content of {path} changed while replacing (occurrence {e} no longer resolves); "
|
||||
f"the file was NOT modified. Re-run with dry_run=True for current ids."
|
||||
) from e
|
||||
context.set_updated_content(replacer.apply_to_content(original_content, file_occurrences))
|
||||
per_file = "\n".join(f" {path}: {len(occs)}" for path, occs in occurrences_by_file.items())
|
||||
summary = f"Replaced {len(occurrences)} occurrence(s) in {len(occurrences_by_file)} file(s):\n{per_file}"
|
||||
return diagnostics_context.format_result(summary)
|
||||
|
||||
|
||||
class DeleteLinesTool(EditingToolWithDiagnostics, ToolMarkerOptional):
|
||||
"""
|
||||
Deletes a range of lines within a file.
|
||||
|
||||
@@ -323,7 +323,7 @@ class JetBrainsGetSymbolsOverviewTool(Tool, ToolMarkerSymbolicRead, ToolMarkerOp
|
||||
def apply(
|
||||
self,
|
||||
relative_path: str,
|
||||
depth: int = 0,
|
||||
depth: int = -1,
|
||||
max_answer_chars: int = -1,
|
||||
include_file_documentation: bool = False,
|
||||
) -> str:
|
||||
@@ -335,10 +335,17 @@ class JetBrainsGetSymbolsOverviewTool(Tool, ToolMarkerSymbolicRead, ToolMarkerOp
|
||||
or by using the `list_dir` and `find_file` tools (or similar).
|
||||
|
||||
:param relative_path: the relative path to the file to get the overview of
|
||||
:param depth: depth up to which descendants shall be retrieved (e.g., use 1 to also retrieve immediate children).
|
||||
:param depth: depth up to which descendants shall be retrieved.
|
||||
Default (-1) results in a language specific choice: 1 for java and kotlin and 0 for other languages
|
||||
:param max_answer_chars: max characters for the result (-1 for default). If exceeded, no content/a shortened result is returned.
|
||||
:param include_file_documentation: whether to include the file's docstring. Default False.
|
||||
"""
|
||||
if depth == -1:
|
||||
if relative_path.endswith((".java", ".kt")):
|
||||
depth = 1
|
||||
else:
|
||||
depth = 0
|
||||
|
||||
relative_path = self._sanitize_input_param(relative_path)
|
||||
with JetBrainsPluginClient.from_project(self.project) as client:
|
||||
symbol_overview = client.get_symbols_overview(
|
||||
|
||||
@@ -40,20 +40,26 @@ class GetSymbolsOverviewTool(Tool, ToolMarkerSymbolicRead):
|
||||
|
||||
symbol_dict_grouper = LanguageServerSymbolDictGrouper(["kind"], ["kind"], collapse_singleton=True)
|
||||
|
||||
def apply(self, relative_path: str, depth: int = 0, max_answer_chars: int = -1) -> str:
|
||||
def apply(self, relative_path: str, depth: int = -1, max_answer_chars: int = -1) -> str:
|
||||
"""
|
||||
Use this tool to get a high-level understanding of the code symbols in a file.
|
||||
This should be the first tool to call when you want to understand a new file, unless you already know
|
||||
what you are looking for.
|
||||
|
||||
:param relative_path: the relative path to the file to get the overview of
|
||||
:param depth: depth up to which descendants of top-level symbols shall be retrieved
|
||||
(e.g. 1 retrieves immediate children). Default 0.
|
||||
:param depth: depth up to which descendants shall be retrieved.
|
||||
Default (-1) results in a language specific choice: 1 for java and kotlin and 0 for other languages
|
||||
:param max_answer_chars: if the overview is longer than this number of characters,
|
||||
no content will be returned. -1 means the default value from the config will be used.
|
||||
Don't adjust unless there is really no other way to get the content required for the task.
|
||||
:return: a JSON object containing symbols grouped by kind in a compact format.
|
||||
"""
|
||||
if depth == -1:
|
||||
if relative_path.endswith((".java", ".kt")):
|
||||
depth = 1
|
||||
else:
|
||||
depth = 0
|
||||
|
||||
result = self.get_symbol_overview(relative_path, depth=depth)
|
||||
|
||||
# capture kind names and depth-0 snapshots before grouping, which mutates the dicts
|
||||
|
||||
@@ -1,4 +1,5 @@
|
||||
import fnmatch
|
||||
import hashlib
|
||||
import logging
|
||||
import os
|
||||
import re
|
||||
@@ -528,6 +529,162 @@ class ContentReplacer:
|
||||
return updated_content
|
||||
|
||||
|
||||
@dataclass
|
||||
class ReplacementOccurrence:
|
||||
"""A single prospective replacement of a pattern match within one file."""
|
||||
|
||||
occurrence_id: str
|
||||
"""stable, content-anchored identifier: '<relative_path>:<index_in_file>@<digest>'"""
|
||||
relative_path: str
|
||||
index_in_file: int
|
||||
"""0-based index of this match among the matches within its file (in position order)"""
|
||||
start: int
|
||||
"""character offset of the match start within the file content"""
|
||||
end: int
|
||||
"""character offset of the match end within the file content"""
|
||||
matched_text: str
|
||||
replacement: str
|
||||
"""the fully expanded replacement text (backreferences already resolved)"""
|
||||
start_line: int
|
||||
"""0-based line number of the match start"""
|
||||
end_line: int
|
||||
"""0-based line number of the match end"""
|
||||
is_ambiguous: bool = False
|
||||
"""whether the pattern matches again within the matched text (possible over-match)"""
|
||||
|
||||
|
||||
class MultiFileContentReplacer:
|
||||
"""
|
||||
Occurrence-level counterpart of :class:`ContentReplacer` operating on multiple files:
|
||||
finds every match of a pattern across a set of file contents, assigns each occurrence a
|
||||
stable content-anchored id, renders minimal line diffs for previewing, and computes the
|
||||
updated content of a file for a selected subset of occurrences.
|
||||
"""
|
||||
|
||||
OCCURRENCE_ID_REGEX = re.compile(r"^(?P<path>.+):(?P<index>\d+)@(?P<digest>[0-9a-f]{6})$")
|
||||
_DIGEST_LEN = 6
|
||||
|
||||
def __init__(self, mode: Literal["literal", "regex"], regex_multiline: bool = True):
|
||||
"""
|
||||
:param mode: whether the needle is a literal string ("literal") or a regular expression ("regex")
|
||||
:param regex_multiline: whether to apply multi-line regex matching, enabling the flags re.DOTALL and re.MULTILINE
|
||||
"""
|
||||
if mode not in ("literal", "regex"):
|
||||
raise ValueError(f"Invalid mode: '{mode}', expected 'literal' or 'regex'.")
|
||||
self.mode = mode
|
||||
self._flags = (re.MULTILINE | re.DOTALL) if regex_multiline else 0
|
||||
|
||||
def _compile(self, needle: str) -> re.Pattern:
|
||||
return re.compile(re.escape(needle) if self.mode == "literal" else needle, flags=self._flags)
|
||||
|
||||
@classmethod
|
||||
def _digest(cls, matched_text: str) -> str:
|
||||
return hashlib.sha1(matched_text.encode("utf-8")).hexdigest()[: cls._DIGEST_LEN]
|
||||
|
||||
@classmethod
|
||||
def make_occurrence_id(cls, relative_path: str, index_in_file: int, matched_text: str) -> str:
|
||||
return f"{relative_path}:{index_in_file}@{cls._digest(matched_text)}"
|
||||
|
||||
@staticmethod
|
||||
def _expand_backreferences(match: re.Match, repl_template: str) -> str:
|
||||
"""Expands $!1, $!2, ... in the replacement template (same syntax as :class:`ContentReplacer`)."""
|
||||
|
||||
def expand(m: re.Match) -> str:
|
||||
group_value = match.group(int(m.group(1)))
|
||||
return group_value if group_value is not None else m.group(0)
|
||||
|
||||
return re.sub(r"\$!(\d+)", expand, repl_template)
|
||||
|
||||
def find_occurrences(self, files: list[tuple[str, str]], needle: str, repl: str) -> list[ReplacementOccurrence]:
|
||||
"""
|
||||
Finds all matches of the needle in the given files.
|
||||
|
||||
:param files: (relative_path, content) pairs; processed in the given order
|
||||
:param needle: the search expression (literal string or regex, depending on the mode)
|
||||
:param repl: the replacement template (may contain $!N backreferences in regex mode)
|
||||
:return: occurrences in deterministic order (file order, then position within the file)
|
||||
"""
|
||||
pattern = self._compile(needle)
|
||||
occurrences: list[ReplacementOccurrence] = []
|
||||
for relative_path, content in files:
|
||||
for index_in_file, match in enumerate(pattern.finditer(content)):
|
||||
matched_text = match.group(0)
|
||||
replacement = self._expand_backreferences(match, repl) if self.mode == "regex" else repl
|
||||
# same over-match heuristic as ContentReplacer: for a multi-line match, the pattern
|
||||
# matching again within the matched text indicates the match may have swallowed
|
||||
# more than intended
|
||||
is_ambiguous = "\n" in matched_text and pattern.search(matched_text[1:]) is not None
|
||||
occurrences.append(
|
||||
ReplacementOccurrence(
|
||||
occurrence_id=self.make_occurrence_id(relative_path, index_in_file, matched_text),
|
||||
relative_path=relative_path,
|
||||
index_in_file=index_in_file,
|
||||
start=match.start(),
|
||||
end=match.end(),
|
||||
matched_text=matched_text,
|
||||
replacement=replacement,
|
||||
start_line=content.count("\n", 0, match.start()),
|
||||
end_line=content.count("\n", 0, match.end()),
|
||||
is_ambiguous=is_ambiguous,
|
||||
)
|
||||
)
|
||||
return occurrences
|
||||
|
||||
@staticmethod
|
||||
def apply_to_content(content: str, occurrences: list[ReplacementOccurrence]) -> str:
|
||||
"""
|
||||
Applies the given occurrences (which must have been derived from exactly this content)
|
||||
and returns the updated content.
|
||||
"""
|
||||
for occ in sorted(occurrences, key=lambda o: o.start, reverse=True):
|
||||
assert content[occ.start : occ.end] == occ.matched_text, (
|
||||
f"Occurrence {occ.occurrence_id} does not match the content it is being applied to"
|
||||
)
|
||||
content = content[: occ.start] + occ.replacement + content[occ.end :]
|
||||
return content
|
||||
|
||||
@staticmethod
|
||||
def _format_block(block: str, prefix: str, max_lines: int, max_line_chars: int) -> list[str]:
|
||||
lines = block.split("\n")
|
||||
shown = lines[:max_lines]
|
||||
result = []
|
||||
for line in shown:
|
||||
if len(line) > max_line_chars:
|
||||
line = line[:max_line_chars] + f"… (+{len(line) - max_line_chars} chars)"
|
||||
result.append(f" {prefix} {line}")
|
||||
if len(lines) > max_lines:
|
||||
result.append(f" {prefix} … ({len(lines) - max_lines} more lines)")
|
||||
return result
|
||||
|
||||
def render_occurrence_diff(
|
||||
self, occ: ReplacementOccurrence, content: str, max_lines_per_side: int = 6, max_line_chars: int = 200
|
||||
) -> str:
|
||||
"""
|
||||
Renders a minimal line diff for the occurrence: the full lines spanned by the match,
|
||||
before and after the replacement.
|
||||
|
||||
:param occ: the occurrence (must have been derived from exactly this content)
|
||||
:param content: the file content the occurrence was found in
|
||||
:param max_lines_per_side: cap on the number of displayed lines per diff side
|
||||
:param max_line_chars: cap on the number of displayed characters per line
|
||||
:return: the rendered diff
|
||||
"""
|
||||
line_start = content.rfind("\n", 0, occ.start) + 1
|
||||
line_end = content.find("\n", occ.end)
|
||||
if line_end == -1:
|
||||
line_end = len(content)
|
||||
old_block = content[line_start:line_end]
|
||||
new_block = content[line_start : occ.start] + occ.replacement + content[occ.end : line_end]
|
||||
location = f"line {occ.start_line}" if occ.start_line == occ.end_line else f"lines {occ.start_line}-{occ.end_line}"
|
||||
header = f" [{occ.occurrence_id}] {location}"
|
||||
if occ.is_ambiguous:
|
||||
header += " (WARNING: the pattern matches again inside this match — possible over-match, verify the diff)"
|
||||
diff_lines = [header]
|
||||
diff_lines += self._format_block(old_block, "-", max_lines_per_side, max_line_chars)
|
||||
diff_lines += self._format_block(new_block, "+", max_lines_per_side, max_line_chars)
|
||||
return "\n".join(diff_lines)
|
||||
|
||||
|
||||
@dataclass
|
||||
class TextCoords:
|
||||
line: int
|
||||
|
||||
+39
-3
@@ -1,5 +1,7 @@
|
||||
import logging
|
||||
import os
|
||||
import tempfile
|
||||
import time
|
||||
from collections.abc import Sequence
|
||||
from enum import Enum
|
||||
from typing import Any
|
||||
@@ -199,9 +201,43 @@ def normalise_yaml_comments(commented_map: CommentedMap, comment_normalisation:
|
||||
|
||||
def save_yaml(path: str, data: dict | CommentedMap, preserve_comments: bool = True) -> None:
|
||||
yaml = _create_yaml(preserve_comments)
|
||||
os.makedirs(os.path.dirname(path), exist_ok=True)
|
||||
with open(path, "w", encoding=SERENA_FILE_ENCODING) as f:
|
||||
yaml.dump(data, f)
|
||||
target_dir = os.path.dirname(path)
|
||||
os.makedirs(target_dir, exist_ok=True)
|
||||
# Atomic write: dump to a temp file in the SAME directory, then os.replace onto the target.
|
||||
# A plain truncate-and-write (open(path, "w")) is NOT atomic: a concurrent writer (e.g. a second
|
||||
# Serena process updating the auto-managed registered-projects list) or an interrupted write can
|
||||
# leave the file half-overwritten — writing a shorter value over a longer one leaves a stale tail,
|
||||
# which corrupts the YAML so it no longer parses and every later load fails. temp + os.replace makes
|
||||
# each write all-or-nothing (last-writer-wins, never a corrupt interleave).
|
||||
fd, tmp = tempfile.mkstemp(dir=target_dir, prefix=os.path.basename(path) + ".", suffix=".tmp")
|
||||
try:
|
||||
with os.fdopen(fd, "w", encoding=SERENA_FILE_ENCODING) as f:
|
||||
yaml.dump(data, f)
|
||||
_replace_with_retry(tmp, path)
|
||||
except BaseException:
|
||||
try:
|
||||
os.unlink(tmp)
|
||||
except OSError:
|
||||
pass
|
||||
raise
|
||||
|
||||
|
||||
def _replace_with_retry(src: str, dst: str, *, attempts: int = 10, delay_s: float = 0.05) -> None:
|
||||
"""``os.replace(src, dst)`` with a short retry on a Windows sharing violation.
|
||||
|
||||
On Windows the atomic rename fails with ``PermissionError`` (WinError 5/32) if another process
|
||||
momentarily holds ``dst`` open — e.g. a second Serena process reading or replacing the same
|
||||
config. A brief bounded retry rides out that contention; the temp file is still complete, so we
|
||||
never fall back to a non-atomic write.
|
||||
"""
|
||||
for attempt in range(attempts):
|
||||
try:
|
||||
os.replace(src, dst)
|
||||
return
|
||||
except PermissionError:
|
||||
if attempt == attempts - 1:
|
||||
raise
|
||||
time.sleep(delay_s)
|
||||
|
||||
|
||||
def yaml_comment_entry_is_empty(comment_entry: Any) -> bool:
|
||||
|
||||
@@ -269,6 +269,40 @@ def _write(manager: MemoryManager, name: str, content: str) -> None:
|
||||
manager.save_memory(name, content, is_tool_context=False)
|
||||
|
||||
|
||||
class TestListMemoriesFollowsSymlinks:
|
||||
"""Regression: memories reachable only through a directory symlink (e.g. a monorepo whose
|
||||
``.serena/memories`` symlinks each submodule's memory folder, making them addressable as
|
||||
``<submodule>/<name>``) must be listed and readable. ``Path.rglob`` does not descend into
|
||||
symlinked directories before Python 3.13, which silently hid them from ``list_memories``.
|
||||
"""
|
||||
|
||||
def test_lists_and_reads_memories_behind_directory_symlink(self, tmp_path) -> None:
|
||||
project_dir = tmp_path / "project" / ".serena"
|
||||
manager = MemoryManager(serena_data_folder=project_dir)
|
||||
_write(manager, "own_memory", "# local")
|
||||
|
||||
# a sibling submodule with its own (possibly nested) memories
|
||||
submodule_memories = tmp_path / "submodule" / ".serena" / "memories"
|
||||
(submodule_memories / "nested").mkdir(parents=True)
|
||||
(submodule_memories / "sub_a.md").write_text("# a", encoding="utf-8")
|
||||
(submodule_memories / "nested" / "sub_b.md").write_text("# b", encoding="utf-8")
|
||||
|
||||
# symlink the submodule's memory dir into the project's memories folder
|
||||
link = project_dir / "memories" / "submodule"
|
||||
try:
|
||||
link.symlink_to(submodule_memories, target_is_directory=True)
|
||||
except (OSError, NotImplementedError) as e:
|
||||
pytest.skip(f"cannot create directory symlinks on this platform/permissions: {e}")
|
||||
|
||||
listed = manager.list_memories().get_full_list()
|
||||
assert "own_memory" in listed
|
||||
assert "submodule/sub_a" in listed
|
||||
assert "submodule/nested/sub_b" in listed
|
||||
# the listed names must round-trip through the read path (which resolves the symlink)
|
||||
assert manager.load_memory("submodule/sub_a") == "# a"
|
||||
assert manager.load_memory("submodule/nested/sub_b") == "# b"
|
||||
|
||||
|
||||
class TestValidateReferentialIntegrity:
|
||||
def test_clean_report_when_all_references_resolve(self, fs_manager: MemoryManager) -> None:
|
||||
_write(fs_manager, "auth", "# auth notes")
|
||||
|
||||
@@ -26,6 +26,7 @@ from serena.tools import (
|
||||
GetDiagnosticsForFileTool,
|
||||
InitialInstructionsTool,
|
||||
ReplaceContentTool,
|
||||
ReplaceInFilesTool,
|
||||
ReplaceSymbolBodyTool,
|
||||
SafeDeleteSymbol,
|
||||
Tool,
|
||||
@@ -1166,6 +1167,57 @@ class TestSerenaAgent:
|
||||
new_content = read_project_file(serena_agent.get_active_project(), relative_path)
|
||||
assert repl in new_content
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"serena_agent",
|
||||
[
|
||||
pytest.param(Language.PYTHON, marks=get_pytest_markers(Language.PYTHON), id="python_replace_in_files"),
|
||||
],
|
||||
indirect=["serena_agent"],
|
||||
)
|
||||
def test_replace_in_files_dry_run_then_selective_apply(self, serena_agent: SerenaAgent):
|
||||
"""A dry run lists every occurrence as a diff with an id and modifies nothing; a follow-up call
|
||||
restricted to one of the ids replaces exactly that occurrence.
|
||||
"""
|
||||
relative_path = os.path.join("test_repo", "models.py")
|
||||
needle = "name: str | None = None"
|
||||
repl = "name: str | None = MARKER_DEFAULT"
|
||||
tool = serena_agent.get_tool(ReplaceInFilesTool)
|
||||
with project_file_modification_context(serena_agent, relative_path):
|
||||
original_content = read_project_file(serena_agent.get_active_project(), relative_path)
|
||||
assert original_content.count(needle) >= 2
|
||||
|
||||
listing = tool.apply(needle=needle, repl=repl, mode="literal", relative_path=relative_path, dry_run=True)
|
||||
assert "DRY RUN" in listing
|
||||
occurrence_ids = re.findall(r"\[([^\[\]]+:\d+@[0-9a-f]{6})\]", listing)
|
||||
assert len(occurrence_ids) == original_content.count(needle)
|
||||
assert read_project_file(serena_agent.get_active_project(), relative_path) == original_content
|
||||
|
||||
result = tool.apply(needle=needle, repl=repl, mode="literal", relative_path=relative_path, occurrence_ids=[occurrence_ids[0]])
|
||||
assert "Replaced 1 occurrence(s) in 1 file(s)" in result
|
||||
new_content = read_project_file(serena_agent.get_active_project(), relative_path)
|
||||
assert new_content.count("MARKER_DEFAULT") == 1
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"serena_agent",
|
||||
[
|
||||
pytest.param(Language.PYTHON, marks=get_pytest_markers(Language.PYTHON), id="python_replace_in_files_guard"),
|
||||
],
|
||||
indirect=["serena_agent"],
|
||||
)
|
||||
def test_replace_in_files_expected_count_guard_returns_listing(self, serena_agent: SerenaAgent):
|
||||
"""A blind call with a wrong expected_count must change nothing and return the prospective
|
||||
changes (the failed guard doubles as a dry run).
|
||||
"""
|
||||
relative_path = os.path.join("test_repo", "models.py")
|
||||
needle = "name: str | None = None"
|
||||
tool = serena_agent.get_tool(ReplaceInFilesTool)
|
||||
with project_file_modification_context(serena_agent, relative_path):
|
||||
original_content = read_project_file(serena_agent.get_active_project(), relative_path)
|
||||
with pytest.raises(ValueError, match="NO changes were applied") as exc_info:
|
||||
tool.apply(needle=needle, repl="X", mode="literal", relative_path=relative_path, expected_count=1)
|
||||
assert re.search(r"\[[^\[\]]+:\d+@[0-9a-f]{6}\]", str(exc_info.value)) # the listing with ids is included
|
||||
assert read_project_file(serena_agent.get_active_project(), relative_path) == original_content
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"serena_agent",
|
||||
[
|
||||
|
||||
@@ -2,7 +2,7 @@ import re
|
||||
|
||||
import pytest
|
||||
|
||||
from serena.util.text_utils import LineType, search_files, search_text
|
||||
from serena.util.text_utils import LineType, MultiFileContentReplacer, search_files, search_text
|
||||
|
||||
|
||||
class TestSearchText:
|
||||
@@ -575,3 +575,65 @@ class TestExpandBraces:
|
||||
from serena.util.text_utils import expand_braces
|
||||
|
||||
assert sorted(expand_braces(pattern)) == sorted(expected)
|
||||
|
||||
|
||||
class TestMultiFileContentReplacer:
|
||||
FILES = [
|
||||
("a/first.py", "import old_pkg\n\nvalue = old_pkg.compute()\n"),
|
||||
("b/second.py", "import old_pkg\nother = 1\n"),
|
||||
]
|
||||
|
||||
def test_find_occurrences_order_ids_and_lines(self):
|
||||
replacer = MultiFileContentReplacer(mode="literal")
|
||||
occurrences = replacer.find_occurrences(self.FILES, "old_pkg", "new_pkg")
|
||||
assert [o.relative_path for o in occurrences] == ["a/first.py", "a/first.py", "b/second.py"]
|
||||
assert [o.index_in_file for o in occurrences] == [0, 1, 0]
|
||||
assert [o.start_line for o in occurrences] == [0, 2, 0]
|
||||
for o in occurrences:
|
||||
assert o.matched_text == "old_pkg"
|
||||
assert o.replacement == "new_pkg"
|
||||
assert MultiFileContentReplacer.OCCURRENCE_ID_REGEX.match(o.occurrence_id)
|
||||
# ids are content-anchored: same matched text at the same index yields the same id across calls
|
||||
again = replacer.find_occurrences(self.FILES, "old_pkg", "new_pkg")
|
||||
assert [o.occurrence_id for o in again] == [o.occurrence_id for o in occurrences]
|
||||
|
||||
def test_regex_backreference_expansion(self):
|
||||
replacer = MultiFileContentReplacer(mode="regex")
|
||||
files = [("f.txt", "name=alpha\nname=beta\n")]
|
||||
occurrences = replacer.find_occurrences(files, r"name=(\w+)", "id=$!1")
|
||||
assert [o.replacement for o in occurrences] == ["id=alpha", "id=beta"]
|
||||
|
||||
def test_apply_to_content_selected_subset(self):
|
||||
replacer = MultiFileContentReplacer(mode="literal")
|
||||
path, content = self.FILES[0]
|
||||
occurrences = [o for o in replacer.find_occurrences(self.FILES, "old_pkg", "new_pkg") if o.relative_path == path]
|
||||
updated = replacer.apply_to_content(content, occurrences[1:]) # only the second occurrence
|
||||
assert updated == "import old_pkg\n\nvalue = new_pkg.compute()\n"
|
||||
updated_all = replacer.apply_to_content(content, occurrences)
|
||||
assert updated_all == "import new_pkg\n\nvalue = new_pkg.compute()\n"
|
||||
|
||||
def test_ambiguous_multiline_match_is_flagged(self):
|
||||
replacer = MultiFileContentReplacer(mode="regex")
|
||||
files = [("f.txt", "start A\nstart B\nend\n")]
|
||||
occurrences = replacer.find_occurrences(files, r"start.*?end", "X")
|
||||
assert len(occurrences) == 1
|
||||
assert occurrences[0].is_ambiguous
|
||||
|
||||
def test_render_occurrence_diff_minimal_lines(self):
|
||||
replacer = MultiFileContentReplacer(mode="literal")
|
||||
path, content = self.FILES[0]
|
||||
occ = replacer.find_occurrences([(path, content)], "old_pkg.compute()", "new_pkg.compute_all()")[0]
|
||||
diff = replacer.render_occurrence_diff(occ, content)
|
||||
assert occ.occurrence_id in diff
|
||||
assert "line 2" in diff
|
||||
assert " - value = old_pkg.compute()" in diff
|
||||
assert " + value = new_pkg.compute_all()" in diff
|
||||
# only the affected line is shown, not the whole file
|
||||
assert "import old_pkg" not in diff
|
||||
|
||||
def test_apply_to_content_rejects_drifted_occurrence(self):
|
||||
replacer = MultiFileContentReplacer(mode="literal")
|
||||
path, content = self.FILES[0]
|
||||
occ = replacer.find_occurrences([(path, content)], "old_pkg", "new_pkg")[0]
|
||||
with pytest.raises(AssertionError):
|
||||
replacer.apply_to_content("completely different content", [occ])
|
||||
@@ -0,0 +1,80 @@
|
||||
"""Tests for serena.util.yaml — focused on the atomic-write guarantee of ``save_yaml``.
|
||||
|
||||
Regression: ``~/.serena/serena_config.yml`` was corrupted in the field when a non-atomic
|
||||
truncate-and-write saved a SHORTER value over a longer existing file (or two Serena processes
|
||||
saved concurrently), leaving a stale tail (e.g. a dangling ``ena`` line) that made the YAML
|
||||
unparsable, which wedged every later load. ``save_yaml`` now writes to a temp file and
|
||||
``os.replace``s it onto the target, so each write is all-or-nothing.
|
||||
"""
|
||||
|
||||
import os
|
||||
import threading
|
||||
|
||||
from ruamel.yaml import YAML
|
||||
|
||||
from serena.util.yaml import load_yaml, save_yaml
|
||||
|
||||
|
||||
def _read_text(path: str) -> str:
|
||||
with open(path, encoding="utf-8") as f:
|
||||
return f.read()
|
||||
|
||||
|
||||
def test_save_yaml_roundtrips(tmp_path):
|
||||
path = str(tmp_path / "config.yml")
|
||||
save_yaml(path, {"projects": ["a", "b", "c"], "scalar": 1})
|
||||
loaded = load_yaml(path)
|
||||
assert list(loaded["projects"]) == ["a", "b", "c"]
|
||||
assert loaded["scalar"] == 1
|
||||
|
||||
|
||||
def test_shorter_write_over_longer_leaves_no_stale_tail(tmp_path):
|
||||
"""The exact field corruption: a long file overwritten by a much shorter one.
|
||||
|
||||
A plain ``open(path, "w")`` truncates first, so this would pass even when buggy; the point
|
||||
of the assertion is that the final file parses and contains EXACTLY the new content — no
|
||||
leftover bytes from the longer previous version.
|
||||
"""
|
||||
path = str(tmp_path / "config.yml")
|
||||
long_projects = [f"C:/Users/misch/Projects/some/really/long/path/number/{i}/serena" for i in range(50)]
|
||||
save_yaml(path, {"projects": long_projects})
|
||||
|
||||
save_yaml(path, {"projects": ["C:/Users/misch/Projects/oraios/serena"]})
|
||||
|
||||
loaded = load_yaml(path)
|
||||
assert list(loaded["projects"]) == ["C:/Users/misch/Projects/oraios/serena"]
|
||||
# no stale tail from the longer write survived
|
||||
assert "number/49" not in _read_text(path)
|
||||
# and it is still valid YAML on a fresh strict parse
|
||||
with open(path, encoding="utf-8") as f:
|
||||
YAML().load(f)
|
||||
|
||||
|
||||
def test_no_tmp_files_left_behind(tmp_path):
|
||||
path = str(tmp_path / "config.yml")
|
||||
save_yaml(path, {"projects": ["a"]})
|
||||
leftovers = [p for p in os.listdir(tmp_path) if p != "config.yml"]
|
||||
assert leftovers == [], f"temp files not cleaned up: {leftovers}"
|
||||
|
||||
|
||||
def test_concurrent_saves_never_corrupt(tmp_path):
|
||||
"""Many concurrent writers must yield a file that always parses (last-writer-wins is fine;
|
||||
a corrupt interleave is not).
|
||||
"""
|
||||
path = str(tmp_path / "config.yml")
|
||||
save_yaml(path, {"projects": ["seed"]})
|
||||
|
||||
def writer(n: int) -> None:
|
||||
for _ in range(20):
|
||||
save_yaml(path, {"projects": [f"p{n}-{k}" for k in range(n + 1)]})
|
||||
|
||||
threads = [threading.Thread(target=writer, args=(n,)) for n in range(6)]
|
||||
for t in threads:
|
||||
t.start()
|
||||
for t in threads:
|
||||
t.join()
|
||||
|
||||
# whatever won, the file must be complete + parseable (no corruption)
|
||||
loaded = load_yaml(path)
|
||||
assert "projects" in loaded
|
||||
assert all(isinstance(p, str) for p in loaded["projects"])
|
||||
Reference in new issue
Block a user