Merge pull request #1477 from oraios/feature/rename-memory-references

Revamp Memory System
This commit is contained in:
Dr. Dominik Jain authored and GitHub committed 2026-05-18 14:04:29 +02:00
commit 9eef1cba36
30 files changed
+2597 -603

No files matched your search

+26
View File
@@ -0,0 +1,26 @@
# Conventions
## Style (project-instructed)
- Idiomatic, object-oriented design. Non-trivial interfaces use **explicitly typed abstractions** (strategy pattern etc.) rather than bare functions/callbacks.
- Avoid low-level data structures where an OO abstraction fits. For simple data containers use **dataclasses**, not dicts/tuples.
- Structure function bodies into **functional blocks separated by blank lines**, each prefixed with a short elliptical phrase (lowercase, no leading capital) describing the block's purpose.
- **Docstrings: reStructuredText.** Param/return/raises use `:param x:`, `:return:`, `:raises X:`.
- Parameter / method / class descriptions begin with a precise elliptical phrase defining *what* the thing is; details in subsequent sentences.
## Formatting / lint (ruff)
- Line length 140, double quotes, target `py311`.
- Many "annoying" rules are disabled — see `[tool.ruff.lint] ignore` in `pyproject.toml` before adding workarounds (e.g. `Optional[T]` is preferred over `T | None`, `Union` is allowed, relative imports forbidden, `% string formatting` allowed).
- `ruff format` runs on `src scripts test`; same set for `ruff check`.
- mccabe complexity cap: 20.
## Typing (mypy strict)
- `disallow_untyped_defs = true`, `disallow_incomplete_defs = true`, `strict_equality`, `strict_optional`, `warn_unreachable`, `no_implicit_optional`.
- Excluded paths: `^build/`, `^docs/`, `^test/resources/`.
- Test files relax `disallow_untyped_defs` (still type-checked otherwise).
## Tests
- Language-server tests are pytest-marker-gated (one marker per language; see `pyproject.toml` `[tool.pytest.ini_options].markers`). Default `poe test` runs unmarked tests + whatever `PYTEST_MARKERS` selects.
- Snapshot tests use **syrupy** with custom `--snapshot-patch-pycharm-diff` plugin (auto-added via `addopts`).
## Memories
- Follow `mem:memory_maintenance` for any new/updated memory in `.serena/memories/`.
+29
View File
@@ -0,0 +1,29 @@
# Serena — Project Core
Serena is an MCP-based "IDE for coding agents": semantic code retrieval/editing/refactoring tools driven by language servers.
## Source map
- `src/serena/` — agent, MCP server, tools, project/config layer
- `agent.py`, `mcp.py`, `project_server.py`, `cli.py`, `hooks.py` — entrypoints/wiring
- `tools/` — tool implementations (memory_tools, symbol_tools, file_tools, workflow_tools, query_project_tools, config_tools, cmd_tools, jetbrains_tools)
- `tools/tools_base.py` — base classes for all tools
- `config/` — `serena_config.py`, `context_mode.py`, `client_setup.py`
- `resources/config/contexts/*.yml`, `resources/config/modes/*.yml` — context/mode definitions
- `code_editor.py`, `symbol.py`, `ls_manager.py` — symbolic editing / LS lifecycle
- `dashboard.py`, `gui_log_viewer.py` — web dashboard / log viewer
- `prompt_factory.py` + `generated/generated_prompt_factory.py` — prompts (regenerate with `scripts/gen_prompt_factory.py`)
- `src/solidlsp/` — LSP client framework; per-language servers under `language_servers/`
- `src/interprompt/` — prompt template library (synced from external repo; see `.syncCommitId.*`)
- `test/serena/`, `test/solidlsp/<lang>/` — pytest suites; per-language tests gated by pytest markers
- `test/resources/repos/<lang>/` — fixture projects used by language-server tests
- `scripts/` — utilities (prompt regen, tool overview, profiling, agno agent)
- `docs/` — Jupyter Book sources; build via poe `doc-build`
## Project-wide invariants
- Package name (PyPI): `serena-agent`; import root: `serena`. Wheel includes `serena`, `interprompt`, `solidlsp`.
- Python: `>=3.11, <3.15`. Dependencies are exact-pinned in `pyproject.toml` (uvx installs from git, lockfile ignored — pin exactly).
- Entry points: `serena` → `serena.cli:top_level`; `serena-hooks` → `serena.hooks:hook_commands`.
- Per-project state lives under `<project>/.serena/` (config + `memories/` as `.md` files).
- See `mem:tech_stack`, `mem:suggested_commands`, `mem:conventions`, `mem:task_completion`.
+33
View File
@@ -0,0 +1,33 @@
# Memory Maintenance
## Discovery Model
- Core principle: progressive discovery through references, building a graph of memories.
- Initially, agents are provided with the list of all memories (names only).
- Agents should read `mem:core` as the top-level entry point (graph root).
This memory should contain references to other memories covering major project domains.
The referenced memories shall, in turn, shall contain references to even more specific memories, and so on.
The depth of the graph shall depend on the project complexity.
- Use topics/folders to group related memories in order to make the content structure explicit.
Folders can mirror project structure (e.g. modules like frontend/backend) or topics like debugging, architecture, etc.
- Memory references must use a mem: prefix inside backticks, e.g. `mem:frontend/core`.
The surrounding text should clearly indicate when to read the memory/which content to expect.
The text should provide more precise guidance than the memory name alone,
i.e. avoid a reference like "frontend debugging: `mem:frontend/debugging` and instead make clear which aspects of frontend debugging are covered.
- Memories themselves should not contain information about when to read them; this is the responsibility of the referring memory.
## Style
Dense agent notes, not prose docs. Prefer invariants, terse bullets.
Avoid obvious context, rationale, and examples unless they prevent likely mistakes.
Keep guidance durable and generalizable, not task-local.
## Add/update threshold
Add or update memories only with stable, non-obvious project conventions that avoid complex rediscovery in the future.
Do not add: quick-read facts; generic language/framework knowledge; one-off task notes; volatile line-level details; behavior likely to change soon.
## Maintenance Actions
- Renaming memories: References are updated automatically if handled via Serena's memory rename tool.
- Checking for stale memories (e.g. after deletion): Call `serena memories check` for a report.
@@ -1,175 +0,0 @@
# Serena Core Concepts and Architecture
## High-Level Architecture
Serena is built around a dual-layer architecture:
1. **SerenaAgent** - The main orchestrator that manages projects, tools, and user interactions
2. **SolidLanguageServer** - A unified wrapper around Language Server Protocol (LSP) implementations
## Core Components
### 1. SerenaAgent (`src/serena/agent.py`)
The central coordinator that:
- Manages active projects and their configurations
- Coordinates between different tools and contexts
- Handles language server lifecycle
- Manages memory persistence
- Provides MCP (Model Context Protocol) server interface
Key responsibilities:
- **Project Management** - Activating, switching between projects
- **Tool Registry** - Loading and managing available tools based on context/mode
- **Language Server Integration** - Starting/stopping language servers per project
- **Memory Management** - Persistent storage of project knowledge
- **Task Execution** - Coordinating complex multi-step operations
### 2. SolidLanguageServer (`src/solidlsp/ls.py`)
A unified abstraction over multiple language servers that provides:
- **Language-agnostic interface** for symbol operations
- **Caching layer** for performance optimization
- **Error handling and recovery** for unreliable language servers
- **Uniform API** regardless of underlying LSP implementation
Core capabilities:
- Symbol discovery and navigation
- Code completion and hover information
- Find references and definitions
- Document and workspace symbol search
- File watching and change notifications
### 3. Tool System (`src/serena/tools/`)
Modular tool architecture with several categories:
#### File Tools (`file_tools.py`)
- File system operations (read, write, list directories)
- Text search and pattern matching
- Regex-based replacements
#### Symbol Tools (`symbol_tools.py`)
- Language-aware symbol finding and navigation
- Symbol body replacement and insertion
- Reference finding across codebase
#### Memory Tools (`memory_tools.py`)
- Project knowledge persistence
- Memory retrieval and management
- Onboarding information storage
#### Configuration Tools (`config_tools.py`)
- Project activation and switching
- Mode and context management
- Tool inclusion/exclusion
### 4. Configuration System (`src/serena/config/`)
Multi-layered configuration supporting:
- **Contexts** - Define available tools and their behavior
- **Modes** - Specify operational patterns (interactive, editing, etc.)
- **Projects** - Per-project settings and language server configs
- **Tool Sets** - Grouped tool collections for different use cases
## Language Server Integration
### Language Support Model
Each supported language has:
1. **Language Server Implementation** (`src/solidlsp/language_servers/`)
2. **Runtime Dependencies** - Managed downloads of language servers
3. **Test Repository** (`test/resources/repos/<language>/`)
4. **Test Suite** (`test/solidlsp/<language>/`)
### Language Server Lifecycle
1. **Discovery** - Find language servers or download them automatically
2. **Initialization** - Start server process and perform LSP handshake
3. **Project Setup** - Open workspace and configure language-specific settings
4. **Operation** - Handle requests/responses with caching and error recovery
5. **Shutdown** - Clean shutdown of server processes
### Supported Languages
Current language support includes:
- **C#** - Microsoft.CodeAnalysis.LanguageServer (.NET 9)
- **Python** - Pyright or Jedi
- **TypeScript/JavaScript** - TypeScript Language Server
- **Rust** - rust-analyzer
- **Go** - gopls
- **Java** - Eclipse JDT Language Server
- **Kotlin** - Kotlin Language Server
- **PHP** - Intelephense
- **Ruby** - Solargraph
- **Clojure** - clojure-lsp
- **Elixir** - ElixirLS
- **Dart** - Dart Language Server
- **C/C++** - clangd
- **Terraform** - terraform-ls
## Memory and Knowledge Management
### Memory System
- **Markdown-based storage** in `.serena/memories/` directory
- **Contextual retrieval** - memories loaded based on relevance
- **Project-specific** knowledge persistence
- **Onboarding support** - guided setup for new projects
### Knowledge Categories
- **Project Structure** - Directory layouts, build systems
- **Architecture Patterns** - How the codebase is organized
- **Development Workflows** - Testing, building, deployment
- **Domain Knowledge** - Business logic and requirements
## MCP Server Interface
Serena exposes its functionality through Model Context Protocol:
- **Tool Discovery** - AI agents can enumerate available tools
- **Context-Aware Operations** - Tools behave based on active project/mode
- **Stateful Sessions** - Maintains project state across interactions
- **Error Handling** - Graceful degradation when tools fail
## Error Handling and Resilience
### Language Server Reliability
- **Timeout Management** - Configurable timeouts for LSP requests
- **Process Recovery** - Automatic restart of crashed language servers
- **Fallback Behavior** - Graceful degradation when LSP unavailable
- **Caching Strategy** - Reduces impact of server failures
### Project Activation Safety
- **Validation** - Verify project structure before activation
- **Error Isolation** - Project failures don't affect other projects
- **Recovery Mechanisms** - Automatic cleanup and retry logic
## Performance Considerations
### Caching Strategy
- **Symbol Cache** - In-memory caching of expensive symbol operations
- **File System Cache** - Reduced disk I/O for repeated operations
- **Language Server Cache** - Persistent cache across sessions
### Resource Management
- **Language Server Pooling** - Reuse servers across projects when possible
- **Memory Management** - Automatic cleanup of unused resources
- **Background Operations** - Async operations don't block user interactions
## Extension Points
### Adding New Languages
1. Implement language server class in `src/solidlsp/language_servers/`
2. Add runtime dependencies configuration
3. Create test repository and test suite
4. Update language enumeration and configuration
### Adding New Tools
1. Inherit from `Tool` base class in `tools_base.py`
2. Implement required methods and parameter validation
3. Register tool in appropriate tool registry
4. Add to context/mode configurations as needed
### Custom Contexts and Modes
- Define new contexts in YAML configuration files
- Specify tool sets and operational patterns
- Configure for specific development workflows
+19 -21
View File
@@ -1,26 +1,24 @@
# Suggested Commands
## Development Tasks (using uv and poe)
Run via `uv run poe <task>` (or `poe <task>` inside the activated venv). Poe executor is `simple`, so plain `poe` works without uv re-resolving.
The following tasks should generally be executed using `uv run poe <task_name>`.
## Dev loop
- `poe test` — pytest on `test/` (per-language tests are marker-gated; pass `-m <marker>` to enable).
- `poe lint` — ruff format-check + ruff check (no fixes).
- `poe format` — ruff `--fix` then `ruff format` (mutates files).
- `poe type-check` — mypy on `src/serena`, `src/solidlsp`, and `test/` (test pass disables `no-untyped-def`).
- Single test file: `uv run pytest test/path/to/test_x.py -vv`. Language-gated: add `-m python` etc.
- `format`: This is the **only** allowed command for formatting. Run as `uv run poe format`.
- `type-check`: This is the **only** allowed command for type checking. Run as `uv run poe type-check`.
- `test`: This is the preferred command for running tests (`uv run poe test [args]`). You can select subsets of tests with markers,
the current markers are
```toml
markers = [
"python: language server running for Python",
"go: language server running for Go",
"java: language server running for Java",
"rust: language server running for Rust",
"typescript: language server running for TypeScript",
"php: language server running for PHP",
"snapshot: snapshot tests for symbolic editing operations",
]
```
By default, `uv run poe test` uses the markers set in the env var `PYTEST_MARKERS`, or, if it unset, uses `-m "not java and not rust and not isolated process"`.
You can override this behavior by simply passing the `-m` option to `uv run poe test`, e.g. `uv run poe test -m "python or go"`.
## Docs
- `poe doc-build` — clean + autogen + sphinx (uses `rm -rf`; needs a unix-like shell, e.g. Git Bash on Windows).
For finishing a task, make sure format, type-check and test pass! Run them at the end of the task
and if needed fix any issues that come up and run them again until they pass.
## Entrypoints
- `uv run serena ...` — main CLI (`serena.cli:top_level`).
- `uv run serena-hooks ...` — hook helpers.
- `python scripts/gen_prompt_factory.py` — regenerate `src/serena/generated/generated_prompt_factory.py` after editing prompt templates.
## Windows shell notes (PowerShell 7+ is the project shell)
- Use `Remove-Item -Recurse -Force <path>` instead of `rm -rf` (the `doc-clean` poe task uses unix `rm -rf` and requires bash).
- Env vars: `$env:NAME = 'value'` (not `export`).
- Path separator in `pyproject.toml` poe tasks uses forward slashes; PowerShell accepts them in arguments.
- `git`, `uv`, `poe`, `pytest` behave identically to unix.
+11
View File
@@ -0,0 +1,11 @@
# Task Completion Checklist
After any code change in `src/` or `test/`, run:
1. `uv run poe format` — applies ruff fixes + formatting (mutates files).
2. `uv run poe type-check` — mypy strict on `src/serena`, `src/solidlsp`, `test/`.
3. `uv run poe test` — pytest on affected files or affected languages, using `-m` markers.
If prompt templates changed: `uv run python scripts/gen_prompt_factory.py` (regenerates `src/serena/generated/generated_prompt_factory.py`; use `uv run poe format` and commit the result).
If memories were edited/renamed/split: run `uv run serena memories check` from the project root to find broken `mem:` references.
+10
View File
@@ -0,0 +1,10 @@
# Tech Stack
- Language: Python 3.11–3.14 (`requires-python = ">=3.11, <3.15"`).
- Package/dep manager: **uv** (uv.lock present). Exact version pins in `pyproject.toml` because `uvx` installs from git and ignores the lockfile.
- Build backend: `hatchling`. Packages: `src/serena`, `src/interprompt`, `src/solidlsp`.
- Task runner: **poethepoet** (`poe <task>`). Poe executor is `simple` (does NOT shell out via uv) — avoids env recreation while MCP server is running.
- Key runtime deps: `mcp`, `flask` (dashboard), `pydantic`, `pygls` + `lsprotocol` (LSP), `anthropic`, `jinja2`, `ruamel.yaml`, `pywebview`/`pystray` (GUI), `pythonnet` (Windows only).
- Dev deps: `ruff` (lint+format), `mypy` (strict), `pytest` (+ `pytest-xdist`, `pytest-timeout`, `syrupy` snapshots), `sphinx`/`jupyter-book` for docs.
- Optional extras: `agno` (Agno agent integration), `google` (gemini).
- LSP client core lives under `src/solidlsp/`; one subdir per supported language server under `language_servers/`.
+17
View File
@@ -14,6 +14,23 @@ Status of the `main` branch. Changes prior to the next official version change w
- Modal markup cleanup: extract shared CSS classes (`.modal-info`, `.modal-hint`, `.modal-prompt`, `.modal-field`, `.modal-input`, `.modal-select`, `.modal-textarea`, `.modal-actions`, `.btn-secondary`) and remove duplicated inline styles from all seven modals. Inputs and textareas get an accent-colored focus ring; the modal backdrop has a subtle blur.
- Add Language: replace the native `<select>` with a filterable combobox — type to filter, keyboard navigation (Up/Down/Enter/Esc), substring highlight, click-outside to close. The typed value is validated against the available languages list before submission.
* Memories:
- Memories can now reference each other using the `mem:<name>` convention. Renames
propagate to all references automatically. See the [reference convention](https://oraios.github.io/serena/02-usage/045_memories.html#referencing-memories-from-other-memories).
- Onboarding now seeds a `memory_maintenance` memory describing the memory-style and conventions,
and the agent is instructed to read it before writing any other memories.
A `global/memory_maintenance` memory takes precedence over the per-project seed.
See the [memory maintenance section](https://oraios.github.io/serena/02-usage/045_memories.html#the-memory-maintenance-memory).
* CLI:
- Add `serena memories` CLI command group: `list`, `read`, `write`, `check` (referential
integrity report) and `auto-prefix-references` (heuristic rewrite of bare occurrences).
See the [CLI subcommands](https://oraios.github.io/serena/02-usage/045_memories.html#cli-subcommands).
* Tools:
- `search_for_pattern`: Add parameter `multiline`
- Delete `check_onboarding_performed` tool (instead extend project activation message)
# v1.3.0 (2026-05-11)
* General:
+10 -2
View File
@@ -143,9 +143,17 @@ and to store memories, which it can then draw upon in future interactions.
In general, **memories** provide a way for Serena to store and retrieve
information about the project, relevant conventions, and other relevant aspects.
Memories may reference each other using the `` `mem:NAME` `` convention; references
are kept in sync across renames, and a `serena memories check` command is available
to report stale references.
For more information on this, including how to manage
or disable these features, see [Memories & Onboarding](045_memories).
During the first onboarding, Serena seeds a `memory_maintenance` memory describing the
conventions (style, references) that subsequent memories should follow, and the
agent is instructed to read it before writing any project-specific memories.
For more information on this, including the target memory layout, the `mem:` reference
convention, the `serena memories` CLI subcommands, and how to manage or disable these
features, see [Memories & Onboarding](045_memories).
## Preparing Your Project
+119 -5
View File
@@ -4,10 +4,11 @@ Serena provides the functionality of a fully featured agent, and a useful aspect
Despite its simplicity, we received positive feedback from many users who tend to combine it with their
agent's internal memory management (e.g., `AGENTS.md` files).
(memories)=
## Memories
Memories are simple, human-readable Markdown files that both you and
your agent can create, read, and edit.
your agent can create, read, reference, and edit.
Serena differentiates between
* **project-specific memories**, which are stored in the `.serena/memories/` directory within your project folder, and
@@ -18,12 +19,76 @@ inferring appropriateness from the file name.
When the agent starts working on a project, it receives the list of available memories.
The agent should be instructed to update memories by the user when appropriate.
### Design Rationale
Serena's memory system is intentionally minimal. It was designed to satisfy the following
criteria:
1. **Human-readable and editable.** Memories must remain directly readable and editable
in any text editor. The agent is typically the day-to-day consumer, but a human author
or reviewer must always be able to step in without going through the agent.
2. **Versionable with the project.** Project memories live alongside the code and can
be committed, reviewed in PRs, and reverted like any other repository artifact.
3. **Progressive disclosure.** Agents receive the full memory *name list* up
front as part of their initial instructions; any further references are described inside
the memory content itself - typically a `mem:core` entry point pointing at focused
memories. The agent decides what to read based on names plus the references it has
already seen.
4. **Prefer references to search.** Given an intelligent agent and well-structured references, search is
unnecessary - and it adds noise: any retrieval method (lexical or semantic)
produces both false positives and false negatives. Explicit, name-based references
decided by the agent are deterministic and avoid both error modes. Basic search via regex/grep
is sufficient to complement the references when needed and is available to any agent.
5. **Prefer deliberate reads to triggers.** The agent decides what to read and when. The harness does
not inject memory content on the agent's behalf.
6. **Framework-agnostic.** The storage format is plain Markdown files in
a simple directory layout. The only Serena-specific convention is the `mem:`
prefix for references to memories, which does not prevent using the memory files outside
of Serena.
7. **Configurable and composable.** Two orthogonal memory scopes -
[per-project](memories) (committed alongside the code) and [global](global-memories)
(shared across all your projects) - can be combined freely. Within either scope,
regex patterns in the global or project configuration can mark subsets as read-only
or [hide them entirely](ignoring-memories) from the agent. This lets a project mix
personal cross-project knowledge with checked-in project conventions, and selectively
freeze either set, without custom plumbing.
Taken together, these criteria rule out several common alternatives:
- **Database-backed memory** (SQLite, graph databases, vector stores) is excluded by
criteria 1, 4, and 6.
- **`AGENTS.md` and similar single-file conventions** are excluded by criteria 3 and 5.
- **Hooks and harness-internal memory systems** are excluded by criteria 5 and 6.
To our knowledge, no existing system satisfies our design goal, which is
why Serena ships its own memory layer rather than reusing one.
The closest existing approaches are in the family of Markdown-based personal knowledge
management tools - **Obsidian**, **Logseq**, **Foam**.
### Organizing Memories
Memories can be organized into **topics** by using `/` in the memory name (e.g. `modules/frontend`).
The structure is mapped to the file system, where topics correspond to subdirectories.
The `list_memories` tool can filter by topic, allowing the agent to explore even large numbers of memories in a structured way.
(memory-references)=
### Referencing Memories from Other Memories
Memories may reference each other. Serena recognizes a reference as a memory name prefixed with
`mem:` and wrapped in backticks, for example `` `mem:auth/login` `` or `` `mem:suggested_commands` ``.
This convention has two practical consequences:
- **Renames keep references intact.** When you rename or move a memory with the `rename_memory`
tool, Serena rewrites every `` `mem:OLD_NAME` `` occurrence across all memories to point to
the new name. References that do not use the `mem:` prefix will not be updated automatically.
- **Integrity checks** (see [below](memory-cli)) report any `` `mem:NAME` `` whose target does
not resolve to an existing memory, and propose similarly-named candidates as likely intended
targets.
The full convention - including style, add/update thresholds, and how to structure references across
`core` memories - is shipped to every onboarded project as the `memory_maintenance` memory; see the
[Onboarding section](onboarding) below.
(global-memories)=
### Global Memories
@@ -39,6 +104,7 @@ Since global memories are not versioned alongside your project files,
it can be helpful to track global memories with git (i.e. to make `~/.serena/memories/` a git repository)
in order to have a history of changes and the possibility to revert them if needed.
(ignoring-memories)=
### Ignoring Memories
Projects that accumulate large numbers of archived memory files can use `ignored_memory_patterns`
@@ -49,7 +115,7 @@ global or project-level [configuration](050_configuration):
ignored_memory_patterns: ["_archive/.*", "_episodes/.*"]
```
Ignored memories are completely excluded — they cannot be accessed via `read_memory`, `write_memory`,
Ignored memories are completely excluded - they cannot be accessed via `read_memory`, `write_memory`,
or any other memory tool. To read an ignored memory file, use the `read_file` tool on the raw file path
(e.g., `.serena/memories/_archive/2026-03/some-topic.md`).
@@ -66,8 +132,8 @@ viewing, creating, editing, and deleting memories while Serena is running.
By default, Serena performs an **onboarding process** when it encounters a project
for the first time (i.e., when no project memories exist yet).
The goal of the onboarding is for Serena to get familiar with the project —
its structure, build system, testing setup, and other essential aspects —
The goal of the onboarding is for Serena to get familiar with the project -
its structure, build system, testing setup, and other essential aspects -
and to store this knowledge as memories for future interactions.
In further project activations, Serena will check whether onboarding was already
@@ -80,7 +146,32 @@ process if memories are found.
performed (by checking if any memories exist).
2. If no memories are found, Serena triggers the onboarding process, which
reads key files and directories to understand the project.
3. The gathered information is written into project-specific memory files (see above).
3. Before any project memory is written, Serena materializes a project-local
`memory_maintenance` memory (see below). The agent is then instructed to read it
first and follow the conventions it describes.
4. The gathered information is written into project-specific memory files following
the onboarding prompt instructions and the conventions outlined in `memory_maintenance`.
(memory-maintenance-memory)=
### The `memory_maintenance` Memory
To make memory conventions discoverable to both the LLM and the user, Serena seeds
a `memory_maintenance` memory on first onboarding. The seed is copied from a template
shipped with the Serena package and contains the dense agent-notes style, the
`mem:` reference convention, the reference model around `core` memories, the
add/update threshold, and the maintenance actions (rename / delete / split).
The seeding follows a strict precedence:
1. If you already maintain a `global/memory_maintenance` memory, Serena uses that
and **does not** create a project-local copy. This is the recommended approach
for teams that want one shared convention document across all projects.
2. Otherwise, if the project already has a `memory_maintenance` memory, it is left
untouched.
3. Otherwise, the shipped template is written to `.serena/memories/memory_maintenance.md`.
Existing files are never overwritten - you can freely customize the project copy.
To refresh from the shipped template, delete the existing memory first.
### Tips for Onboarding
@@ -92,6 +183,29 @@ process if memories are found.
- **Review the results**: After onboarding, we recommend having a quick look at the
generated memories and editing them or adding new ones as needed.
(memory-cli)=
### CLI Subcommands
While the recommended way to manage memories is through the **MCP integration**,
Serena also offers memory-related CLI commands.
The following commands have **no MCP tool counterpart** and are intended for human execution:
- `serena memories check` — referential-integrity report. By default reports stale
`` `mem:NAME` `` references; additional scans (bare occurrences and fuzzy near-misses)
are opt-in via flags. Run `serena memories check --help` for the full flag list.
- `serena memories auto-prefix-references` — heuristic rewrite of bare occurrences to add
the `mem:` prefix; supports `--dry-run`.
- `serena memories initialize` will seed the `memory_maintenance` memory for the project.
The remaining commands mirror the MCP tools, you can thus instruct your agent to manage memories with
serena without having a running MCP server. Discover the full surface and per-command flags via:
```shell
serena memories --help
serena memories <subcommand> --help
```
## Disabling Memories and Onboarding
If you do not require the functionality described in this section, you can selectively disable it.
+1 -1
View File
@@ -362,7 +362,7 @@ markers = [
[tool.codespell]
# Ref: https://github.com/codespell-project/codespell#using-a-config-file
skip = '.git*,*.svg,*.lock,*.min.*'
skip = '.git*,*.svg,*.lock,*.min.*,*test_memories_manager.py'
check-hidden = true
ignore-regex = '\.\w+'
ignore-words-list = 'paket,EDN,als'
+7 -3
View File
@@ -41,12 +41,14 @@ from serena.config.serena_config import (
)
from serena.dashboard import SerenaDashboardAPI, SerenaDashboardTrayManager, SerenaDashboardViewer, open_url_in_browser
from serena.ls_manager import LanguageServerManager
from serena.project import MemoriesManager, Project
from serena.memories.memory_manager import MemoryManager
from serena.project import Project
from serena.prompt_factory import SerenaPromptFactory
from serena.task_executor import TaskExecutor
from serena.tools import (
ActivateProjectTool,
GetCurrentConfigTool,
OnboardingTool,
OpenDashboardTool,
ReadMemoryTool,
ReplaceContentTool,
@@ -917,7 +919,7 @@ class SerenaAgent:
"""
available_tools = self._active_tools
available_markers = available_tools.tool_marker_names
global_memories = MemoriesManager(
global_memories = MemoryManager(
serena_data_folder=None, read_only_memory_patterns=self.serena_config.read_only_memory_patterns
).list_global_memories()
global_memories_str = dict_string(global_memories.to_dict()) if len(global_memories) > 0 else ""
@@ -975,12 +977,14 @@ class SerenaAgent:
# add list of memories (if memories are enabled)
include_memories = self._active_tools.contains_tool_class(ReadMemoryTool)
if include_memories:
project_memories = proj.memories_manager.list_project_memories()
project_memories = proj.memory_manager.list_project_memories()
if project_memories:
msg += (
f"\n{json.dumps(project_memories.to_dict())}\n"
+ "Use the `read_memory` tool to read these memories later if they are relevant to the task."
)
elif self._active_tools.contains_tool_class(OnboardingTool):
msg += "Onboarding has not been performed yet, you should call Serena's `onboarding` tool now to set up project memories."
# add prompts for modes that were dynamically activated by the project
modes_with_prompts = self._project_prompt_status.get_modes_with_prompts_to_be_provided_for_project_activation(session_id)
+282 -2
View File
@@ -9,7 +9,7 @@ import time
from collections.abc import Iterator, Sequence
from logging import Logger
from pathlib import Path
from typing import Any, Literal
from typing import TYPE_CHECKING, Any, Literal
import click
from sensai.util import logging
@@ -43,6 +43,9 @@ from solidlsp.ls_config import Language
from solidlsp.ls_types import SymbolKind
from solidlsp.util.subprocess_util import subprocess_kwargs
if TYPE_CHECKING:
from serena.memories.memory_manager import MemoryManager
log = logging.getLogger(__name__)
_MAX_CONTENT_WIDTH = 200
@@ -1113,6 +1116,282 @@ class ToolCommands(AutoRegisteringGroup):
click.echo(mcp_tool.description)
class MemoryCommands(AutoRegisteringGroup):
"""Group for 'memories' subcommands; manage and inspect a project's memory files."""
def __init__(self) -> None:
super().__init__(
name="memories",
help="Inspect, read, write, and validate Serena memories. "
"You can run `serena memories <command> --help` for more info on each command.",
)
@staticmethod
def _load_memories_manager(project: str) -> "MemoryManager":
"""
Load the project at ``project`` and return its memories manager.
Requires the project to already be a Serena project (``.serena/project.yml`` must
exist at the configured location). Never auto-creates the ``.serena`` directory —
if no project configuration is found, the user is directed at ``serena project create``.
"""
from serena.project import Project
serena_config = SerenaConfig.from_config_file()
project_path = os.path.abspath(project)
try:
proj = Project.load(project_path, serena_config=serena_config, autogenerate=False)
except FileNotFoundError as e:
raise click.UsageError(
f"{e}\nNo Serena project found at {project_path}. Create one first with:\n serena project create {project_path}"
) from e
return proj.memory_manager
@staticmethod
@click.command(
"initialize",
help=(
"Initialize this project's memory layout by seeding the `memory_maintenance` memory. "
"Requires the project to already exist as a Serena project (run `serena project create` first); "
"this command does not create a `.serena` directory on its own. "
"If a `global/memory_maintenance` memory exists, it takes precedence and no project copy is created."
),
context_settings={"max_content_width": _MAX_CONTENT_WIDTH},
)
@click.argument("project", type=click.Path(exists=True, file_okay=False, dir_okay=True), default=os.getcwd())
def initialize(project: str) -> None:
manager = MemoryCommands._load_memories_manager(project)
name = manager.ensure_memory_maintenance_memory()
click.echo(f"Memory maintenance memory ready: `mem:{name}`.")
@staticmethod
@click.command(
"list",
help="List memories of the active project (and global memories).",
context_settings={"max_content_width": _MAX_CONTENT_WIDTH},
)
@click.argument("project", type=click.Path(exists=True, file_okay=False, dir_okay=True), default=os.getcwd())
@click.option("--topic", "-t", type=str, default="", help="Restrict the listing to a single topic (e.g. 'auth' or 'global/style').")
def list(project: str, topic: str) -> None:
manager = MemoryCommands._load_memories_manager(project)
memories = manager.list_memories(topic=topic)
if not len(memories):
click.echo("(no memories found)")
return
if memories.memories:
click.echo("Project memories:")
for name in sorted(memories.memories):
click.echo(f" - {name}")
if memories.read_only_memories:
click.echo("Read-only memories:")
for name in sorted(memories.read_only_memories):
click.echo(f" - {name}")
@staticmethod
@click.command("read", help="Print the content of a memory to stdout.", context_settings={"max_content_width": _MAX_CONTENT_WIDTH})
@click.argument("memory_name", type=str)
@click.argument("project", type=click.Path(exists=True, file_okay=False, dir_okay=True), default=os.getcwd())
def read(memory_name: str, project: str) -> None:
manager = MemoryCommands._load_memories_manager(project)
click.echo(manager.load_memory(memory_name))
@staticmethod
@click.command(
"write",
help="Write a memory file. Reads the content from --content, --file, or stdin (in that order of precedence).",
context_settings={"max_content_width": _MAX_CONTENT_WIDTH},
)
@click.argument("memory_name", type=str)
@click.argument("project", type=click.Path(exists=True, file_okay=False, dir_okay=True), default=os.getcwd())
@click.option("--content", type=str, default=None, help="Memory content provided directly on the command line.")
@click.option(
"--file", "file_path", type=click.Path(exists=True, dir_okay=False), default=None, help="Read memory content from the given file."
)
def write(memory_name: str, project: str, content: str | None, file_path: str | None) -> None:
if content is None and file_path is not None:
with open(file_path, encoding="utf-8") as f:
content = f.read()
if content is None:
content = sys.stdin.read()
manager = MemoryCommands._load_memories_manager(project)
click.echo(manager.save_memory(memory_name, content, is_tool_context=False))
@staticmethod
@click.command(
"delete",
help="Delete a memory file. Use the `global/` prefix to address a global memory (e.g. `global/style_guide`).",
context_settings={"max_content_width": _MAX_CONTENT_WIDTH},
)
@click.argument("memory_name", type=str)
@click.argument("project", type=click.Path(exists=True, file_okay=False, dir_okay=True), default=os.getcwd())
def delete(memory_name: str, project: str) -> None:
manager = MemoryCommands._load_memories_manager(project)
click.echo(manager.delete_memory(memory_name, is_tool_context=False))
@staticmethod
@click.command(
"rename",
help=(
"Rename or move a memory and update every `mem:OLD_NAME` reference across all memories. "
"Use `/` in the name to organize into topics; use the `global/` prefix to address a "
"global memory. Moving between project and global scope is supported."
),
context_settings={"max_content_width": _MAX_CONTENT_WIDTH},
)
@click.argument("old_name", type=str)
@click.argument("new_name", type=str)
@click.argument("project", type=click.Path(exists=True, file_okay=False, dir_okay=True), default=os.getcwd())
def rename(old_name: str, new_name: str, project: str) -> None:
manager = MemoryCommands._load_memories_manager(project)
message, n_refs = manager.rename_memory_and_propagate_references(old_name, new_name, is_tool_context=False)
click.echo(message)
if n_refs > 0:
click.echo(f"Updated {n_refs} `mem:` reference occurrence(s) across the memory graph.")
@staticmethod
@click.command(
"edit",
help=(
"Replace content matching a pattern in a memory. By default operates in literal "
"(non-regex) mode and refuses to replace more than one occurrence; pass --mode regex "
"to enable regex matching and --allow-multiple-occurrences to permit multiple hits."
),
context_settings={"max_content_width": _MAX_CONTENT_WIDTH},
)
@click.argument("memory_name", type=str)
@click.argument("project", type=click.Path(exists=True, file_okay=False, dir_okay=True), default=os.getcwd())
@click.option("--needle", type=str, required=True, help="The text to search for (literal by default; regex if --mode=regex).")
@click.option("--repl", type=str, required=True, help="The replacement text (verbatim).")
@click.option(
"--mode",
type=click.Choice(["literal", "regex"]),
default="literal",
show_default=True,
help="Treat --needle as literal text or as a Python regex (MULTILINE and DOTALL flags enabled).",
)
@click.option(
"--allow-multiple-occurrences",
is_flag=True,
default=False,
help="Permit and apply multiple matches; without this, multiple matches raise an error.",
)
def edit(
memory_name: str,
project: str,
needle: str,
repl: str,
mode: Literal["literal", "regex"],
allow_multiple_occurrences: bool,
) -> None:
manager = MemoryCommands._load_memories_manager(project)
click.echo(
manager.edit_memory(
memory_name,
needle,
repl,
mode,
allow_multiple_occurrences,
is_tool_context=False,
)
)
@staticmethod
@click.command(
"check",
help=(
"Check referential integrity across all memories of the project (and global memories). "
"By default reports only stale `mem:` references. Pass --include-unmarked to also report "
"bare occurrences of existing memory names (exact matches) and --fuzzy-matching (only "
"meaningful in combination with --include-unmarked) to additionally report fuzzy "
"near-misses. Read-only and never writes. Always exits 0."
),
context_settings={"max_content_width": _MAX_CONTENT_WIDTH},
)
@click.argument("project", type=click.Path(exists=True, file_okay=False, dir_okay=True), default=os.getcwd())
@click.option(
"--include-unmarked",
is_flag=True,
default=False,
help="Also report bare exact occurrences of existing memory names (i.e. without the `mem:` prefix).",
)
@click.option(
"--fuzzy-matching",
is_flag=True,
default=False,
help=(
"Additionally report fuzzy near-misses (long bare tokens that similarity-match an existing "
"memory name). Only meaningful together with --include-unmarked; ignored otherwise."
),
)
def check(project: str, include_unmarked: bool, fuzzy_matching: bool) -> None:
if fuzzy_matching and not include_unmarked:
click.echo(
"Warning: --fuzzy-matching has no effect without --include-unmarked; ignoring it.",
err=True,
)
manager = MemoryCommands._load_memories_manager(project)
report = manager.validate_referential_integrity(
include_unmarked=include_unmarked,
include_fuzzy_matching=fuzzy_matching,
)
click.echo(report.format())
@staticmethod
@click.command(
"auto-prefix-references",
help=(
"Rewrite exact bare occurrences of existing memory names by adding the `mem:` prefix. "
"This is a heuristic, file-mutating operation: a word that happens to coincide with a memory name "
"will be rewritten as a reference even if it was intended as ordinary prose. "
"Use --dry-run to preview the rewrites without modifying any files. "
"\n\n"
"Scope is narrower than what `serena memories check` reports. Only EXACT bare occurrences are rewritten "
"(the body text must equal an existing memory name verbatim); fuzzy near-miss findings surfaced by `check` "
"are NOT autofixable here, since rewriting them would require substring substitution rather than a prefix "
"addition — they are reported for manual review. "
"By default the rewrite is further restricted to memory names containing `/` or longer than the configured "
"threshold, and skips global and read-only memories; use the --include-* flags below to widen the scope."
),
context_settings={"max_content_width": _MAX_CONTENT_WIDTH},
)
@click.argument("project", type=click.Path(exists=True, file_okay=False, dir_okay=True), default=os.getcwd())
@click.option(
"--dry-run",
is_flag=True,
default=False,
help="Preview the rewrites this command would apply; do not modify any files.",
)
@click.option(
"--include-flat-names",
is_flag=True,
default=False,
help="Also rewrite short, flat memory names (no `/`, below the length threshold). Raises false-positive risk significantly.",
)
@click.option(
"--include-read-only",
is_flag=True,
default=False,
help="Also rewrite occurrences inside read-only memories.",
)
@click.option(
"--include-global",
is_flag=True,
default=False,
help="Also rewrite occurrences inside global memories (affects every project consuming them).",
)
def auto_prefix_references(
project: str, dry_run: bool, include_flat_names: bool, include_read_only: bool, include_global: bool
) -> None:
manager = MemoryCommands._load_memories_manager(project)
report = manager.auto_prefix_bare_references(
include_flat_names=include_flat_names,
include_read_only=include_read_only,
include_global=include_global,
dry_run=dry_run,
)
click.echo(report.format())
class PromptCommands(AutoRegisteringGroup):
def __init__(self) -> None:
super().__init__(name="prompts", help="Commands related to Serena's prompts that are outside of contexts and modes.")
@@ -1244,10 +1523,11 @@ _project = ProjectCommands()
_config = SerenaConfigCommands()
_tools = ToolCommands()
_prompts = PromptCommands()
_memories = MemoryCommands()
# Expose so we can use this as an entrypoint
top_level = TopLevelCommands()
# needed for the help script to work - register all subcommands to the top-level group
for subgroup in (_mode, _context, _project, _config, _tools, _prompts):
for subgroup in (_mode, _context, _project, _config, _tools, _prompts, _memories):
top_level.add_command(subgroup)
+5 -7
View File
@@ -566,7 +566,7 @@ class SerenaDashboardAPI:
# Get available memories if ReadMemoryTool is active
available_memories = None
if self._agent.tool_is_active("read_memory") and project is not None:
available_memories = project.memories_manager.list_memories().get_full_list()
available_memories = project.memory_manager.list_memories().get_full_list()
# Get list of languages for the active project
languages = []
@@ -620,7 +620,7 @@ class SerenaDashboardAPI:
if project is None:
raise ValueError("No active project")
content = project.memories_manager.load_memory(request_get_memory.memory_name)
content = project.memory_manager.load_memory(request_get_memory.memory_name)
return ResponseGetMemory(content=content, memory_name=request_get_memory.memory_name)
return self._agent.execute_task(run, logged=False)
@@ -630,7 +630,7 @@ class SerenaDashboardAPI:
project = self._agent.get_active_project()
if project is None:
raise ValueError("No active project")
project.memories_manager.save_memory(request_save_memory.memory_name, request_save_memory.content, is_tool_context=False)
project.memory_manager.save_memory(request_save_memory.memory_name, request_save_memory.content, is_tool_context=False)
self._agent.execute_task(run, logged=True, name="SaveMemory")
@@ -639,7 +639,7 @@ class SerenaDashboardAPI:
project = self._agent.get_active_project()
if project is None:
raise ValueError("No active project")
project.memories_manager.delete_memory(request_delete_memory.memory_name, is_tool_context=False)
project.memory_manager.delete_memory(request_delete_memory.memory_name, is_tool_context=False)
self._agent.execute_task(run, logged=True, name="DeleteMemory")
@@ -649,9 +649,7 @@ class SerenaDashboardAPI:
if project is None:
raise ValueError("No active project")
return project.memories_manager.move_memory(
request_rename_memory.old_name, request_rename_memory.new_name, is_tool_context=False
)
return project.memory_manager.move_memory(request_rename_memory.old_name, request_rename_memory.new_name, is_tool_context=False)
return self._agent.execute_task(run, logged=True, name="RenameMemory")
@@ -22,7 +22,7 @@ class PromptFactory(PromptFactoryBase):
def get_onboarding_prompt_template_string(self) -> str:
return self.get_prompt_template_string("onboarding_prompt")
def create_onboarding_prompt(self, *, system: Any) -> str:
def create_onboarding_prompt(self, *, memory_maintenance_name: Any, system: Any) -> str:
return self._render_prompt("onboarding_prompt", locals())
def get_connection_prompt_template_string(self) -> str:
+407
View File
@@ -0,0 +1,407 @@
import logging
import os
import re
import shutil
from collections.abc import Sequence
from pathlib import Path
from typing import Literal
from serena.config.serena_config import (
SerenaPaths,
)
from serena.constants import SERENA_FILE_ENCODING
from serena.util.text_utils import ContentReplacer
from .memory_reference_analysis import (
MEMORY_REF_PREFIX,
AutofixReport,
MemoryReferenceAnalyzer,
ReferentialIntegrityReport,
)
log = logging.getLogger(__name__)
class MemoryManager:
GLOBAL_TOPIC = "global"
_global_memory_dir = SerenaPaths().global_memories_path
_MEMORY_REF_PREFIX = MEMORY_REF_PREFIX
def __init__(
self,
serena_data_folder: str | Path | None,
read_only_memory_patterns: Sequence[str] = (),
ignored_memory_patterns: Sequence[str] = (),
):
"""
:param serena_data_folder: the absolute path to the project's .serena data folder
:param read_only_memory_patterns: whether to allow writing global memories in tool execution contexts
:param ignored_memory_patterns: regex patterns for memories to completely exclude from listing, reading, and writing.
Matching memories will not appear in list_memories or activate_project output and cannot be accessed
via read_memory or write_memory. Use read_file on the raw path to access ignored memory files.
"""
self._project_memory_dir: Path | None = None
if serena_data_folder is not None:
self._project_memory_dir = Path(serena_data_folder) / "memories"
self._project_memory_dir.mkdir(parents=True, exist_ok=True)
self._encoding = SERENA_FILE_ENCODING
self._read_only_memory_patterns = [re.compile(pattern) for pattern in set(read_only_memory_patterns)]
self._ignored_memory_patterns = [re.compile(pattern) for pattern in set(ignored_memory_patterns)]
def _is_read_only_memory(self, name: str) -> bool:
for pattern in self._read_only_memory_patterns:
if pattern.fullmatch(name):
return True
return False
def _is_ignored_memory(self, name: str) -> bool:
for pattern in self._ignored_memory_patterns:
if pattern.fullmatch(name):
return True
return False
def _check_not_ignored(self, name: str) -> None:
if self._is_ignored_memory(name):
raise ValueError(
f"Memory '{name}' matches an ignored_memory_patterns pattern and cannot be accessed. "
f"Use the read_file tool on the raw file path instead."
)
def _is_global(self, name: str) -> bool:
return name == self.GLOBAL_TOPIC or name.startswith(self.GLOBAL_TOPIC + "/")
@classmethod
def _sanitize_name(cls, name: str) -> str:
"""Corrects the name for common mistakes made by LLMs (``mem:`` prefix, ``.md`` suffix, OS-specific separators)."""
name = name.removeprefix(cls._MEMORY_REF_PREFIX)
if name.endswith(".md"):
name = name[:-3]
return name.replace(os.sep, "/")
@classmethod
def _add_reference_prefix(cls, name: str) -> str:
name = cls._sanitize_name(name)
return cls._MEMORY_REF_PREFIX + name
MEMORY_MAINTENANCE_NAME: str = "memory_maintenance"
_MEMORY_MAINTENANCE_TEMPLATE_PATH: Path = Path(__file__).parent / "resources" / "memory_maintenance.md"
def ensure_memory_maintenance_memory(self) -> str:
"""
Ensures a memory describing how memories should be maintained exists for this project,
and returns the name to reference it by.
Precedence:
1. If a global copy exists at ``global/memory_maintenance``, return that name; no
project copy is created (the global version takes precedence).
2. Else if a project copy already exists, return its name unchanged.
3. Else seed a project copy from the package-shipped template and return that name.
Existing memory files are never overwritten; users may have customized them. To
refresh from the shipped template, delete the existing memory first.
:return: the bare name to reference the maintenance memory by (without the ``mem:``
prefix); either ``"global/memory_maintenance"`` or ``"memory_maintenance"``.
:raises FileNotFoundError: if the shipped template is missing on disk.
:raises AssertionError: if this manager has no associated project directory.
"""
global_name = f"{self.GLOBAL_TOPIC}/{self.MEMORY_MAINTENANCE_NAME}"
if self.get_memory_file_path(global_name).exists():
return global_name
if self.get_memory_file_path(self.MEMORY_MAINTENANCE_NAME).exists():
return self.MEMORY_MAINTENANCE_NAME
# seed a project copy from the shipped template
template_path = self._MEMORY_MAINTENANCE_TEMPLATE_PATH
if not template_path.exists():
raise FileNotFoundError(f"Memory maintenance template not found at {template_path}")
content = template_path.read_text(encoding=self._encoding)
self.save_memory(self.MEMORY_MAINTENANCE_NAME, content, is_tool_context=False)
return self.MEMORY_MAINTENANCE_NAME
def rename_references_to_memory(self, content: str, old_name: str, new_name: str) -> tuple[str, int]:
r"""
Replaces all occurrences of a memory reference (e.g. ``mem:foo``) in ``content`` with
the reference to ``new_name``.
Matches only references whose name is exactly ``old_name``: the match must not be
embedded in a longer memory name. A memory name consists of the character class
``[A-Za-z0-9_\\-/]`` (alphanumerics, underscore, hyphen, slash for topic separation),
which determines the boundary of the match. The surrounding delimiters (backticks,
quotes, parentheses, whitespace, etc.) are intentionally unconstrained.
:param content: the text to search through
:param old_name: the memory name being renamed away from (without the ``mem:`` prefix)
:param new_name: the memory name being renamed to (without the ``mem:`` prefix)
:return: a tuple of (updated content, number of replacements made)
"""
# define the character class that constitutes a memory name; matches inside such a run are not real references
name_char = r"[A-Za-z0-9_\-/]"
ref_old = self._add_reference_prefix(old_name)
ref_new = self._add_reference_prefix(new_name)
# build a pattern that anchors the reference on both sides so it cannot be embedded inside a longer name
pattern = rf"(?<!{name_char}){re.escape(ref_old)}(?!{name_char})"
# use a callable replacement to avoid backreference interpretation of characters in ref_new
return re.subn(pattern, lambda _m: ref_new, content)
def get_memory_file_path(self, name: str) -> Path:
name = self._sanitize_name(name)
parts = name.split("/")
if ".." in parts:
raise ValueError(f"Memory name cannot contain '..' segments for security reasons. Got: {name}")
if self._is_global(name):
if name == self.GLOBAL_TOPIC:
raise ValueError(
f'Bare "{self.GLOBAL_TOPIC}" is not a valid memory name. Use "{self.GLOBAL_TOPIC}/<name>" to address a global memory.'
)
# Strip "global/" prefix and resolve against global dir
sub_name = name[len(self.GLOBAL_TOPIC) + 1 :]
parts = sub_name.split("/")
filename = f"{parts[-1]}.md"
if len(parts) > 1:
subdir = self._global_memory_dir / "/".join(parts[:-1])
subdir.mkdir(parents=True, exist_ok=True)
return subdir / filename
return self._global_memory_dir / filename
# Project-local memory
assert self._project_memory_dir is not None, "Project dir was not passed at initialization"
filename = f"{parts[-1]}.md"
if len(parts) > 1:
# Create subdirectory path
subdir = self._project_memory_dir / "/".join(parts[:-1])
subdir.mkdir(parents=True, exist_ok=True)
return subdir / filename
return self._project_memory_dir / filename
def _check_write_access(self, name: str, is_tool_context: bool) -> None:
# in tool context, memories can be read-only
if is_tool_context and self._is_read_only_memory(name):
raise PermissionError(f"Attempted to write to read_only memory: '{name}')")
def load_memory(self, name: str) -> str:
name = self._sanitize_name(name)
self._check_not_ignored(name)
memory_file_path = self.get_memory_file_path(name)
if not memory_file_path.exists():
return f"Memory file {name} not found, consider creating it with the `write_memory` tool if you need it."
with open(memory_file_path, encoding=self._encoding) as f:
return f.read()
def save_memory(self, name: str, content: str, is_tool_context: bool) -> str:
name = self._sanitize_name(name)
self._check_not_ignored(name)
self._check_write_access(name, is_tool_context)
memory_file_path = self.get_memory_file_path(name)
with open(memory_file_path, "w", encoding=self._encoding) as f:
f.write(content)
return f"Memory {name} written."
class MemoriesList:
def __init__(self) -> None:
self.memories: list[str] = []
self.read_only_memories: list[str] = []
def __len__(self) -> int:
return len(self.memories) + len(self.read_only_memories)
def add(self, memory_name: str, is_read_only: bool) -> None:
if is_read_only:
self.read_only_memories.append(memory_name)
else:
self.memories.append(memory_name)
def extend(self, other: "MemoryManager.MemoriesList") -> None:
self.memories.extend(other.memories)
self.read_only_memories.extend(other.read_only_memories)
def to_dict(self) -> dict[str, list[str]]:
result = {}
if self.memories:
result["memories"] = sorted(self.memories)
if self.read_only_memories:
result["read_only_memories"] = sorted(self.read_only_memories)
return result
def get_full_list(self) -> list[str]:
return sorted(self.memories + self.read_only_memories)
def _list_memories(self, search_dir: Path, base_dir: Path, prefix: str = "") -> MemoriesList:
result = self.MemoriesList()
if not search_dir.exists():
return result
for md_file in search_dir.rglob("*.md"):
rel = str(md_file.relative_to(base_dir).with_suffix("")).replace(os.sep, "/")
memory_name = prefix + rel
if self._is_ignored_memory(memory_name):
continue
result.add(memory_name, is_read_only=self._is_read_only_memory(memory_name))
return result
def list_global_memories(self, subtopic: str = "") -> MemoriesList:
dir_path = self._global_memory_dir
if subtopic:
dir_path = dir_path / subtopic.replace("/", os.sep)
return self._list_memories(dir_path, self._global_memory_dir, self.GLOBAL_TOPIC + "/")
def list_project_memories(self, topic: str = "") -> MemoriesList:
assert self._project_memory_dir is not None, "Project dir was not passed at initialization"
dir_path = self._project_memory_dir
if topic:
dir_path = dir_path / topic.replace("/", os.sep)
return self._list_memories(dir_path, self._project_memory_dir)
def list_memories(self, topic: str = "") -> MemoriesList:
"""
Lists all memories, optionally filtered by topic.
If the topic is omitted, both global and project-specific memories are returned.
"""
memories: MemoryManager.MemoriesList
if topic:
if self._is_global(topic):
topic_parts = topic.split("/")
subtopic = "/".join(topic_parts[1:])
memories = self.list_global_memories(subtopic=subtopic)
else:
memories = self.list_project_memories(topic=topic)
else:
memories = self.list_project_memories()
memories.extend(self.list_global_memories())
return memories
def delete_memory(self, name: str, is_tool_context: bool) -> str:
name = self._sanitize_name(name)
self._check_not_ignored(name)
self._check_write_access(name, is_tool_context)
memory_file_path = self.get_memory_file_path(name)
if not memory_file_path.exists():
return f"Memory {name} not found."
memory_file_path.unlink()
return f"Memory {name} deleted."
def move_memory(self, old_name: str, new_name: str, is_tool_context: bool) -> str:
"""
Rename or move a memory file.
Moving between global and project scope (e.g. "global/foo" -> "bar") is supported.
"""
old_name = self._sanitize_name(old_name)
new_name = self._sanitize_name(new_name)
self._check_not_ignored(old_name)
self._check_not_ignored(new_name)
self._check_write_access(new_name, is_tool_context)
old_path = self.get_memory_file_path(old_name)
new_path = self.get_memory_file_path(new_name)
if not old_path.exists():
raise FileNotFoundError(f"Memory {old_name} not found.")
if new_path.exists():
raise FileExistsError(f"Memory {new_name} already exists.")
new_path.parent.mkdir(parents=True, exist_ok=True)
shutil.move(old_path, new_path)
return f"Memory renamed from {old_name} to {new_name}."
def rename_memory_and_propagate_references(self, old_name: str, new_name: str, is_tool_context: bool) -> tuple[str, int]:
"""
Renames a memory and updates every ``mem:OLD_NAME`` reference across all memories.
Memories whose content does not contain a reference to ``old_name`` are left
untouched (no spurious mtime changes). Memories that do are rewritten via
:meth:`save_memory`.
:param old_name: the current memory name (the source of the rename)
:param new_name: the target memory name
:param is_tool_context: forwarded to :meth:`save_memory` for read-only enforcement
:return: a tuple of (rename message returned by :meth:`move_memory`, total number of
``mem:`` reference occurrences rewritten across all memories).
"""
renaming_message = self.move_memory(old_name, new_name, is_tool_context=is_tool_context)
total_updates = 0
for memory_name in self.list_memories().get_full_list():
content = self.load_memory(memory_name)
updated_content, n_replacements = self.rename_references_to_memory(content, old_name, new_name)
if n_replacements > 0:
self.save_memory(memory_name, updated_content, is_tool_context=is_tool_context)
total_updates += n_replacements
return renaming_message, total_updates
def edit_memory(
self,
name: str,
needle: str,
repl: str,
mode: Literal["literal", "regex"],
allow_multiple_occurrences: bool,
is_tool_context: bool,
regex_multiline: bool = True,
) -> str:
"""
Edit a memory by replacing content matching a pattern.
:param name: the memory name
:param needle: the string or regex to search for
:param repl: the replacement string
:param mode: "literal" or "regex"
:param allow_multiple_occurrences:
:param is_tool_context: whether the call originates from a tool invocation (affects write-access checks)
:param regex_multiline: whether to apply multi-line regex matching, enabling the flags re.DOTALL and re.MULTILINE
"""
name = self._sanitize_name(name)
self._check_not_ignored(name)
self._check_write_access(name, is_tool_context)
memory_file_path = self.get_memory_file_path(name)
if not memory_file_path.exists():
raise FileNotFoundError(f"Memory {name} not found.")
with open(memory_file_path, encoding=self._encoding) as f:
original_content = f.read()
replacer = ContentReplacer(mode=mode, allow_multiple_occurrences=allow_multiple_occurrences, regex_multiline=regex_multiline)
updated_content = replacer.replace(original_content, needle, repl)
with open(memory_file_path, "w", encoding=self._encoding) as f:
f.write(updated_content)
return f"Memory {name} edited successfully."
def validate_referential_integrity(
self, include_unmarked: bool = True, include_fuzzy_matching: bool = True
) -> ReferentialIntegrityReport:
"""
Validates referential integrity across this manager's memories.
Thin wrapper around :meth:`MemoryReferenceAnalyzer.validate_referential_integrity`;
see that method for the full description of behavior and parameters.
"""
return MemoryReferenceAnalyzer(self).validate_referential_integrity(
include_unmarked=include_unmarked,
include_fuzzy_matching=include_fuzzy_matching,
)
def auto_prefix_bare_references(
self,
include_flat_names: bool = False,
include_read_only: bool = False,
include_global: bool = False,
dry_run: bool = False,
) -> AutofixReport:
"""
Rewrites bare occurrences of existing memory names to include the ``mem:`` prefix.
Thin wrapper around :meth:`MemoryReferenceAnalyzer.auto_prefix_bare_references`;
see that method for the full description of behavior and parameters.
"""
return MemoryReferenceAnalyzer(self).auto_prefix_bare_references(
include_flat_names=include_flat_names,
include_read_only=include_read_only,
include_global=include_global,
dry_run=dry_run,
)
@@ -0,0 +1,780 @@
"""
Memory reference matching, similarity scoring, and integrity reporting.
This module contains the pure helpers and data types used by
:class:`serena.project.MemoriesManager` to (a) detect and rank references
between memories, and (b) report referential integrity issues. The
:class:`MemoryReferenceAnalyzer` class drives the validation and autofix
workflows against a live ``MemoriesManager`` instance.
Kept separate from the manager so the matching heuristics can be evolved
and tested in isolation from filesystem and lifecycle concerns.
"""
from __future__ import annotations
import re
from collections.abc import Iterator
from dataclasses import dataclass, field
from difflib import SequenceMatcher
from typing import TYPE_CHECKING
from sensai.util.string import TextBuilder
if TYPE_CHECKING:
from .memory_manager import MemoryManager
# ---------------------------------------------------------------------------
# Constants
# ---------------------------------------------------------------------------
MEMORY_REF_PREFIX: str = "mem:"
"""Reference prefix used inside memory bodies to point at another memory."""
NAME_SIMILARITY_THRESHOLD: float = 0.55
"""Minimum :func:`compute_name_similarity` score for a candidate to be reported."""
HIGH_CONFIDENCE_NAME_LENGTH: int = 10
"""Length threshold above which a flat memory name is considered "name-shaped" (i.e.
unlikely to coincide with ordinary prose)."""
SHORT_NAME_FLOOR: int = 3
"""Below this basename length, only an exact token match keeps similarity above 0."""
FUZZY_BARE_TOKEN_JACCARD_FLOOR: float = 0.6
"""For fuzzy bare-text matching, the bare token and the candidate existing name must
share at least this fraction of their tokenized name parts. Containment / SequenceMatcher
signals alone over-match — e.g. a generic prose word like ``repository`` is a substring of
``serena_repository_structure`` but their token Jaccard is only 1/3, so we reject it at
this floor."""
BASENAME_JACCARD_FLOOR: float = 0.34
"""In :func:`compute_name_similarity` case 2 (basenames differ), when the two names also
carry *differing* topic prefixes, demand at least one strong basename signal: token
Jaccard >= this floor, basename containment, or typo-level seq ratio. Without this gate,
names like ``frontend/x-subtleties`` and ``backend/y-subtleties`` get matched purely on the
shared trailing token via the SequenceMatcher signal."""
BASENAME_TYPO_SEQ_FLOOR: float = 0.75
"""Companion to :data:`BASENAME_JACCARD_FLOOR`: a basename seq ratio above this is treated
as typo-level edit distance and bypasses the Jaccard/containment requirement."""
MAX_STALE_REFERENCE_CANDIDATES: int = 3
"""Hard cap on candidates proposed per stale reference. Beyond a small number the list
stops aiding disambiguation and becomes noise."""
WORDS_TO_IGNORE_AS_MEMORY_NAME_CANDIDATES: frozenset[str] = frozenset({"core"})
"""Names that coincide with common English words and are filtered out of the unmarked-
reference scan to avoid false positives."""
NAME_CHAR_CLASS: str = r"[A-Za-z0-9_\-/]"
"""Character class delimiting a memory name. Used to anchor regex matches so we never
consume a partial name (kept in sync with the boundary rule in
:meth:`MemoriesManager.rename_references_to_memory`)."""
_VERSION_SUFFIX_PATTERN: re.Pattern[str] = re.compile(r"_(?:v\d+|old|new|legacy|bak)$", re.IGNORECASE)
# ---------------------------------------------------------------------------
# Pure helpers: normalization, tokenization, similarity
# ---------------------------------------------------------------------------
def normalize_for_similarity(name: str) -> str:
"""
:param name: a memory name
:return: a lowercased copy of ``name`` with version/legacy-style trailing suffixes stripped
(e.g. ``"auth_v2"`` -> ``"auth"``); used as the canonical form for similarity scoring.
"""
return _VERSION_SUFFIX_PATTERN.sub("", name.lower())
def tokenize_name(name: str) -> set[str]:
"""
:param name: a memory name
:return: the set of lowercase tokens extracted by splitting ``name`` on ``/``, ``_``, ``-``
and at camelCase boundaries; empty tokens are dropped.
"""
parts = re.split(r"[/_\-]|(?<=[a-z])(?=[A-Z])", name)
return {p.lower() for p in parts if p}
def compute_name_similarity(a: str, b: str) -> float:
"""
Computes a similarity score in ``[0, 1]`` between two memory names. Examples
(with default threshold ``0.55``)::
auth/login == auth/login -> 1.00 (exact)
auth_v1 ~ auth_v2 -> 1.00 (version suffix normalized)
login ~ auth/login -> 1.00 (flat <-> topic move)
auth/login ~ auth/v2/login -> 0.75 (shared prefix token + basename)
auth/login ~ security/login -> 0.50 (basename only; disjoint topics)
auth ~ authentication -> 0.64 (substring containment)
foo ~ for -> 0.00 (short-name floor)
frontend/x-subtleties ~ backend/y-subtleties -> 0.00 (different topics + only a
shared generic trailing token)
:param a: first memory name
:param b: second memory name
:return: the similarity score
"""
# normalize: lowercase + strip version/legacy suffixes
norm_a, norm_b = normalize_for_similarity(a), normalize_for_similarity(b)
if norm_a == norm_b:
return 1.0
# split into prefix / basename
basename_a = norm_a.rsplit("/", 1)[-1]
basename_b = norm_b.rsplit("/", 1)[-1]
prefix_a = norm_a.rsplit("/", 1)[0] if "/" in norm_a else ""
prefix_b = norm_b.rsplit("/", 1)[0] if "/" in norm_b else ""
# case 1: basenames match -- score is determined entirely by prefix relationship
if basename_a == basename_b:
if not prefix_a or not prefix_b:
# one side is flat -> canonical topic move, strongly confident
return 1.0
# both sides carry a topic prefix -> blend basename match (0.5 floor) with prefix token similarity
prefix_tokens_a, prefix_tokens_b = tokenize_name(prefix_a), tokenize_name(prefix_b)
prefix_jaccard = (
len(prefix_tokens_a & prefix_tokens_b) / len(prefix_tokens_a | prefix_tokens_b) if (prefix_tokens_a | prefix_tokens_b) else 0.0
)
return 0.5 + 0.5 * prefix_jaccard
# case 2: basenames differ -- score the *basenames* only, so that a shared topic prefix
# plus a generic trailing token (e.g. "frontend/X-subtleties" vs "frontend/Y-subtleties")
# does not inflate the SequenceMatcher signal into a false positive.
basename_tokens_a, basename_tokens_b = tokenize_name(basename_a), tokenize_name(basename_b)
union_basename = basename_tokens_a | basename_tokens_b
basename_jaccard = (len(basename_tokens_a & basename_tokens_b) / len(union_basename)) if union_basename else 0.0
basename_contained = bool(basename_a) and bool(basename_b) and (basename_a in basename_b or basename_b in basename_a)
seq = SequenceMatcher(None, basename_a, basename_b).ratio()
# when both names carry *differing* topic prefixes, demand at least one strong basename
# signal — otherwise a single coincidental shared token at the end of long basenames
# under different topic roots slips through purely on the seq-ratio
if prefix_a and prefix_b and prefix_a != prefix_b:
if basename_jaccard < BASENAME_JACCARD_FLOOR and not basename_contained and seq < BASENAME_TYPO_SEQ_FLOOR:
return 0.0
# containment is a meaningful rename signal on its own (e.g. auth -> authentication),
# so we lift it above the threshold floor when one basename fully contains the other,
# with the length-ratio gating how confident we are
contained = 0.0
if basename_contained:
length_ratio = min(len(basename_a), len(basename_b)) / max(len(basename_a), len(basename_b))
contained = 0.5 + 0.5 * length_ratio
# short-name floor: for very short basenames, only accept candidates with a strong token signal
if min(len(basename_a), len(basename_b)) <= SHORT_NAME_FLOOR and basename_jaccard < 1.0:
return 0.0
return max(basename_jaccard, seq, contained)
def find_stale_reference_candidates(missing_name: str, existing_names: list[str], threshold: float | None = None) -> list[str]:
"""
:param missing_name: the unresolved reference target (without the ``mem:`` prefix).
:param existing_names: the names of all currently existing memories.
:param threshold: optional override for the minimum similarity required; defaults to
:data:`NAME_SIMILARITY_THRESHOLD`.
:return: existing memory names whose similarity to ``missing_name`` meets or exceeds
``threshold``, sorted in descending order of similarity (ties broken alphabetically)
and capped at :data:`MAX_STALE_REFERENCE_CANDIDATES` entries to keep the output
focused on the most plausible matches.
"""
cutoff = NAME_SIMILARITY_THRESHOLD if threshold is None else threshold
scored: list[tuple[float, str]] = []
for existing in existing_names:
score = compute_name_similarity(missing_name, existing)
if score >= cutoff:
scored.append((score, existing))
scored.sort(key=lambda pair: (-pair[0], pair[1]))
return [name for _, name in scored[:MAX_STALE_REFERENCE_CANDIDATES]]
def iter_referenced_names_in_content(content: str) -> Iterator[str]:
"""
:param content: arbitrary memory content
:return: an iterator over the memory names appearing as ``mem:NAME`` references in
``content``; duplicate occurrences are yielded once each.
"""
# use the same boundary rule as MemoriesManager.rename_references_to_memory so the two stay consistent
pattern = rf"(?<!{NAME_CHAR_CLASS}){re.escape(MEMORY_REF_PREFIX)}({NAME_CHAR_CLASS}+?)(?!{NAME_CHAR_CLASS})"
for match in re.finditer(pattern, content):
yield match.group(1)
def find_bare_occurrences(content: str, name: str) -> int:
"""
:param content: arbitrary memory content
:param name: a memory name to look for
:return: the count of bare (i.e. not already ``mem:``-prefixed) word-boundary-anchored
occurrences of ``name`` in ``content``. The match must not be preceded by ``mem:``
and must not be embedded within a longer memory-name-like run of characters.
"""
pattern = (
rf"(?<!{re.escape(MEMORY_REF_PREFIX)})"
rf"(?<!{NAME_CHAR_CLASS})"
rf"{re.escape(name)}"
rf"(?!{NAME_CHAR_CLASS})"
)
return len(re.findall(pattern, content))
def add_bare_occurrences_prefix(content: str, name: str) -> tuple[str, int]:
"""
:param content: arbitrary memory content
:param name: the memory name whose bare occurrences should be rewritten
:return: ``(new_content, n_replacements)`` after prefixing each bare occurrence of
``name`` with ``mem:`` (using the same boundary rule as :func:`find_bare_occurrences`).
"""
pattern = (
rf"(?<!{re.escape(MEMORY_REF_PREFIX)})"
rf"(?<!{NAME_CHAR_CLASS})"
rf"{re.escape(name)}"
rf"(?!{NAME_CHAR_CLASS})"
)
replacement = MEMORY_REF_PREFIX + name
return re.subn(pattern, lambda _m: replacement, content)
def iter_long_bare_tokens(content: str) -> Iterator[tuple[str, int]]:
"""
:param content: arbitrary memory content
:return: an iterator over ``(token, count)`` pairs for each distinct name-shaped
token in ``content`` whose length meets :data:`HIGH_CONFIDENCE_NAME_LENGTH`
and which is not preceded by ``mem:`` (those are already valid references).
The boundaries follow :data:`NAME_CHAR_CLASS`, so embedded substrings of
longer name-character runs are not matched.
"""
pattern = (
rf"(?<!{re.escape(MEMORY_REF_PREFIX)})"
rf"(?<!{NAME_CHAR_CLASS})"
rf"({NAME_CHAR_CLASS}{{{HIGH_CONFIDENCE_NAME_LENGTH},}})"
rf"(?!{NAME_CHAR_CLASS})"
)
counts: dict[str, int] = {}
for match in re.finditer(pattern, content):
token = match.group(1)
counts[token] = counts.get(token, 0) + 1
return iter(counts.items())
def is_self_reference(source_memory: str, suspected_name: str) -> bool:
"""
:return: True iff ``suspected_name`` would refer to ``source_memory`` itself
(including the case where ``suspected_name`` equals the basename of a topic-path
source memory).
"""
if suspected_name == source_memory:
return True
source_basename = source_memory.rsplit("/", 1)[-1]
return suspected_name == source_basename
# ---------------------------------------------------------------------------
# Report dataclasses
# ---------------------------------------------------------------------------
@dataclass(frozen=True)
class StaleReference:
"""
A ``mem:NAME`` reference whose target memory does not exist.
:ivar source_memory: the name of the memory whose content contains the broken reference.
:ivar referenced_name: the name following ``mem:`` that did not resolve to an existing memory.
:ivar candidates: existing memory names proposed as likely intended targets, ranked by
decreasing similarity. May be empty if no candidate exceeded the similarity threshold.
:ivar source_is_read_only: whether the source memory is read-only.
"""
source_memory: str
referenced_name: str
candidates: list[str]
source_is_read_only: bool
@dataclass(frozen=True)
class UnmarkedReferenceWarning:
"""
A bare occurrence in a memory's content that looks like a forgotten reference to an
existing memory.
Two flavours of finding share this class:
* **exact match** — the bare text equals an existing memory name verbatim
(``actual_token`` either equals ``suspected_name`` or is left empty).
* **fuzzy near-miss** — the bare text does not equal any existing memory name, but
a long, distinctive token in the body similarity-matches a high-confidence existing
memory name (``actual_token`` is the bare text actually found, distinct from
``suspected_name``). Such findings are reported but **not** rewritten by
:meth:`MemoryReferenceAnalyzer.auto_prefix_bare_references`, since they would
require substring substitution rather than a prefix addition.
:ivar source_memory: the name of the memory whose content contains the bare occurrence.
:ivar suspected_name: the existing memory name proposed as the intended target.
:ivar occurrences: the number of occurrences of ``actual_token`` in the source memory's content.
:ivar is_high_confidence: True when ``suspected_name`` contains a ``/`` separator or
exceeds the configured length threshold; such names are unlikely to coincide with
ordinary prose. False otherwise (a low-confidence warning).
:ivar source_is_read_only: whether the source memory is read-only.
:ivar actual_token: the bare text actually found in the source memory's content.
Defaults to ``""``, which is taken to mean ``suspected_name`` (the exact-match case).
When non-empty and different from ``suspected_name``, this is a fuzzy near-miss.
"""
source_memory: str
suspected_name: str
occurrences: int
is_high_confidence: bool
source_is_read_only: bool
actual_token: str = ""
@property
def is_exact_match(self) -> bool:
""":return: True iff the bare text in the body equals ``suspected_name`` (i.e. not a fuzzy near-miss)."""
return self.actual_token == "" or self.actual_token == self.suspected_name
@dataclass(frozen=True)
class AutofixedReference:
"""
A bare occurrence rewritten to include the ``mem:`` prefix.
:ivar source_memory: the name of the memory whose content was modified.
:ivar referenced_name: the memory name whose bare occurrences were prefixed.
:ivar n_replacements: the number of bare occurrences replaced in the source memory.
"""
source_memory: str
referenced_name: str
n_replacements: int
@dataclass
class ReferentialIntegrityReport:
"""
Outcome of :meth:`MemoryReferenceAnalyzer.validate_referential_integrity`.
:ivar stale_references: ``mem:NAME`` references whose target memory does not exist.
:ivar high_confidence_unmarked_memories: bare references whose suspected target name is unlikely
to be coincidental prose (topic-path or sufficiently long).
:ivar low_confidence_unmarked_memories: bare references whose suspected target name could plausibly
appear in ordinary prose (short, flat names).
"""
stale_references: list[StaleReference] = field(default_factory=list)
high_confidence_unmarked_memories: list[UnmarkedReferenceWarning] = field(default_factory=list)
low_confidence_unmarked_memories: list[UnmarkedReferenceWarning] = field(default_factory=list)
def is_clean(self) -> bool:
""":return: True iff no stale references and no warnings of any confidence level were found."""
return not (self.stale_references or self.high_confidence_unmarked_memories or self.low_confidence_unmarked_memories)
def format(self) -> str:
""":return: a human-readable rendering suitable for CLI display."""
tb = TextBuilder()
if self.is_clean():
tb.with_line("✓ No referential integrity issues found.")
return tb.build()
# stale references — grouped by writable / read-only for clarity
if self.stale_references:
tb.with_line(f"Stale references ({len(self.stale_references)}):")
for ref in self.stale_references:
ro_tag = " [read-only source]" if ref.source_is_read_only else ""
tb.with_line(f" - `mem:{ref.referenced_name}` in `{ref.source_memory}`{ro_tag}")
if ref.candidates:
candidates_str = ", ".join(f"`mem:{c}`" for c in ref.candidates)
tb.with_line(f" candidates: {candidates_str}")
else:
tb.with_line(" candidates: (none above similarity threshold)")
tb.with_line("")
# unmarked-reference warnings — split exact vs fuzzy near-miss, then by confidence
exact_high = [w for w in self.high_confidence_unmarked_memories if w.is_exact_match]
exact_low = [w for w in self.low_confidence_unmarked_memories if w.is_exact_match]
fuzzy = [w for w in self.high_confidence_unmarked_memories if not w.is_exact_match] + [
w for w in self.low_confidence_unmarked_memories if not w.is_exact_match
]
# the "may be intentional" caveat is meaningful only for exact matches (where the bare
# text equals a real memory name and may simply be ordinary prose); print it once,
# before the first exact-match section
if exact_high or exact_low:
tb.with_line("Possibly unmarked references — exact matches:")
tb.with_line(" (note: bare matches in prose may be intentional and not actual references)")
for label, warnings in (("high confidence", exact_high), ("low confidence", exact_low)):
if not warnings:
continue
tb.with_line(f" {label} ({len(warnings)}):")
for w in warnings:
ro_tag = " [read-only source]" if w.source_is_read_only else ""
occ = "occurrence" if w.occurrences == 1 else "occurrences"
tb.with_line(
f" - {w.occurrences} {occ} of `{w.suspected_name}` in `{w.source_memory}`{ro_tag}"
f" (suggested: `mem:{w.suspected_name}`)"
)
tb.with_line("")
if fuzzy:
tb.with_line(f"Possibly unmarked references — fuzzy near-misses ({len(fuzzy)}):")
for w in fuzzy:
ro_tag = " [read-only source]" if w.source_is_read_only else ""
occ = "occurrence" if w.occurrences == 1 else "occurrences"
tb.with_line(
f" - {w.occurrences} {occ} of `{w.actual_token}` in `{w.source_memory}`{ro_tag}"
f" — near-miss (suggested: `mem:{w.suspected_name}`)"
)
tb.with_line("")
return tb.build()
@dataclass
class AutofixReport:
"""
Outcome of :meth:`MemoryReferenceAnalyzer.auto_prefix_bare_references`.
:ivar autofixed: per-(source, target) records of bare references that were rewritten.
When the call was a dry run, these records describe what *would* have been
written; the files themselves are unchanged.
:ivar dry_run: True if the run was a preview that did not write any files.
:ivar skipped_read_only: warnings whose source memory was read-only and therefore not
modified (only populated when ``include_read_only`` is False).
:ivar skipped_flat: warnings skipped because their suspected target had no ``/``
separator and was not long enough to be high-confidence (only populated when
``include_flat_names`` is False).
:ivar skipped_global: warnings skipped because their source memory was global and
``include_global`` was False.
:ivar skipped_fuzzy: warnings whose bare text in the source memory differs from the
suspected target (fuzzy near-misses). These require substring substitution rather
than a prefix addition and are never autofixed; surface them to the user for
manual review instead.
"""
autofixed: list[AutofixedReference] = field(default_factory=list)
dry_run: bool = False
skipped_read_only: list[UnmarkedReferenceWarning] = field(default_factory=list)
skipped_flat: list[UnmarkedReferenceWarning] = field(default_factory=list)
skipped_global: list[UnmarkedReferenceWarning] = field(default_factory=list)
skipped_fuzzy: list[UnmarkedReferenceWarning] = field(default_factory=list)
@property
def total_replacements(self) -> int:
return sum(a.n_replacements for a in self.autofixed)
def format(self) -> str:
""":return: a human-readable rendering suitable for CLI display."""
tb = TextBuilder()
verb = "Would apply" if self.dry_run else "Applied"
if self.autofixed:
tb.with_line(
f"{verb} {self.total_replacements} replacement(s) across "
f"{len(self.autofixed)} memory/target pair(s)" + (" (dry run; no files were modified)" if self.dry_run else "") + ":"
)
for a in self.autofixed:
tb.with_line(f" - {a.n_replacements} x `{a.referenced_name}` -> `mem:{a.referenced_name}` in `{a.source_memory}`")
tb.with_line("")
else:
tb.with_line("No replacements" + (" would be" if self.dry_run else "") + " applied.")
tb.with_line("")
for label, items in (
("Skipped (read-only source; pass --include-read-only to override)", self.skipped_read_only),
("Skipped (flat name; pass --include-flat-names to include)", self.skipped_flat),
("Skipped (global memory; pass --include-global to include)", self.skipped_global),
(
"Skipped (fuzzy near-miss; bare text differs from target name — not safe to auto-rewrite, review manually)",
self.skipped_fuzzy,
),
):
if not items:
continue
tb.with_line(f"{label}:")
for w in items:
if w.is_exact_match:
tb.with_line(f" - `{w.suspected_name}` in `{w.source_memory}`")
else:
tb.with_line(f" - `{w.actual_token}` in `{w.source_memory}` (suggested: `mem:{w.suspected_name}`)")
tb.with_line("")
return tb.build()
# ---------------------------------------------------------------------------
# Analyzer: ties the helpers above to a live MemoriesManager instance
# ---------------------------------------------------------------------------
class MemoryReferenceAnalyzer:
"""
Drives validation and autofix workflows against a :class:`MemoriesManager`.
Composition rather than inheritance — the analyzer only needs the manager's
memory enumeration, loading, saving, and is-global helpers, and is otherwise
independent of project-management concerns.
"""
def __init__(self, manager: MemoryManager):
self._manager = manager
def validate_referential_integrity(
self, include_unmarked: bool = True, include_fuzzy_matching: bool = True
) -> ReferentialIntegrityReport:
"""
Scans every (non-ignored) memory's content for referential integrity issues.
The scan covers both project-local and global memories. Three kinds of finding are
produced, each independently gated:
* **stale references** — occurrences of ``mem:NAME`` where ``NAME`` does not resolve
to an existing memory. For each, a list of similarly-named existing memories is
proposed as candidate intended targets (see :func:`compute_name_similarity`),
capped at :data:`MAX_STALE_REFERENCE_CANDIDATES`. Always reported.
* **exact unmarked-reference warnings** — bare occurrences of an existing memory
name that appear without the ``mem:`` prefix. Split into high-confidence (the
suspected name contains a ``/`` or exceeds :data:`HIGH_CONFIDENCE_NAME_LENGTH`)
and low-confidence groups. Candidate names whose basename is in
:data:`WORDS_TO_IGNORE_AS_MEMORY_NAME_CANDIDATES` are skipped. Gated by
``include_unmarked``.
* **fuzzy near-miss warnings** — long, distinctive bare tokens in a memory body
that *do not* match an existing name exactly but similarity-match a
high-confidence existing name AND share at least
:data:`FUZZY_BARE_TOKEN_JACCARD_FLOOR` of their tokenized name parts. Always
reported as high-confidence; the actual bare text is preserved on
:attr:`UnmarkedReferenceWarning.actual_token`. Gated by ``include_unmarked
and include_fuzzy_matching`` — fuzzy matches are noisy and only meaningful when
unmarked-reference checking is enabled.
Self-references (a memory's content mentioning its own name or basename) and empty
memories are skipped silently. This method has no side effects.
:param include_unmarked: if False, skip both the exact unmarked scan and the fuzzy
near-miss scan; only stale references are reported.
:param include_fuzzy_matching: if False, skip the fuzzy near-miss scan even when
unmarked-reference checking is otherwise enabled. Has no effect when
``include_unmarked`` is False.
:return: a :class:`ReferentialIntegrityReport` summarizing all findings.
"""
report = ReferentialIntegrityReport()
# gather all existing memory names (project + global) with their read-only status
memories_list = self._manager.list_memories()
all_names = sorted(set(memories_list.memories) | set(memories_list.read_only_memories))
read_only_names = set(memories_list.read_only_memories)
existing_set = set(all_names)
# only existing names that are themselves "memory-name-like" qualify as fuzzy candidates,
# otherwise short flat names would match arbitrary long prose words via containment
high_confidence_names = [n for n in all_names if "/" in n or len(n) >= HIGH_CONFIDENCE_NAME_LENGTH]
for source_name in all_names:
# content of this memory should not be analyzed regarding stale references
if source_name == self._manager.MEMORY_MAINTENANCE_NAME:
continue
content = self._manager.load_memory(source_name)
if not content:
# skip empty memories silently — no references can exist in empty content
continue
source_is_read_only = source_name in read_only_names
# stale references: every mem:NAME whose NAME is not an existing memory
stale_seen: set[str] = set()
for referenced in iter_referenced_names_in_content(content):
if referenced in existing_set or referenced in stale_seen:
continue
stale_seen.add(referenced)
candidates = find_stale_reference_candidates(referenced, all_names)
report.stale_references.append(
StaleReference(
source_memory=source_name,
referenced_name=referenced,
candidates=candidates,
source_is_read_only=source_is_read_only,
)
)
if not include_unmarked:
continue
# exact unmarked references: bare occurrences of any existing memory name (except self)
for candidate_name in all_names:
if is_self_reference(source_name, candidate_name):
continue
# skip candidate names whose basename is an ambiguous common word
if candidate_name.rsplit("/", 1)[-1] in WORDS_TO_IGNORE_AS_MEMORY_NAME_CANDIDATES:
continue
n = find_bare_occurrences(content, candidate_name)
if n == 0:
continue
is_high_conf = "/" in candidate_name or len(candidate_name) >= HIGH_CONFIDENCE_NAME_LENGTH
warning = UnmarkedReferenceWarning(
source_memory=source_name,
suspected_name=candidate_name,
occurrences=n,
is_high_confidence=is_high_conf,
source_is_read_only=source_is_read_only,
actual_token=candidate_name,
)
if is_high_conf:
report.high_confidence_unmarked_memories.append(warning)
else:
report.low_confidence_unmarked_memories.append(warning)
if not include_fuzzy_matching:
continue
# fuzzy near-misses: long bare tokens that similarity-match a high-confidence existing name
for token, n in iter_long_bare_tokens(content):
if token in existing_set:
# exact-match path already handled this token (or self-referenced and was skipped)
continue
# filter bare tokens that coincide with common English words; the same filter is
# applied to candidate names in the exact-match loop, this is the symmetric side
token_norm = normalize_for_similarity(token)
if token_norm in WORDS_TO_IGNORE_AS_MEMORY_NAME_CANDIDATES:
continue
token_tokens = tokenize_name(token)
best_name: str | None = None
best_score = 0.0
for existing in high_confidence_names:
if existing.rsplit("/", 1)[-1] in WORDS_TO_IGNORE_AS_MEMORY_NAME_CANDIDATES:
continue
# require substring containment between bare token and existing name: fuzzy
# matches without a containment relationship are rarely useful as rewrite
# suggestions (auto_prefix_bare_references can't fix them anyway) and dominate
# the false-positive rate, e.g. "grid-layout" vs "layout-grid-subtleties".
existing_norm = normalize_for_similarity(existing)
if token_norm not in existing_norm and existing_norm not in token_norm:
continue
score = compute_name_similarity(token, existing)
if score < NAME_SIMILARITY_THRESHOLD:
continue
# safeguard against substring-only matches: require meaningful token overlap so
# generic prose words don't get flagged just because they happen to be substrings
# of an existing memory name
existing_tokens = tokenize_name(existing)
union = token_tokens | existing_tokens
if not union:
continue
jaccard = len(token_tokens & existing_tokens) / len(union)
if jaccard < FUZZY_BARE_TOKEN_JACCARD_FLOOR:
continue
if score > best_score:
best_score = score
best_name = existing
if best_name is None:
continue
# skip self-suggestions: a memory's body containing a near-miss of its own
# name (e.g. capitalised "Conventions" in source `conventions`) would otherwise
# produce a warning pointing the source at itself
if normalize_for_similarity(best_name) == normalize_for_similarity(source_name):
continue
report.high_confidence_unmarked_memories.append(
UnmarkedReferenceWarning(
source_memory=source_name,
suspected_name=best_name,
occurrences=n,
is_high_confidence=True,
source_is_read_only=source_is_read_only,
actual_token=token,
)
)
return report
def auto_prefix_bare_references(
self,
include_flat_names: bool = False,
include_read_only: bool = False,
include_global: bool = False,
dry_run: bool = False,
) -> AutofixReport:
"""
Rewrites *exact* bare occurrences of existing memory names by adding the ``mem:`` prefix.
.. warning::
**This is a heuristic, file-mutating operation** (unless ``dry_run`` is True). A
bare word that happens to coincide with a memory name will be rewritten as a
reference, even if it was intended as ordinary prose. Pass ``dry_run=True`` to
preview the rewrites before applying them.
Scope is intentionally narrower than what :meth:`validate_referential_integrity`
reports:
* Only **exact** bare occurrences are rewritten — i.e. the bare text in the source
body must equal an existing memory name verbatim. Fuzzy near-miss findings
(where the actual token differs from the suspected target) require substring
substitution rather than a prefix addition and are routed into
:attr:`AutofixReport.skipped_fuzzy` for manual review.
* By default the rewrite is restricted to *high-confidence* findings only — those
whose suspected target name contains a ``/`` separator or exceeds the
configured length threshold — and skips global memories and read-only memories.
The defaults intentionally err toward false negatives over false positives.
:param include_flat_names: if True, also rewrite low-confidence findings (flat,
short memory names). Increases recall but markedly raises false-positive risk.
:param include_read_only: if True, also rewrite occurrences inside read-only
memories. Use with care, as read-only memories are typically considered
authoritative.
:param include_global: if True, also rewrite occurrences inside global memories.
Modifying a global memory affects every project that consumes it.
:param dry_run: if True, the report describes the rewrites that *would* be applied
but no files are modified.
:return: an :class:`AutofixReport` describing every replacement (made or planned)
and every warning that was deliberately skipped.
"""
report = AutofixReport(dry_run=dry_run)
# build the integrity report first so autofix decisions are made on the same data
validation = self.validate_referential_integrity()
# combine the warning groups according to the include_flat_names policy
warnings: list[UnmarkedReferenceWarning] = list(validation.high_confidence_unmarked_memories)
if include_flat_names:
warnings.extend(validation.low_confidence_unmarked_memories)
else:
report.skipped_flat.extend(validation.low_confidence_unmarked_memories)
# apply replacements per source memory, grouping warnings to avoid re-reading content
warnings_by_source: dict[str, list[UnmarkedReferenceWarning]] = {}
for w in warnings:
# fuzzy near-misses are never auto-rewritten — the bare text differs from the target
if not w.is_exact_match:
report.skipped_fuzzy.append(w)
continue
# filter remaining warnings according to scope flags
if w.source_is_read_only and not include_read_only:
report.skipped_read_only.append(w)
continue
if self._manager._is_global(w.source_memory) and not include_global:
report.skipped_global.append(w)
continue
warnings_by_source.setdefault(w.source_memory, []).append(w)
for source_memory, source_warnings in warnings_by_source.items():
content = self._manager.load_memory(source_memory)
total_n = 0
per_target_records: list[AutofixedReference] = []
# apply replacements sequentially; each pass uses the (potentially updated) content
for w in source_warnings:
content, n = add_bare_occurrences_prefix(content, w.suspected_name)
if n > 0:
per_target_records.append(
AutofixedReference(
source_memory=source_memory,
referenced_name=w.suspected_name,
n_replacements=n,
)
)
total_n += n
if total_n > 0:
if not dry_run:
# use is_tool_context=False so we don't trip read-only protection when --include-read-only is set
self._manager.save_memory(source_memory, content, is_tool_context=False)
report.autofixed.extend(per_target_records)
return report
+7 -242
View File
@@ -1,11 +1,8 @@
import logging
import os
import re
import shutil
import threading
from collections.abc import Sequence
from pathlib import Path
from typing import TYPE_CHECKING, Any, Literal, Optional
from typing import TYPE_CHECKING, Any, Optional
import pathspec
from sensai.util.logging import LogTime
@@ -14,12 +11,11 @@ from sensai.util.string import TextBuilder, ToStringMixin
from serena.config.serena_config import (
ProjectConfig,
SerenaConfig,
SerenaPaths,
)
from serena.constants import SERENA_FILE_ENCODING
from serena.ls_manager import LanguageServerFactory, LanguageServerManager
from serena.memories.memory_manager import MemoryManager
from serena.util.file_system import GitignoreParser, match_path
from serena.util.text_utils import ContentReplacer, MatchedConsecutiveLines, search_files
from serena.util.text_utils import MatchedConsecutiveLines, search_files
from solidlsp import SolidLanguageServer
from solidlsp.ls_config import Language
from solidlsp.ls_utils import FileUtils
@@ -30,240 +26,6 @@ if TYPE_CHECKING:
log = logging.getLogger(__name__)
class MemoriesManager:
GLOBAL_TOPIC = "global"
_global_memory_dir = SerenaPaths().global_memories_path
def __init__(
self,
serena_data_folder: str | Path | None,
read_only_memory_patterns: Sequence[str] = (),
ignored_memory_patterns: Sequence[str] = (),
):
"""
:param serena_data_folder: the absolute path to the project's .serena data folder
:param read_only_memory_patterns: whether to allow writing global memories in tool execution contexts
:param ignored_memory_patterns: regex patterns for memories to completely exclude from listing, reading, and writing.
Matching memories will not appear in list_memories or activate_project output and cannot be accessed
via read_memory or write_memory. Use read_file on the raw path to access ignored memory files.
"""
self._project_memory_dir: Path | None = None
if serena_data_folder is not None:
self._project_memory_dir = Path(serena_data_folder) / "memories"
self._project_memory_dir.mkdir(parents=True, exist_ok=True)
self._encoding = SERENA_FILE_ENCODING
self._read_only_memory_patterns = [re.compile(pattern) for pattern in set(read_only_memory_patterns)]
self._ignored_memory_patterns = [re.compile(pattern) for pattern in set(ignored_memory_patterns)]
def _is_read_only_memory(self, name: str) -> bool:
for pattern in self._read_only_memory_patterns:
if pattern.fullmatch(name):
return True
return False
def _is_ignored_memory(self, name: str) -> bool:
for pattern in self._ignored_memory_patterns:
if pattern.fullmatch(name):
return True
return False
def _check_not_ignored(self, name: str) -> None:
if self._is_ignored_memory(name):
raise ValueError(
f"Memory '{name}' matches an ignored_memory_patterns pattern and cannot be accessed. "
f"Use the read_file tool on the raw file path instead."
)
def _is_global(self, name: str) -> bool:
return name == self.GLOBAL_TOPIC or name.startswith(self.GLOBAL_TOPIC + "/")
def get_memory_file_path(self, name: str) -> Path:
# Strip .md extension if present
name = name.replace(".md", "").replace(os.sep, "/")
parts = name.split("/")
if ".." in parts:
raise ValueError(f"Memory name cannot contain '..' segments for security reasons. Got: {name}")
if self._is_global(name):
if name == self.GLOBAL_TOPIC:
raise ValueError(
f'Bare "{self.GLOBAL_TOPIC}" is not a valid memory name. Use "{self.GLOBAL_TOPIC}/<name>" to address a global memory.'
)
# Strip "global/" prefix and resolve against global dir
sub_name = name[len(self.GLOBAL_TOPIC) + 1 :]
parts = sub_name.split("/")
filename = f"{parts[-1]}.md"
if len(parts) > 1:
subdir = self._global_memory_dir / "/".join(parts[:-1])
subdir.mkdir(parents=True, exist_ok=True)
return subdir / filename
return self._global_memory_dir / filename
# Project-local memory
assert self._project_memory_dir is not None, "Project dir was not passed at initialization"
filename = f"{parts[-1]}.md"
if len(parts) > 1:
# Create subdirectory path
subdir = self._project_memory_dir / "/".join(parts[:-1])
subdir.mkdir(parents=True, exist_ok=True)
return subdir / filename
return self._project_memory_dir / filename
def _check_write_access(self, name: str, is_tool_context: bool) -> None:
# in tool context, memories can be read-only
if is_tool_context and self._is_read_only_memory(name):
raise PermissionError(f"Attempted to write to read_only memory: '{name}')")
def load_memory(self, name: str) -> str:
self._check_not_ignored(name)
memory_file_path = self.get_memory_file_path(name)
if not memory_file_path.exists():
return f"Memory file {name} not found, consider creating it with the `write_memory` tool if you need it."
with open(memory_file_path, encoding=self._encoding) as f:
return f.read()
def save_memory(self, name: str, content: str, is_tool_context: bool) -> str:
self._check_not_ignored(name)
self._check_write_access(name, is_tool_context)
memory_file_path = self.get_memory_file_path(name)
with open(memory_file_path, "w", encoding=self._encoding) as f:
f.write(content)
return f"Memory {name} written."
class MemoriesList:
def __init__(self) -> None:
self.memories: list[str] = []
self.read_only_memories: list[str] = []
def __len__(self) -> int:
return len(self.memories) + len(self.read_only_memories)
def add(self, memory_name: str, is_read_only: bool) -> None:
if is_read_only:
self.read_only_memories.append(memory_name)
else:
self.memories.append(memory_name)
def extend(self, other: "MemoriesManager.MemoriesList") -> None:
self.memories.extend(other.memories)
self.read_only_memories.extend(other.read_only_memories)
def to_dict(self) -> dict[str, list[str]]:
result = {}
if self.memories:
result["memories"] = sorted(self.memories)
if self.read_only_memories:
result["read_only_memories"] = sorted(self.read_only_memories)
return result
def get_full_list(self) -> list[str]:
return sorted(self.memories + self.read_only_memories)
def _list_memories(self, search_dir: Path, base_dir: Path, prefix: str = "") -> MemoriesList:
result = self.MemoriesList()
if not search_dir.exists():
return result
for md_file in search_dir.rglob("*.md"):
rel = str(md_file.relative_to(base_dir).with_suffix("")).replace(os.sep, "/")
memory_name = prefix + rel
if self._is_ignored_memory(memory_name):
continue
result.add(memory_name, is_read_only=self._is_read_only_memory(memory_name))
return result
def list_global_memories(self, subtopic: str = "") -> MemoriesList:
dir_path = self._global_memory_dir
if subtopic:
dir_path = dir_path / subtopic.replace("/", os.sep)
return self._list_memories(dir_path, self._global_memory_dir, self.GLOBAL_TOPIC + "/")
def list_project_memories(self, topic: str = "") -> MemoriesList:
assert self._project_memory_dir is not None, "Project dir was not passed at initialization"
dir_path = self._project_memory_dir
if topic:
dir_path = dir_path / topic.replace("/", os.sep)
return self._list_memories(dir_path, self._project_memory_dir)
def list_memories(self, topic: str = "") -> MemoriesList:
"""
Lists all memories, optionally filtered by topic.
If the topic is omitted, both global and project-specific memories are returned.
"""
memories: MemoriesManager.MemoriesList
if topic:
if self._is_global(topic):
topic_parts = topic.split("/")
subtopic = "/".join(topic_parts[1:])
memories = self.list_global_memories(subtopic=subtopic)
else:
memories = self.list_project_memories(topic=topic)
else:
memories = self.list_project_memories()
memories.extend(self.list_global_memories())
return memories
def delete_memory(self, name: str, is_tool_context: bool) -> str:
self._check_not_ignored(name)
self._check_write_access(name, is_tool_context)
memory_file_path = self.get_memory_file_path(name)
if not memory_file_path.exists():
return f"Memory {name} not found."
memory_file_path.unlink()
return f"Memory {name} deleted."
def move_memory(self, old_name: str, new_name: str, is_tool_context: bool) -> str:
"""
Rename or move a memory file.
Moving between global and project scope (e.g. "global/foo" -> "bar") is supported.
"""
self._check_not_ignored(old_name)
self._check_not_ignored(new_name)
self._check_write_access(new_name, is_tool_context)
old_path = self.get_memory_file_path(old_name)
new_path = self.get_memory_file_path(new_name)
if not old_path.exists():
raise FileNotFoundError(f"Memory {old_name} not found.")
if new_path.exists():
raise FileExistsError(f"Memory {new_name} already exists.")
new_path.parent.mkdir(parents=True, exist_ok=True)
shutil.move(old_path, new_path)
return f"Memory renamed from {old_name} to {new_name}."
def edit_memory(
self, name: str, needle: str, repl: str, mode: Literal["literal", "regex"], allow_multiple_occurrences: bool, is_tool_context: bool
) -> str:
"""
Edit a memory by replacing content matching a pattern.
:param name: the memory name
:param needle: the string or regex to search for
:param repl: the replacement string
:param mode: "literal" or "regex"
:param allow_multiple_occurrences:
"""
self._check_not_ignored(name)
self._check_write_access(name, is_tool_context)
memory_file_path = self.get_memory_file_path(name)
if not memory_file_path.exists():
raise FileNotFoundError(f"Memory {name} not found.")
with open(memory_file_path, encoding=self._encoding) as f:
original_content = f.read()
replacer = ContentReplacer(mode=mode, allow_multiple_occurrences=allow_multiple_occurrences)
updated_content = replacer.replace(original_content, needle, repl)
with open(memory_file_path, "w", encoding=self._encoding) as f:
f.write(updated_content)
return f"Memory {name} edited successfully."
class Project(ToStringMixin):
def __init__(
self,
@@ -282,7 +44,7 @@ class Project(ToStringMixin):
read_only_memory_patterns = serena_config.read_only_memory_patterns + project_config.read_only_memory_patterns
ignored_memory_patterns = serena_config.ignored_memory_patterns + project_config.ignored_memory_patterns
self.memories_manager = MemoriesManager(
self.memory_manager = MemoryManager(
self._serena_data_folder,
read_only_memory_patterns=read_only_memory_patterns,
ignored_memory_patterns=ignored_memory_patterns,
@@ -582,6 +344,7 @@ class Project(ToStringMixin):
context_lines_after: int = 0,
paths_include_glob: str | None = None,
paths_exclude_glob: str | None = None,
multiline: bool = True,
) -> list[MatchedConsecutiveLines]:
"""
Search for a pattern across all (non-ignored) source files
@@ -592,6 +355,7 @@ class Project(ToStringMixin):
:param context_lines_after: Number of lines of context to include after each match
:param paths_include_glob: Glob pattern to filter which files to include in the search
:param paths_exclude_glob: Glob pattern to filter which files to exclude from the search. Takes precedence over paths_include_glob.
:param multiline: Whether to compile the regex with the DOTALL flag (``.`` matches newlines).
:return: List of matched consecutive lines with context
"""
relative_file_paths = self.gather_source_files(relative_path=relative_path)
@@ -604,6 +368,7 @@ class Project(ToStringMixin):
context_lines_after=context_lines_after,
paths_include_glob=paths_include_glob,
paths_exclude_glob=paths_exclude_glob,
multiline=multiline,
)
def retrieve_content_around_line(
@@ -9,4 +9,3 @@ excluded_tools:
- rename_memory
- list_memories
- onboarding
- check_onboarding_performed
@@ -3,4 +3,3 @@ prompt: |
The onboarding process is not applied.
excluded_tools:
- onboarding
- check_onboarding_performed
@@ -4,33 +4,38 @@
prompts:
onboarding_prompt: |
You are viewing the project for the first time.
Your task is to assemble relevant high-level information about the project which
will be saved to memory files in the following steps.
The information should be sufficient to understand what the project is about,
and the most important commands for developing code.
Your task is to assemble durable, non-obvious information about the project and write it
to memory files that future agents will consult.
The project is being developed on the system: {{ system }}.
You need to identify at least the following information:
* the project's purpose
* the tech stack used
* the code style and conventions used (including naming, type hints, docstrings, etc.)
* which commands to run when a task is completed (linting, formatting, testing, etc.)
* the rough structure of the codebase
* the commands for testing, formatting, and linting
* the commands for running the entrypoints of the project
* the util commands for the system, like `git`, `ls`, `cd`, `grep`, `find`, etc. Keep in mind that the system is {{ system }},
so the commands might be different than on a regular unix system.
* whether there are particular guidelines, styles, design patterns, etc. that one should know about
**Before writing anything, read `mem:{{ memory_maintenance_name }}`** using the `read_memory` tool.
It defines the style, naming, reference, and add/update threshold conventions that every
memory in this project must follow. Do not skip this step.
This list is not exhaustive, you can add more information if you think it is relevant.
Target memory layout — use the `write_memory` tool, one call per memory:
For doing that, you will need to acquire information about the project with the corresponding tools.
Read only the necessary files and directories to avoid loading too much data into memory.
If you cannot find everything you need from the project itself, you should ask the user for more information.
* `mem:core` — top-level source map and project-wide invariants that don't belong in a
focused memory.
* `mem:tech_stack` — language(s), framework(s), build tools, package manager, version
pins where they matter.
* `mem:suggested_commands` — project commands the user will actually run (dev, test, lint,
format, run entrypoints) and any system util commands (`git`, `ls`, `grep`, …) whose
form differs on {{ system }} from a standard unix shell. Skip generic commands that
behave identically across systems.
* `mem:conventions` — code style, naming, type hints, docstring conventions, design
patterns specific to this codebase.
* `mem:task_completion` — exact commands to run when a coding task is considered done
(linter, formatter, test runner, type checker, etc.).
After collecting all the information, you will use the `write_memory` tool (in multiple calls) to save it to various memory files.
A particularly important memory file will be the `suggested_commands.md` file, which should contain
a list of commands that the user should know about to develop code in this project.
Moreover, you should create memory files for the style and conventions and a dedicated memory file for
what should be done when a task is completed.
**Important**: after done with the onboarding task, remember to call the `write_memory` to save the collected information!
If the project has clearly distinct modules (e.g. frontend/backend), create per-module
`mem:<module>/core` memories with module-specific references to further memories instead
of pushing everything into `mem:core`.
Acquire information using the available tools. Read only the files needed; do not load
entire directory trees. If essential information is missing, ask the user.
When all memories are written, sanity-check the references by mentioning that the user can run
`serena memories check` from the project root.
**Important**: the onboarding is only complete once you have actually called `write_memory`
for each memory above — do not summarize the content in chat and skip writing.
@@ -0,0 +1,33 @@
# Memory Maintenance
## Discovery Model
- Core principle: progressive discovery through references, building a graph of memories.
- Initially, agents are provided with the list of all memories (names only).
- Agents should read `mem:core` as the top-level entry point (graph root).
This memory should contain references to other memories covering major project domains.
The referenced memories shall, in turn, shall contain references to even more specific memories, and so on.
The depth of the graph shall depend on the project complexity.
- Use topics/folders to group related memories in order to make the content structure explicit.
Folders can mirror project structure (e.g. modules like frontend/backend) or topics like debugging, architecture, etc.
- Memory references must use a mem: prefix inside backticks, e.g. `mem:frontend/core`.
The surrounding text should clearly indicate when to read the memory/which content to expect.
The text should provide more precise guidance than the memory name alone,
i.e. avoid a reference like "frontend debugging: `mem:frontend/debugging` and instead make clear which aspects of frontend debugging are covered.
- Memories themselves should not contain information about when to read them; this is the responsibility of the referring memory.
## Style
Dense agent notes, not prose docs. Prefer invariants, terse bullets.
Avoid obvious context, rationale, and examples unless they prevent likely mistakes.
Keep guidance durable and generalizable, not task-local.
## Add/update threshold
Add or update memories only with stable, non-obvious project conventions that avoid complex rediscovery in the future.
Do not add: quick-read facts; generic language/framework knowledge; one-off task notes; volatile line-level details; behavior likely to change soon.
## Maintenance Actions
- Renaming memories: References are updated automatically if handled via Serena's memory rename tool.
- Checking for stale memories (e.g. after deletion): Call `serena memories check` for a report.
+15 -33
View File
@@ -331,44 +331,24 @@ class SearchForPatternTool(Tool):
paths_exclude_glob: str = "",
relative_path: str = "",
restrict_search_to_code_files: bool = False,
multiline: bool = True,
max_answer_chars: int = -1,
) -> str:
"""
Offers a flexible search for arbitrary patterns in the codebase, including the
possibility to search in non-code files.
Searches for a regex pattern across project files, returning whole matched lines (plus optional context).
Prefer symbolic operations if you know which symbols you are looking for!
Pattern Matching Logic:
For each match, the returned result will contain the full lines where the
substring pattern is found, as well as optionally some lines before and after it. The pattern will be compiled with
DOTALL, meaning that the dot will match all characters including newlines.
This also means that it never makes sense to have .* at the beginning or end of the pattern,
but it may make sense to have it in the middle for complex patterns.
If a pattern matches multiple lines, all those lines will be part of the match.
Be careful to not use greedy quantifiers unnecessarily, it is usually better to use non-greedy quantifiers like .*? to avoid
matching too much content.
:param substring_pattern: Regular expression for a substring pattern to search for
:param context_lines_before: Number of lines of context to include before each match
:param context_lines_after: Number of lines of context to include after each match
:param paths_include_glob: optional glob pattern specifying files to include in the search.
Matches against relative file paths from the project root (e.g., "*.py", "src/**/*.ts").
Supports standard glob patterns (*, ?, [seq], **, etc.) and brace expansion {a,b,c}.
Only matches files, not directories. If left empty, all non-ignored files will be included.
:param paths_exclude_glob: optional glob pattern specifying files to exclude from the search.
Matches against relative file paths from the project root (e.g., "*test*", "**/*_generated.py").
Supports standard glob patterns (*, ?, [seq], **, etc.) and brace expansion {a,b,c}.
Takes precedence over paths_include_glob. Only matches files, not directories. If left empty, no files are excluded.
:param relative_path: only subpaths of this path (relative to the repo root) will be analyzed. If a path to a single
file is passed, only that will be searched. The path must exist, otherwise a `FileNotFoundError` is raised.
:param max_answer_chars: if the output is longer than this number of characters,
no content will be returned.
-1 means the default value from the config will be used.
Don't adjust unless there is really no other way to get the content
required for the task. Instead, if the output is too long, you should
make a stricter query.
:param restrict_search_to_code_files: whether to restrict the search to source files.
Otherwise, will search all non-ignored files (default).
:param substring_pattern: regular expression to search for.
:param context_lines_before: number of context lines to include before each match.
:param context_lines_after: number of context lines to include after each match.
:param paths_include_glob: optional glob (relative to project root, e.g. ``"src/**/*.ts"``) restricting which files are searched.
:param paths_exclude_glob: optional glob to exclude files; takes precedence over `paths_include_glob`.
:param relative_path: restricts the search to this file or subdirectory of the project root
:param restrict_search_to_code_files: whether to search only files containing analyzable code symbols
(useful when looking for class/method definitions); otherwise also search non-code files.
:param multiline: whether to apply multi-line matching (default: True), enabling the flags re.DOTALL and re.MULTILINE
:param max_answer_chars: if the output exceeds this many characters, a progressively shortened summary is returned instead.
``-1`` uses the configured default.
:return: A mapping from file paths to matched consecutive lines (0-based line numbers).
"""
relative_path = relative_path.strip()
@@ -387,6 +367,7 @@ class SearchForPatternTool(Tool):
context_lines_after=context_lines_after,
paths_include_glob=paths_include_glob.strip(),
paths_exclude_glob=paths_exclude_glob.strip(),
multiline=multiline,
)
else:
if os.path.isfile(abs_path):
@@ -410,6 +391,7 @@ class SearchForPatternTool(Tool):
root_path=self.get_project_root(),
paths_include_glob=paths_include_glob,
paths_exclude_glob=paths_exclude_glob,
multiline=multiline,
)
# group matches by file
+34 -27
View File
@@ -1,7 +1,10 @@
import logging
from typing import Literal
from serena.tools import Tool, ToolMarkerCanEdit
log = logging.getLogger(__name__)
class WriteMemoryTool(Tool, ToolMarkerCanEdit):
"""
@@ -11,13 +14,15 @@ class WriteMemoryTool(Tool, ToolMarkerCanEdit):
def apply(self, memory_name: str, content: str, max_chars: int = -1) -> str:
"""
Write information (utf-8-encoded) about this project that can be useful for future tasks to a memory in md format.
The memory name should be meaningful and can include "/" to organize into topics (e.g., "auth/login/logic").
If explicitly instructed, use the "global/" prefix for writing a memory that is shared across projects
(e.g., "global/java/style_guide")
Write information about this project that can be useful for future tasks in md format.
The name should be meaningful and can include "/" to organize into topics.
If explicitly instructed, use the "global/" prefix for writing a memory that is shared across projects.
References to other memories should be inside backticks and prefixed with mem:,
e.g., `mem:auth`.
:param max_chars: the maximum number of characters to write. By default, determined by the config,
change only if instructed to do so.
:param memory_name: memory name
:param content: memory content, utf8-encoded
:param max_chars: see other tools
"""
# NOTE: utf-8 encoding is configured in the MemoriesManager
if max_chars == -1:
@@ -27,63 +32,63 @@ class WriteMemoryTool(Tool, ToolMarkerCanEdit):
f"Content for {memory_name} is too long. Max length is {max_chars} characters. " + "Please make the content shorter."
)
return self.memories_manager.save_memory(memory_name, content, is_tool_context=True)
return self.memory_manager.save_memory(memory_name, content, is_tool_context=True)
class ReadMemoryTool(Tool):
"""
Read the content of a memory file. This tool should only be used if the information
is relevant to the current task. You can infer whether the information
is relevant from the memory file name.
You should not read the same memory file multiple times in the same conversation.
Reads the content of a memory file.
"""
def apply(self, memory_name: str) -> str:
"""
Reads the contents of a memory. Should only be used if the information
is likely to be relevant to the current task, inferring relevance from the memory name.
Use to read a memory that is likely to be relevant to the current task, inferring relevance e.g. from the name.
"""
return self.memories_manager.load_memory(memory_name)
return self.memory_manager.load_memory(memory_name)
class ListMemoriesTool(Tool):
"""
List available memories. Any memory can be read using the `read_memory` tool.
Lists available memories.
"""
def apply(self, topic: str = "") -> str:
"""
Lists available memories, optionally filtered by topic.
"""
return self._to_json(self.memories_manager.list_memories(topic).to_dict())
return self._to_json(self.memory_manager.list_memories(topic).to_dict())
class DeleteMemoryTool(Tool, ToolMarkerCanEdit):
"""
Delete a memory file. Should only happen if a user asks for it explicitly,
for example by saying that the information retrieved from a memory file is no longer correct
or no longer relevant for the project.
Delete a memory file.
"""
def apply(self, memory_name: str) -> str:
"""
Delete a memory, only call if instructed explicitly or permission was granted by the user.
"""
return self.memories_manager.delete_memory(memory_name, is_tool_context=True)
return self.memory_manager.delete_memory(memory_name, is_tool_context=True)
class RenameMemoryTool(Tool, ToolMarkerCanEdit):
"""
Renames or moves a memory. Moving between project and global scope is supported
(e.g., renaming "global/foo" to "bar" moves it from global to project scope).
Renames or moves a memory, updating references that are marked with the `mem:` prefix.
"""
def apply(self, old_name: str, new_name: str) -> str:
"""
Rename or move a memory, use "/" in the name to organize into topics.
The "global" topic should only be used if explicitly instructed.
References to other memories that are marked with the `mem:` prefix will be updated accordingly.
References in read-only memories are not affected.
"""
return self.memories_manager.move_memory(old_name, new_name, is_tool_context=True)
renaming_message, n_references_updated = self.memory_manager.rename_memory_and_propagate_references(
old_name, new_name, is_tool_context=True
)
if n_references_updated > 0:
log.info(f"Updated {n_references_updated} references to memory {old_name} to {new_name}")
return renaming_message
class EditMemoryTool(Tool, ToolMarkerCanEdit):
@@ -100,16 +105,18 @@ class EditMemoryTool(Tool, ToolMarkerCanEdit):
allow_multiple_occurrences: bool = False,
) -> str:
r"""
Replaces content matching a regular expression in a memory.
Replace content matching a regular expression in a memory.
:param memory_name: the name of the memory
:param needle: the string or regex pattern to search for.
:param needle: the string or regex pattern to search for. In regex mode, be careful to not replace too much!
If `mode` is "literal", this string will be matched exactly.
If `mode` is "regex", this string will be treated as a regular expression (syntax of Python's `re` module,
with flags DOTALL and MULTILINE enabled).
with the MULTILINE and DOTALL flags enabled).
:param repl: the replacement string (verbatim).
:param mode: either "literal" or "regex", specifying how the `needle` parameter is to be interpreted.
:param allow_multiple_occurrences: whether to allow matching and replacing multiple occurrences.
If false and multiple occurrences are found, an error will be returned.
"""
return self.memories_manager.edit_memory(memory_name, needle, repl, mode, allow_multiple_occurrences, is_tool_context=True)
return self.memory_manager.edit_memory(
memory_name, needle, repl, mode, allow_multiple_occurrences, is_tool_context=True, regex_multiline=True
)
+5 -3
View File
@@ -14,7 +14,8 @@ from sensai.util import logging
from sensai.util.string import dict_string
from serena.config.serena_config import LanguageBackend
from serena.project import MemoriesManager, Project
from serena.memories.memory_manager import MemoryManager
from serena.project import Project
from serena.prompt_factory import PromptFactory
from serena.util.class_decorators import singleton
from serena.util.inspection import iter_subclasses
@@ -46,8 +47,8 @@ class Component(ABC):
return self.agent.prompt_factory
@property
def memories_manager(self) -> "MemoriesManager":
return self.project.memories_manager
def memory_manager(self) -> "MemoryManager":
return self.project.memory_manager
def create_language_server_symbol_retriever(self) -> "LanguageServerSymbolRetriever":
from serena.symbol import LanguageServerSymbolRetriever
@@ -528,6 +529,7 @@ class ToolRegistry:
"summarize_changes",
"think_about_whether_you_are_done",
"switch_modes",
"check_onboarding_performed",
]
def __init__(self) -> None:
+5 -33
View File
@@ -4,38 +4,7 @@ Tools supporting the general workflow of the agent
import platform
from serena.tools import ReadMemoryTool, Tool, ToolMarkerDoesNotRequireActiveProject, ToolMarkerOptional, WriteMemoryTool
class CheckOnboardingPerformedTool(Tool):
"""
Checks whether project onboarding was already performed.
"""
def apply(self) -> str:
"""
Checks whether project onboarding was already performed.
You should always call this tool before beginning to actually work on the project/after activating a project.
"""
read_memory_tool_available = self.agent.tool_is_exposed(ReadMemoryTool.get_name_from_cls())
perform_onboarding_tool_available = self.agent.tool_is_exposed(OnboardingTool.get_name_from_cls())
if not read_memory_tool_available:
return "Memory reading tool not activated, skipping onboarding check."
project_memories = self.memories_manager.list_project_memories()
if len(project_memories) == 0:
msg = "Onboarding not performed yet (no memories available). "
if perform_onboarding_tool_available:
msg += "You should perform onboarding by calling the `onboarding` tool before proceeding with the task. "
else:
# Not reporting the list of memories here, as they were already reported at project activation
# (with the system prompt if the project was activated at startup)
msg = (
f"Onboarding was already performed: {len(project_memories)} project memories are available. "
"Consider reading memories if they appear relevant to the task at hand."
)
msg += " If you have not read the 'Serena Instructions Manual', do so now."
return msg
from serena.tools import Tool, ToolMarkerDoesNotRequireActiveProject, ToolMarkerOptional, WriteMemoryTool
class OnboardingTool(Tool):
@@ -54,7 +23,10 @@ class OnboardingTool(Tool):
if not write_memory_tool_available:
return "Memory writing tool not activated, skipping onboarding."
system = platform.system()
return self.prompt_factory.create_onboarding_prompt(system=system)
# seed the project-local memory-maintenance memory (or detect a global override) so
# the prompt can point the agent at the conventions before it writes anything
memory_maintenance_name = self.memory_manager.ensure_memory_maintenance_memory()
return self.prompt_factory.create_onboarding_prompt(system=system, memory_maintenance_name=memory_maintenance_name)
class InitialInstructionsTool(Tool, ToolMarkerDoesNotRequireActiveProject):
+21 -16
View File
@@ -145,6 +145,7 @@ def search_text(
context_lines_before: int = 0,
context_lines_after: int = 0,
is_glob: bool = False,
multiline: bool = True,
) -> list[MatchedConsecutiveLines]:
"""
Search for a pattern in text content. Supports both regex and glob-like patterns.
@@ -159,11 +160,9 @@ def search_text(
:param context_lines_after: Number of context lines to include after matches
:param is_glob: If True, pattern is treated as a glob-like pattern (e.g., "*.py", "test_??.py")
and will be converted to regex internally
:param multiline: whether to apply multi-line matching, enabling the flags re.DOTALL and re.MULTILINE
:return: List of `TextSearchMatch` objects
:raises: ValueError if the pattern is not valid
"""
if source_file_path and content is None:
with open(source_file_path) as f:
@@ -180,8 +179,9 @@ def search_text(
if is_glob:
pattern = glob_to_regex(pattern)
if allow_multiline_match:
# For multiline matches, we need to use the DOTALL flag to make '.' match newlines
compiled_pattern = re.compile(pattern, re.DOTALL)
# For multiline matches, optionally use DOTALL so '.' matches newlines
flags = (re.MULTILINE | re.DOTALL) if multiline else 0
compiled_pattern = re.compile(pattern, flags)
# Search across the entire content as a single string
for match in compiled_pattern.finditer(content):
start_pos = match.start()
@@ -327,20 +327,22 @@ def search_files(
context_lines_after: int = 0,
paths_include_glob: str | None = None,
paths_exclude_glob: str | None = None,
multiline: bool = True,
) -> list[MatchedConsecutiveLines]:
"""
Search for a pattern in a list of files.
:param relative_file_paths: List of relative file paths in which to search
:param pattern: Pattern to search for
:param root_path: Root path to resolve relative paths against (by default, current working directory).
:param file_reader: Function to read a file, by default will just use os.open.
:param relative_file_paths: list of relative file paths in which to search
:param pattern: pattern to search for
:param root_path: root path to resolve relative paths against (by default, current working directory).
:param file_reader: function to read a file, by default will just use os.open.
All files that can't be read by it will be skipped.
:param context_lines_before: Number of context lines to include before matches
:param context_lines_after: Number of context lines to include after matches
:param paths_include_glob: Optional glob pattern to include files from the list
:param paths_exclude_glob: Optional glob pattern to exclude files from the list
:return: List of MatchedConsecutiveLines objects
:param context_lines_before: number of context lines to include before matches
:param context_lines_after: number of context lines to include after matches
:param paths_include_glob: optional glob pattern to include files from the list
:param paths_exclude_glob: optional glob pattern to exclude files from the list
:param multiline: whether to apply multi-line matching, enabling the flags re.DOTALL and re.MULTILINE (default: True)
:return: list of MatchedConsecutiveLines objects
"""
# Pre-filter paths (done sequentially to avoid overhead)
# Use proper glob matching instead of gitignore patterns
@@ -375,6 +377,7 @@ def search_files(
allow_multiline_match=True,
context_lines_before=context_lines_before,
context_lines_after=context_lines_after,
multiline=multiline,
)
if len(search_results) > 0:
log.debug(f"Found {len(search_results)} matches in {path}")
@@ -430,16 +433,18 @@ class ContentReplacer:
provides dual modes for maximum flexibility.
"""
def __init__(self, mode: Literal["literal", "regex"], allow_multiple_occurrences: bool):
def __init__(self, mode: Literal["literal", "regex"], allow_multiple_occurrences: bool, regex_multiline: bool = True):
"""
:param mode: the mode indicating whether to the needle in replacements corresponds to a regular expression
(mode "regex") or to a literal string (mode "literal")
:param allow_multiple_occurrences: whether it is allowed that the search expression matches multiple occurrences.
If False, an error will be raised if more than one match is found.
:param regex_multiline: whether to apply multi-line regex matching, enabling the flags re.DOTALL and re.MULTILINE
"""
self.mode = mode
self.allow_multiple_occurrences = allow_multiple_occurrences
self.regex_multiline = regex_multiline
@staticmethod
def _create_replacement_function(regex_pattern: str, repl_template: str, regex_flags: int) -> Callable[[re.Match], str]:
@@ -505,7 +510,7 @@ class ContentReplacer:
else:
raise ValueError(f"Invalid mode: '{self.mode}', expected 'literal' or 'regex'.")
regex_flags = re.DOTALL | re.MULTILINE
regex_flags = (re.MULTILINE | re.DOTALL) if self.regex_multiline else 0
# create replacement function with validation and backreference handling
repl_fn = self._create_replacement_function(regex, repl, regex_flags=regex_flags)
@@ -7,7 +7,6 @@
# execute `uv run scripts/print_tool_overview.py`.
#
# * `activate_project`: Activates a project by name.
# * `check_onboarding_performed`: Checks whether project onboarding was already performed.
# * `create_text_file`: Creates/overwrites a file in the project directory.
# * `delete_lines`: Deletes a range of lines within a file.
# * `delete_memory`: Deletes a memory from Serena's project-specific memory store.
+4 -4
View File
@@ -16,7 +16,7 @@ from serena.config.serena_config import (
SerenaConfigError,
)
from serena.constants import PROJECT_TEMPLATE_FILE, SERENA_MANAGED_DIR_NAME
from serena.project import MemoriesManager, Project
from serena.project import MemoryManager, Project
from solidlsp.ls_config import Language
from test.conftest import create_default_serena_config
@@ -581,17 +581,17 @@ class TestMemoriesManagerCustomPath:
def test_memories_subdir_is_created(self):
assert not self.data_folder.exists()
MemoriesManager(str(self.data_folder))
MemoryManager(str(self.data_folder))
assert (self.data_folder / "memories").exists()
def test_save_and_load_memory(self):
manager = MemoriesManager(str(self.data_folder))
manager = MemoryManager(str(self.data_folder))
manager.save_memory("test_topic", "test content", is_tool_context=False)
content = manager.load_memory("test_topic")
assert content == "test content"
def test_list_memories(self):
manager = MemoriesManager(str(self.data_folder))
manager = MemoryManager(str(self.data_folder))
manager.save_memory("topic_a", "content a", is_tool_context=False)
manager.save_memory("topic_b", "content b", is_tool_context=False)
memories = manager.list_project_memories()
+686
View File
@@ -0,0 +1,686 @@
"""
Unit tests for :class:`serena.project.MemoriesManager` helpers that do not
require touching the filesystem (in particular, :meth:`rename_references_to_memory`
and :meth:`_prepare_name`).
"""
import os
import pytest
from serena.memories.memory_reference_analysis import (
HIGH_CONFIDENCE_NAME_LENGTH,
MAX_STALE_REFERENCE_CANDIDATES,
NAME_SIMILARITY_THRESHOLD,
compute_name_similarity,
find_stale_reference_candidates,
)
from serena.project import MemoryManager
@pytest.fixture
def manager() -> MemoryManager:
# serena_data_folder=None: the tested helpers do not access the filesystem
return MemoryManager(serena_data_folder=None)
class TestPrepareName:
@pytest.mark.parametrize(
"name",
[
"memory",
"emergency",
"meeting_notes",
"em-dash",
"ml_notes",
"embedding",
"e",
"m",
"mem",
":colon_prefixed_unlikely",
],
)
def test_does_not_strip_partial_prefix_chars_from_real_names(self, manager: MemoryManager, name: str) -> None:
"""A regression guard: previously used ``str.lstrip("mem:")`` which strips any leading m/e/:."""
assert manager._sanitize_name(name) == name
@pytest.mark.parametrize(
("input_name", "expected"),
[
("mem:foo", "foo"),
("mem:foo/bar", "foo/bar"),
("mem:memory", "memory"), # prefix is removed only once, not greedily
("mem:", ""),
],
)
def test_strips_mem_prefix_only_when_actually_a_prefix(self, manager: MemoryManager, input_name: str, expected: str) -> None:
assert manager._sanitize_name(input_name) == expected
@pytest.mark.parametrize(
("input_name", "expected"),
[
("foo.md", "foo"),
("topic/foo.md", "topic/foo"),
("a.md.notes", "a.md.notes"), # ".md" must only be stripped as trailing extension
("foo.markdown", "foo.markdown"),
(".md", ""),
],
)
def test_strips_md_only_as_suffix(self, manager: MemoryManager, input_name: str, expected: str) -> None:
assert manager._sanitize_name(input_name) == expected
def test_normalizes_os_separator(self, manager: MemoryManager) -> None:
assert manager._sanitize_name(f"topic{os.sep}sub{os.sep}name") == "topic/sub/name"
def test_combined_normalizations(self, manager: MemoryManager) -> None:
assert manager._sanitize_name(f"mem:topic{os.sep}foo.md") == "topic/foo"
class TestRenameReferencesToMemory:
@pytest.mark.parametrize(
("content", "old_name", "new_name", "expected_content", "expected_count"),
[
# simple cases
pytest.param("see mem:foo for details", "foo", "bar", "see mem:bar for details", 1, id="simple"),
pytest.param("nothing to see here", "foo", "bar", "nothing to see here", 0, id="no_match"),
pytest.param(
"mem:foo and mem:foo and `mem:foo`",
"foo",
"bar",
"mem:bar and mem:bar and `mem:bar`",
3,
id="multiple_occurrences",
),
# boundaries
pytest.param("mem:foo rest", "foo", "bar", "mem:bar rest", 1, id="start_of_string"),
pytest.param("prefix mem:foo", "foo", "bar", "prefix mem:bar", 1, id="end_of_string"),
pytest.param("mem:foo", "foo", "bar", "mem:bar", 1, id="entire_string"),
# must NOT match: embedded in longer name
pytest.param("`mem:foobar`", "foo", "bar", "`mem:foobar`", 0, id="longer_suffix"),
pytest.param("`mem:foo/sub`", "foo", "bar", "`mem:foo/sub`", 0, id="subtopic"),
pytest.param("mem:foo-extra", "foo", "bar", "mem:foo-extra", 0, id="hyphen_suffix"),
pytest.param("mem:foo_extra", "foo", "bar", "mem:foo_extra", 0, id="underscore_suffix"),
pytest.param("xmem:foo", "foo", "bar", "xmem:foo", 0, id="preceded_by_name_char"),
# boundaries on non-name characters
pytest.param("see `mem:foo.md`", "foo", "bar", "see `mem:bar.md`", 1, id="md_extension_suffix"),
# topic renames
pytest.param("`mem:foo/bar`", "foo/bar", "baz", "`mem:baz`", 1, id="topic_to_flat"),
pytest.param("`mem:foo`", "foo/bar", "baz", "`mem:foo`", 0, id="parent_does_not_match_topic"),
pytest.param("`mem:foo`", "foo", "topic/bar", "`mem:topic/bar`", 1, id="flat_to_topic"),
# regex-meta safety
pytest.param("`mem:a.b`", "a.b", "c", "`mem:c`", 1, id="dot_in_old_name_literal"),
pytest.param("`mem:axb`", "a.b", "c", "`mem:axb`", 0, id="dot_in_old_name_not_wildcard"),
# mixed
pytest.param(
"valid `mem:foo`, embedded mem:foobar, subtopic mem:foo/x, and another `mem:foo`.",
"foo",
"bar",
"valid `mem:bar`, embedded mem:foobar, subtopic mem:foo/x, and another `mem:bar`.",
2,
id="mixed_matching_and_nonmatching",
),
# input names that previously broke due to lstrip("mem:") in _add_reference_prefix
pytest.param("see `mem:memory`", "memory", "diary", "see `mem:diary`", 1, id="old_name_starts_with_m"),
pytest.param("see `mem:emergency`", "emergency", "incident", "see `mem:incident`", 1, id="old_name_starts_with_e"),
pytest.param("see `mem:foo`", "foo", "memory", "see `mem:memory`", 1, id="new_name_starts_with_m"),
],
)
def test_rename(
self,
manager: MemoryManager,
content: str,
old_name: str,
new_name: str,
expected_content: str,
expected_count: int,
) -> None:
result, n = manager.rename_references_to_memory(content, old_name, new_name)
assert result == expected_content
assert n == expected_count
@pytest.mark.parametrize(
"delimiter_pair",
[
# one case per structural class of surrounding character (asymmetric pair, whitespace,
# punctuation, none); the boundary regex only distinguishes "name char" vs "non-name char",
# so further variants exercise the same code path.
("`", "`"),
("(", ")"),
(" ", " "),
("", ""), # raw, no delimiters
],
)
def test_replacement_inside_various_delimiters(self, manager: MemoryManager, delimiter_pair: tuple[str, str]) -> None:
left, right = delimiter_pair
content = f"{left}mem:foo{right}"
result, n = manager.rename_references_to_memory(content, "foo", "bar")
assert result == f"{left}mem:bar{right}"
assert n == 1
class TestNameSimilarity:
@pytest.mark.parametrize(
("a", "b", "expected"),
[
# exact match (and case-insensitive) -> 1.0
pytest.param("foo", "foo", 1.0, id="identical"),
pytest.param("Foo", "foo", 1.0, id="case_insensitive"),
# version-suffix renames are normalized away -> 1.0
pytest.param("auth_v1", "auth_v2", 1.0, id="version_suffix_rename"),
pytest.param("auth_old", "auth", 1.0, id="old_suffix_normalized"),
pytest.param("auth_legacy", "auth", 1.0, id="legacy_suffix_normalized"),
# flat <-> topic basename move: canonical rename, fully confident
pytest.param("a/b/c/login", "login", 1.0, id="topic_move_to_flat"),
pytest.param("login", "auth/login", 1.0, id="flat_to_topic_move"),
# both sides share a prefix token AND basename: high confidence
pytest.param("auth/login", "auth/v2/login", 0.75, id="basename_match_shared_prefix"),
],
)
def test_high_similarity_cases(self, manager: MemoryManager, a: str, b: str, expected: float) -> None:
assert compute_name_similarity(a, b) == pytest.approx(expected)
@pytest.mark.parametrize(
("a", "b"),
[
# completely unrelated names should be below threshold
pytest.param("authentication", "database", id="unrelated_long"),
pytest.param("api/users", "logging/handlers", id="unrelated_topic_paths"),
# short names without strong token signals must hit the short-name floor
pytest.param("foo", "bar", id="short_names_unrelated"),
pytest.param("foo", "for", id="short_names_typo_distance"),
# IMPORTANT: same basename under unrelated topic prefixes is NOT a rename.
# `auth/login` and `security/login` are two different login flows in different
# subsystems; `topic1/topic2/core` and `topic3/topic5/core` are unrelated.
pytest.param("auth/login", "security/login", id="same_basename_unrelated_prefixes"),
pytest.param("topic1/topic2/core", "topic3/topic5/core", id="same_basename_disjoint_prefix_tokens"),
],
)
def test_below_threshold_cases(self, manager: MemoryManager, a: str, b: str) -> None:
assert compute_name_similarity(a, b) < NAME_SIMILARITY_THRESHOLD
@pytest.mark.parametrize(
("a", "b"),
[
# prefix/suffix extensions: caught by sequence ratio / containment, even though Jaccard is 0
pytest.param("auth", "authentication", id="prefix_extension"),
pytest.param("login", "user_login", id="token_extension"),
# typos on longer names: caught by sequence ratio
pytest.param("authentication", "autentication", id="typo_in_long_name"),
# token reorder / partial overlap
pytest.param("user_login_flow", "login_flow", id="partial_token_overlap"),
],
)
def test_above_threshold_cases(self, manager: MemoryManager, a: str, b: str) -> None:
assert compute_name_similarity(a, b) >= NAME_SIMILARITY_THRESHOLD
def test_symmetry(self, manager: MemoryManager) -> None:
# similarity is order-independent
assert compute_name_similarity("auth/login", "security/login") == compute_name_similarity("security/login", "auth/login")
def test_cross_topic_suffix_coincidence_is_rejected(self, manager: MemoryManager) -> None:
"""
Regression: when two long names live under *different* topic prefixes and share
only a single trailing token (e.g. ``-subtleties``), the score must collapse to
zero. Without the cross-topic gate, SequenceMatcher returns ~0.55+ on these
purely on shared characters, flooding stale-reference candidate lists with
unrelated memories from neighbouring subsystems.
"""
target = "frontend/plugin-api-runtime-subtleties"
# different topic prefix + only the generic "subtleties" trailing token shared -> 0
for other in (
"backend/rpc-db-worker-subtleties",
"render-wasm/ffi-rendering-subtleties",
"common/grid-layout-subtleties",
):
assert compute_name_similarity(target, other) == 0.0, f"cross-topic suffix coincidence with {other} must be rejected"
# contrast: *same* prefix is treated leniently — the gate must not apply, so the
# score is non-zero even if it still falls under the similarity threshold
same_prefix_score = compute_name_similarity(target, "frontend/routing-app-shell-subtleties")
assert same_prefix_score > 0.0
def test_find_candidates_ranks_by_descending_similarity(self, manager: MemoryManager) -> None:
existing = ["auth/login", "security/login", "database", "authentication"]
candidates = find_stale_reference_candidates("login", existing)
# both auth/login and security/login share basename `login` -> tied at 1.0; expect them first, alphabetical tiebreak
assert candidates[:2] == ["auth/login", "security/login"]
# `database` is unrelated and must not appear
assert "database" not in candidates
def test_find_candidates_returns_empty_below_threshold(self, manager: MemoryManager) -> None:
existing = ["completely_unrelated", "another_thing"]
assert find_stale_reference_candidates("foo", existing) == []
def test_find_candidates_custom_threshold(self, manager: MemoryManager) -> None:
# raising the threshold filters out borderline matches
existing = ["authentication"]
loose = find_stale_reference_candidates("auth", existing, threshold=0.2)
strict = find_stale_reference_candidates("auth", existing, threshold=0.95)
assert loose == ["authentication"]
assert strict == []
@pytest.fixture
def fs_manager(tmp_path) -> MemoryManager:
"""A MemoriesManager backed by tmp_path with no read-only patterns."""
return MemoryManager(serena_data_folder=tmp_path)
def _write(manager: MemoryManager, name: str, content: str) -> None:
manager.save_memory(name, content, is_tool_context=False)
class TestValidateReferentialIntegrity:
def test_clean_report_when_all_references_resolve(self, fs_manager: MemoryManager) -> None:
_write(fs_manager, "auth", "# auth notes")
_write(fs_manager, "login_flow", "see `mem:auth` for details")
report = fs_manager.validate_referential_integrity()
assert report.is_clean()
assert "No referential integrity issues" in report.format()
def test_stale_reference_with_topic_move_candidate(self, fs_manager: MemoryManager) -> None:
# original memory `login` was moved to `auth/login`; a reference still points to bare `login`
_write(fs_manager, "auth/login", "# moved here")
_write(fs_manager, "docs", "old reference: `mem:login`")
report = fs_manager.validate_referential_integrity()
assert len(report.stale_references) == 1
stale = report.stale_references[0]
assert stale.source_memory == "docs"
assert stale.referenced_name == "login"
assert "auth/login" in stale.candidates
def test_stale_reference_with_no_candidate(self, fs_manager: MemoryManager) -> None:
_write(fs_manager, "completely_unrelated", "# content")
_write(fs_manager, "docs", "broken: `mem:xyz_unknown_thing`")
report = fs_manager.validate_referential_integrity()
assert len(report.stale_references) == 1
assert report.stale_references[0].candidates == []
def test_high_confidence_unmarked_warning_for_topic_path(self, fs_manager: MemoryManager) -> None:
_write(fs_manager, "auth/login", "# notes")
_write(fs_manager, "docs", "the auth/login process is documented elsewhere")
report = fs_manager.validate_referential_integrity()
assert len(report.high_confidence_unmarked_memories) == 1
warning = report.high_confidence_unmarked_memories[0]
assert warning.suspected_name == "auth/login"
assert warning.is_high_confidence is True
assert warning.occurrences == 1
assert report.low_confidence_unmarked_memories == []
def test_low_confidence_unmarked_warning_for_short_flat_name(self, fs_manager: MemoryManager) -> None:
_write(fs_manager, "auth", "# notes")
_write(fs_manager, "docs", "you must use auth here")
report = fs_manager.validate_referential_integrity()
assert report.high_confidence_unmarked_memories == []
assert len(report.low_confidence_unmarked_memories) == 1
assert report.low_confidence_unmarked_memories[0].suspected_name == "auth"
assert report.low_confidence_unmarked_memories[0].is_high_confidence is False
def test_long_flat_name_is_high_confidence(self, fs_manager: MemoryManager) -> None:
# a flat name >= _HIGH_CONFIDENCE_NAME_LENGTH chars is unlikely to be coincidental prose
long_name = "authentication_flow"
assert len(long_name) >= HIGH_CONFIDENCE_NAME_LENGTH
_write(fs_manager, long_name, "# notes")
_write(fs_manager, "docs", f"the {long_name} is documented here")
report = fs_manager.validate_referential_integrity()
assert len(report.high_confidence_unmarked_memories) == 1
def test_fuzzy_near_miss_rejects_substring_only_matches(self, fs_manager: MemoryManager) -> None:
"""
Regression: fuzzy bare-text matching must require meaningful token overlap, not just
substring containment. Without the token-Jaccard floor, generic words like
``Repository`` would be flagged as near-misses for ``serena_repository_structure``
purely because they appear as substrings — a noise flood. The cases below are taken
from a real run on the Serena codebase that produced six false positives.
"""
_write(fs_manager, "serena_repository_structure", "# repo structure notes")
_write(fs_manager, "suggested_commands", "# command notes")
_write(
fs_manager,
"docs",
# generic words ("Repository", "repository") and unrelated code identifiers
# ("_create_launch_command", "create_launch_command", "repository_root_path")
# that happen to be substrings of, or share one token with, an existing memory name
"""\
See _create_launch_command and also create_launch_command.
The Repository pattern is documented elsewhere.
Also repository_root_path is set via env var.
The repository contains many tools.
""",
)
report = fs_manager.validate_referential_integrity()
assert report.is_clean(), (
f"Expected no findings; got high={report.high_confidence_unmarked_memories}, "
f"low={report.low_confidence_unmarked_memories}, stale={report.stale_references}"
)
def test_long_flat_name_near_miss_is_high_confidence(self, fs_manager: MemoryManager) -> None:
"""
Regression: bare text that is a *near-match* of an existing long memory name
(e.g. the same name with a missing suffix) should be flagged as a high-confidence
unmarked-reference warning. Long, distinctive multi-word tokens that appear bare in
prose and similarity-match an existing memory are almost certainly intended references.
Real-world case: ``serena_repository_structure.md`` mentions
``adding_new_language_support`` in prose; the actual memory is
``adding_new_language_support_guide``. Exact-substring matching alone misses this.
"""
existing = "adding_new_language_support_guide"
bare = "adding_new_language_support" # prefix of `existing`; clearly the same concept
assert len(bare) >= HIGH_CONFIDENCE_NAME_LENGTH
_write(fs_manager, existing, "# guide content")
_write(fs_manager, "docs", f"See {bare} for details")
report = fs_manager.validate_referential_integrity()
assert len(report.high_confidence_unmarked_memories) == 1, (
f"expected one high-confidence warning for the near-miss reference, "
f"got {len(report.high_confidence_unmarked_memories)} "
f"(high={report.high_confidence_unmarked_memories}, low={report.low_confidence_unmarked_memories})"
)
warning = report.high_confidence_unmarked_memories[0]
assert warning.suspected_name == existing
assert warning.source_memory == "docs"
def test_fuzzy_accepts_bare_token_when_jaccard_just_above_floor(self, fs_manager: MemoryManager) -> None:
"""
Edge case: bare token is a clean substring of the existing memory name AND
their tokenized Jaccard sits just above ``_FUZZY_BARE_TOKEN_JACCARD_FLOOR``
(with floor = 0.6, this means 2/3 ≈ 0.667).
"""
# existing has 3 name-tokens; bare has 2 of them -> jaccard = 2/3 = 0.667
existing = "data-model-extra"
bare = "data-model"
assert len(bare) >= HIGH_CONFIDENCE_NAME_LENGTH
_write(fs_manager, existing, "# extra notes")
_write(fs_manager, "docs", f"see {bare} for naming")
report = fs_manager.validate_referential_integrity()
fuzzy = [w for w in report.high_confidence_unmarked_memories if not w.is_exact_match]
assert len(fuzzy) == 1
assert fuzzy[0].suspected_name == existing
assert fuzzy[0].actual_token == bare
def test_fuzzy_rejects_bare_token_when_jaccard_just_below_floor(self, fs_manager: MemoryManager) -> None:
"""
Edge case: bare token is still a clean substring of the existing memory name,
but their tokenized Jaccard is just below ``_FUZZY_BARE_TOKEN_JACCARD_FLOOR``
(2/4 = 0.5 with the floor at 0.6). Must be rejected — without this gate,
long candidate names would attract many borderline matches.
"""
existing = "data-model-extra-stuff" # 4 tokens
bare = "data-model" # 2 tokens; jaccard = 2/4 = 0.5
assert len(bare) >= HIGH_CONFIDENCE_NAME_LENGTH
_write(fs_manager, existing, "# notes")
_write(fs_manager, "docs", f"see {bare} for naming")
report = fs_manager.validate_referential_integrity()
assert report.high_confidence_unmarked_memories == []
assert report.low_confidence_unmarked_memories == []
def test_fuzzy_requires_substring_containment(self, fs_manager: MemoryManager) -> None:
"""
Edge case: bare token and existing name share several tokens (and even a topic
prefix) but neither is a substring of the other. Without the containment
requirement, sequence-ratio alone would mis-rank long names like
``frontend/X-subtleties`` vs ``frontend/Y-subtleties`` as near-misses.
"""
existing = "frontend/text-editor-workflow"
bare = "frontend/text-editor-features" # shares "frontend", "text", "editor"; not a substring
assert len(bare) >= HIGH_CONFIDENCE_NAME_LENGTH
_write(fs_manager, existing, "# workflow notes")
_write(fs_manager, "docs", f"see {bare} for layout")
report = fs_manager.validate_referential_integrity()
assert report.high_confidence_unmarked_memories == []
assert report.low_confidence_unmarked_memories == []
def test_fuzzy_accepts_reverse_containment(self, fs_manager: MemoryManager) -> None:
"""
Edge case: the *existing* memory name is a substring of the longer bare token
(rather than the other way around). The containment gate must hold in both
directions, so this is still flagged.
"""
existing = "playwright-gestures"
bare = "playwright-gestures-guide" # existing ⊂ bare
assert len(bare) >= HIGH_CONFIDENCE_NAME_LENGTH
_write(fs_manager, existing, "# notes")
_write(fs_manager, "docs", f"see {bare} for details")
report = fs_manager.validate_referential_integrity()
fuzzy = [w for w in report.high_confidence_unmarked_memories if not w.is_exact_match]
assert len(fuzzy) == 1
assert fuzzy[0].suspected_name == existing
assert fuzzy[0].actual_token == bare
def test_unmarked_exact_match_suppressed_for_ignored_word(self, fs_manager: MemoryManager) -> None:
"""
Names in ``_WORDS_TO_IGNORE_AS_MEMORY_NAME_CANDIDATES`` (e.g. ``core``) are
common-word collisions and must not produce bare-occurrence warnings even
when an exact memory of that name exists.
"""
_write(fs_manager, "core", "# project core notes")
_write(fs_manager, "docs", "this is the core of the system")
report = fs_manager.validate_referential_integrity()
# bare "core" matches the candidate name `core` literally but is filtered out
assert all(w.suspected_name != "core" for w in report.high_confidence_unmarked_memories)
assert all(w.suspected_name != "core" for w in report.low_confidence_unmarked_memories)
def test_unmarked_exact_match_suppressed_for_topic_with_ignored_basename(self, fs_manager: MemoryManager) -> None:
"""
The ignore filter applies to the *basename* of a candidate name, so a memory
named ``frontend/core`` is also exempt from bare-occurrence warnings (the
bare text ``frontend/core`` is more likely a directory reference than a
forgotten memory link).
"""
_write(fs_manager, "frontend/core", "# notes")
_write(fs_manager, "docs", "the frontend/core module is documented here")
report = fs_manager.validate_referential_integrity()
assert all(w.suspected_name != "frontend/core" for w in report.high_confidence_unmarked_memories)
assert all(w.suspected_name != "frontend/core" for w in report.low_confidence_unmarked_memories)
def test_unmarked_exact_match_not_suppressed_for_non_ignored_basename(self, fs_manager: MemoryManager) -> None:
"""
Sanity: the ignore filter must not over-apply to names whose basename happens
to share a prefix with an ignored word.
"""
_write(fs_manager, "frontend/core-utils", "# notes")
_write(fs_manager, "docs", "the frontend/core-utils module is documented here")
report = fs_manager.validate_referential_integrity()
assert any(w.suspected_name == "frontend/core-utils" for w in report.high_confidence_unmarked_memories)
def test_stale_reference_candidates_are_capped(self, fs_manager: MemoryManager) -> None:
"""
The candidate list proposed for an unresolved ``mem:`` reference is capped at
:attr:`MAX_STALE_REFERENCE_CANDIDATES`. Beyond a small number the list stops
aiding disambiguation and becomes noise.
"""
cap = MAX_STALE_REFERENCE_CANDIDATES
# five memories all sharing basename `login` -> all tie at compute_name_similarity == 1.0
candidate_names = ["admin/login", "api/login", "auth/login", "security/login", "users/login"]
for name in candidate_names:
_write(fs_manager, name, "# notes")
_write(fs_manager, "docs", "old reference: `mem:login`")
report = fs_manager.validate_referential_integrity()
assert len(report.stale_references) == 1
candidates = report.stale_references[0].candidates
# cap enforced; alphabetical tiebreak among equally-scored candidates
assert len(candidates) == cap
assert candidates == candidate_names[:cap]
def test_include_unmarked_false_skips_both_scans(self, fs_manager: MemoryManager) -> None:
"""
With ``include_unmarked=False`` only stale references are reported; both the
exact unmarked scan and the fuzzy near-miss scan are entirely skipped.
"""
_write(fs_manager, "auth/login", "# notes")
_write(fs_manager, "playwright-gestures", "# notes")
_write(
fs_manager,
"docs",
"bare auth/login mention and a playwright-gestures-guide near-miss; also `mem:nope`",
)
report = fs_manager.validate_referential_integrity(include_unmarked=False)
assert report.high_confidence_unmarked_memories == []
assert report.low_confidence_unmarked_memories == []
# stale reference is still reported regardless of the flag
assert len(report.stale_references) == 1
assert report.stale_references[0].referenced_name == "nope"
def test_include_fuzzy_matching_false_keeps_exact_only(self, fs_manager: MemoryManager) -> None:
"""
With ``include_unmarked=True`` but ``include_fuzzy_matching=False``, exact
bare-occurrence warnings still fire but fuzzy near-misses are suppressed.
"""
_write(fs_manager, "auth/login", "# notes")
_write(fs_manager, "playwright-gestures", "# notes")
_write(
fs_manager,
"docs",
"bare auth/login mention and a playwright-gestures-guide near-miss",
)
report = fs_manager.validate_referential_integrity(include_unmarked=True, include_fuzzy_matching=False)
# exact match for auth/login is kept
exact = [w for w in report.high_confidence_unmarked_memories if w.is_exact_match]
fuzzy = [w for w in report.high_confidence_unmarked_memories if not w.is_exact_match]
assert len(exact) == 1
assert exact[0].suspected_name == "auth/login"
# fuzzy near-miss for `playwright-gestures-guide` is suppressed
assert fuzzy == []
def test_self_reference_is_ignored(self, fs_manager: MemoryManager) -> None:
# the basename `login` appears in `auth/login`'s own content -> must not be flagged
_write(fs_manager, "auth/login", "this login flow ...")
_write(fs_manager, "auth", "# auth notes")
report = fs_manager.validate_referential_integrity()
# `auth/login` mentioning its own basename `login` should not produce a warning;
# `auth` mentioned in `auth/login` content also shouldn't (it's the prefix), and only
# actual matches matter — assert no warning concerns the source itself
for w in report.high_confidence_unmarked_memories + report.low_confidence_unmarked_memories:
assert not (w.source_memory == "auth/login" and w.suspected_name in ("login", "auth/login"))
def test_marked_reference_does_not_produce_unmarked_warning(self, fs_manager: MemoryManager) -> None:
_write(fs_manager, "auth/login", "# notes")
_write(fs_manager, "docs", "see `mem:auth/login` for details")
report = fs_manager.validate_referential_integrity()
# the marked reference is valid; no warning should fire on that occurrence
assert report.is_clean()
def test_mixed_marked_and_bare_in_same_content(self, fs_manager: MemoryManager) -> None:
_write(fs_manager, "auth/login", "# notes")
_write(fs_manager, "docs", "we have `mem:auth/login` plus a bare auth/login mention")
report = fs_manager.validate_referential_integrity()
# only the bare occurrence should be counted
assert len(report.high_confidence_unmarked_memories) == 1
assert report.high_confidence_unmarked_memories[0].occurrences == 1
def test_empty_memory_skipped(self, fs_manager: MemoryManager) -> None:
_write(fs_manager, "auth/login", "")
_write(fs_manager, "docs", "# nothing relevant")
# must not raise even though one memory is empty
report = fs_manager.validate_referential_integrity()
assert report.is_clean()
def test_read_only_status_propagated(self, tmp_path) -> None:
manager = MemoryManager(serena_data_folder=tmp_path, read_only_memory_patterns=[r"frozen/.*"])
_write(manager, "frozen/notes", "see bare auth/login mention")
_write(manager, "auth/login", "# notes")
report = manager.validate_referential_integrity()
warnings = report.high_confidence_unmarked_memories
assert len(warnings) == 1
assert warnings[0].source_memory == "frozen/notes"
assert warnings[0].source_is_read_only is True
class TestAutoPrefixBareReferences:
def test_prefixes_high_confidence_warning_in_writable_memory(self, fs_manager: MemoryManager) -> None:
_write(fs_manager, "auth/login", "# notes")
_write(fs_manager, "docs", "the auth/login process is documented here")
result = fs_manager.auto_prefix_bare_references()
assert result.total_replacements == 1
assert len(result.autofixed) == 1
# verify the file content was actually mutated
assert fs_manager.load_memory("docs") == "the mem:auth/login process is documented here"
def test_skips_low_confidence_by_default(self, fs_manager: MemoryManager) -> None:
_write(fs_manager, "auth", "# notes")
_write(fs_manager, "docs", "use auth here")
result = fs_manager.auto_prefix_bare_references()
assert result.total_replacements == 0
assert len(result.skipped_flat) == 1
# content unchanged
assert fs_manager.load_memory("docs") == "use auth here"
def test_include_flat_names_widens_scope(self, fs_manager: MemoryManager) -> None:
_write(fs_manager, "auth", "# notes")
_write(fs_manager, "docs", "use auth here")
result = fs_manager.auto_prefix_bare_references(include_flat_names=True)
assert result.total_replacements == 1
assert fs_manager.load_memory("docs") == "use mem:auth here"
def test_skips_read_only_source_by_default(self, tmp_path) -> None:
manager = MemoryManager(serena_data_folder=tmp_path, read_only_memory_patterns=[r"frozen/.*"])
_write(manager, "frozen/notes", "the auth/login process")
_write(manager, "auth/login", "# notes")
result = manager.auto_prefix_bare_references()
assert result.total_replacements == 0
assert len(result.skipped_read_only) == 1
# content unchanged
assert manager.load_memory("frozen/notes") == "the auth/login process"
def test_include_read_only_modifies_read_only_source(self, tmp_path) -> None:
manager = MemoryManager(serena_data_folder=tmp_path, read_only_memory_patterns=[r"frozen/.*"])
_write(manager, "frozen/notes", "the auth/login process")
_write(manager, "auth/login", "# notes")
result = manager.auto_prefix_bare_references(include_read_only=True)
assert result.total_replacements == 1
assert manager.load_memory("frozen/notes") == "the mem:auth/login process"
def test_multiple_occurrences_in_same_memory(self, fs_manager: MemoryManager) -> None:
_write(fs_manager, "auth/login", "# notes")
_write(fs_manager, "docs", "auth/login here and auth/login there")
result = fs_manager.auto_prefix_bare_references()
assert result.total_replacements == 2
assert fs_manager.load_memory("docs") == "mem:auth/login here and mem:auth/login there"
def test_marked_references_left_alone(self, fs_manager: MemoryManager) -> None:
_write(fs_manager, "auth/login", "# notes")
_write(fs_manager, "docs", "see `mem:auth/login` and also auth/login bare")
result = fs_manager.auto_prefix_bare_references()
# only the bare occurrence is rewritten; the already-marked one stays as-is
assert result.total_replacements == 1
assert fs_manager.load_memory("docs") == "see `mem:auth/login` and also mem:auth/login bare"
def test_dry_run_does_not_modify_files(self, fs_manager: MemoryManager) -> None:
_write(fs_manager, "auth/login", "# notes")
_write(fs_manager, "docs", "the auth/login process is documented here")
result = fs_manager.auto_prefix_bare_references(dry_run=True)
# report describes what *would* have been done
assert result.dry_run is True
assert result.total_replacements == 1
assert len(result.autofixed) == 1
# but the file on disk is unchanged
assert fs_manager.load_memory("docs") == "the auth/login process is documented here"
# running again non-dry actually rewrites
result = fs_manager.auto_prefix_bare_references()
assert result.dry_run is False
assert fs_manager.load_memory("docs") == "the mem:auth/login process is documented here"
def test_fuzzy_warnings_are_routed_to_skipped_fuzzy(self, fs_manager: MemoryManager) -> None:
# set up a fuzzy near-miss case (token shares all parts with a longer existing name)
existing = "adding_new_language_support_guide"
bare = "adding_new_language_support"
_write(fs_manager, existing, "# guide content")
_write(fs_manager, "docs", f"See {bare} for details")
result = fs_manager.auto_prefix_bare_references()
# fuzzy match must NOT be auto-rewritten — it would require substring substitution
assert result.total_replacements == 0
assert len(result.skipped_fuzzy) == 1
skipped = result.skipped_fuzzy[0]
assert skipped.actual_token == bare
assert skipped.suspected_name == existing
# source unchanged
assert fs_manager.load_memory("docs") == f"See {bare} for details"
def test_no_double_prefix_on_repeated_runs(self, fs_manager: MemoryManager) -> None:
_write(fs_manager, "auth/login", "# notes")
_write(fs_manager, "docs", "the auth/login process")
fs_manager.auto_prefix_bare_references()
second = fs_manager.auto_prefix_bare_references()
# idempotent: the second run should not touch anything
assert second.total_replacements == 0
assert fs_manager.load_memory("docs") == "the mem:auth/login process"