Project.gather_source_files walks the tree with os.walk, which hands back
directories and files separately, and then calls is_ignored_path on each
one. That method re-derived file-ness from the filesystem: an
os.path.exists and an os.path.isfile in _is_ignored_relative_path, plus
an os.path.isdir in match_path. Three syscalls per path, for an answer
the caller already had.
is_ignored_path, _is_ignored_relative_path and match_path now accept an
optional hint, and gather_source_files supplies it from the os.walk
split. The parameter defaults to None, which determines file-ness from
the filesystem exactly as before, so no existing caller changes
behaviour.
Measured on a repository with 97,549 tracked source files out of 707,889
total (Unreal Engine source plus three game projects; Windows 11,
Python 3.13):
gather_source_files() 67.8s -> 11.6s
The returned file list is byte-identical before and after (sha256 over
the sorted relative paths, 10,390,077 bytes).
For context on where the time went: a bare os.walk of the whole 708k-file
tree takes 10.7s, so this was never I/O-bound. cProfile over 15,000 real
is_ignored_path calls attributed 3.41s to nt._path_exists,
nt._path_isfile and nt._path_isdir - about 82% of the per-call cost.
The new test asserts equivalence rather than specific verdicts: for every
path in a fixture tree, across several ignore configurations, the hinted
call must agree with the unhinted one. It covers the two cases where
guessing file-ness from the name would go wrong - a directory with a
suffix, and an extensionless file - and was checked against a deliberate
inversion of the hint, which it catches.
Refs #2077
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Co-Authored-By: Dominik Jain <dominik.jain@oraios-ai.de>
Previously, Tool.apply_ex derived a session ID from the MCP session object (or
"global" with no context) and injected it into any apply() method that declared
a session_id parameter. This is being changed because:
* the new MCP SDK v2 no longer provides session identifiers
* handling it internally is more robust anyway, since clients did not
consistently use sessions
Session handling
- Remove implicit session ID injection from Tool.apply_ex, along with the
supporting _is_session_aware property and SESSION_ID_PARAM_NAME skip logic
- SerenaAgent.create_system_prompt() now creates the session itself and reports
its id, instead of receiving session_id as an argument
Tool signatures
- InitialInstructionsTool.apply() and ActivateProjectTool.apply() now declare
session_id explicitly and rely on the LLM to pass it, rather than having it
injected
- Rename SerenaReplTool.apply()'s session parameter to session_id for consistency
with the other tools
Resolves#2061
* Many case differentiations in the agent code were replaced by
method calls in the newly introduced LanguageBackend abstraction
* The registry allows new backends to be added dynamically
(via Python packages that implement a specific entrypoint)
- Require a valid Bearer token on heartbeat and project-query requests.
- Send the configured auth_secret from clients and the query-project tool.
- Test accepted and rejected credentials, including real HTTP client requests.
- Set configuration permissions to 0600 before reading and log adjustments.
- Log chmod failures and continue loading; leave Windows permissions unchanged.
- Generate a random UUID when auth_secret is missing, null, or empty.
- Persist generated secrets and preserve configured values across reloads.
- Document the shared secret and test generation, persistence, and direct construction.
- Rename the read-only context to read_project_context and add writable project_context.
- Dispatch LSP-backed edits remotely and enforce read-only access in the calling context.
- Allow project-server facade calls independently of the target project's API restrictions.
- Update dispatch coverage for both backends and access modes; remove obsolete rejection assertions.
SerenaDashboardTrayManager._update_menu() called pystray's Icon.update_menu()
on whatever thread reached it. Four call sites are off the main thread: the
Flask handlers for /register, /update_project and /unregister, and
_alive_check_loop.
pystray does no marshalling. Icon.update_menu() calls the backend directly and
pystray/_darwin.py goes straight to NSStatusItem.setMenu_(), which AppKit
requires on the main thread. On macOS versions that enforce it the tray manager
traps with SIGTRAP inside the request handler, so the agent logs "Failed to
register with tray manager: Remote end closed connection without response" and
the tray icon never becomes usable.
Menu refreshes now go through PyObjCTools.AppHelper.callAfter on Darwin, which
is asynchronous so no Flask handler blocks on the main run loop. Other
platforms call through unchanged. The helper is separate from _update_menu
because _open_dashboard and _run_viewer run on the same Flask threads and will
need it too.
No new dependency: PyObjCTools comes from pyobjc-core, already required on
macOS via pystray -> pyobjc-framework-Quartz -> pyobjc-core.
The crash itself could not be reproduced locally (macOS 26.6.2; the reporter is
on 27.0, everything else matching). Verified instead that the AppKit call moves
from a Flask worker thread to the main thread: NSThread.isMainThread() across
the three HTTP routes reads [False, False] before and [True, True, True] after.
A language server's cache directory was determined by the language_id rather than
the language server identifier's key. The two identifiers coincided in most cases.
Facade availability:
* facades can be optional (`Facade.from_api(..., is_optional=True)`), mirroring optional tools
* `Facade.is_enabled()` is derived: a facade is available iff it has at least one enabled method
* `ApiScope.is_facade_enabled` is thereby obsolete and removed
Opt-in rule (uniform for optional facades, excluded facades and optional methods):
* methods of a facade which is not included and optional methods require explicit inclusion
* all other methods are enabled unless explicitly excluded
Application:
* the `ext` facade is optional; the query-projects mode includes it (`included_apis: [ext]`)
Godot's GDScript parser can report a symbol's end column one column past the
line-end convention every other language server follows (closing a node's range from
the next lookahead token instead of the last consumed one, when that lookahead is a
synthesized newline); `replace_symbol_body` on the last function in a file silently
consumed the separating blank line as a result. `GodotLanguageServer` now corrects this
specific, measured overshoot when building its high-level document symbols (#1974)
---------
Signed-off-by: Amir Fathi <amirfathi.me@gmail.com>
The project registry in serena_config.yml can be added to from the CLI
('project create', 'project index') and from the MCP side
('activate_project'), but it could only be removed from on the MCP side,
via the 'remove_project' tool. From a terminal, hand-editing the file was
the only way to drop an entry.
'serena project remove <PROJECT_NAME|PROJECT_PATH>' takes the same
argument type as 'project index', so a project can be addressed by name
or by path. Only the registry entry is removed; the project's own files,
including its project configuration, are left on disk.
Adds SerenaConfig.remove_registered_project, which mirrors
add_registered_project and removes the given entry rather than the first
entry whose name matches. Resolving a path to an entry and then deleting
by name would remove the wrong project when two registered directories
carry the same project_name.
Refs #2029
`serena.analytics` imported `anthropic.types` at module level for two type
annotations, so every `import serena.cli` / `import serena.agent` paid for
loading the whole anthropic package (2-4 s on the reporter's Windows machine)
even with the default CHAR_COUNT token estimator. That delay can push MCP
startup past a client's initial tool-discovery window.
The import now lives behind `TYPE_CHECKING`; `AnthropicTokenCount` already
imported `anthropic` lazily in `__init__`, and the request body is a plain
dict (what `MessageParam` produces at runtime anyway).
Tests: importing serena.cli / serena.agent / serena.analytics in a fresh
interpreter leaves `anthropic` out of `sys.modules`; the Anthropic estimator
still sends the expected count_tokens request.
Fixes#2012
* fix(tools): save edited source files atomically
Issue #1958 lists three direct open(path, "w") writes that can destroy a
file if the process stops partway through: two in MemoryManager and
CodeEditor._save_edited_file. The memory ones were fixed in #1969, which
is also where write_file_atomic comes from; this is the source-file half
the issue proposes as a follow-up.
A plain open(path, "w") truncates the file before the new content is
complete, so a crash, an OOM kill or a full disk during the write leaves
the file holding neither the old content nor the new one.
The issue asks for symlink and metadata behaviour to be confirmed before
source files use the helper. Measured: the executable bit already
survived, but a bare os.replace onto a symlink path swapped the link out
for a regular file and left its target holding stale content, which
open(path, "w") does not do. write_file_atomic now resolves the path
first, so it follows symlinks the way the call it replaces did, and the
temporary file lands in the destination's real directory.
Refs #1958
* test(tools): make the atomic-write tests platform-independent
Two of the new tests failed on the Windows CI job. Both were the tests,
not the change under test.
- The executable-bit test asserted 0o755 and read back 0o666: Windows
does not model POSIX permission bits, and chmod there only toggles the
read-only flag. Skipped on win32.
- The encoding test left newline unset, which is LineEnding.NATIVE and
therefore newline=None, under which Python translates "\n" to
os.linesep on write; it got b"caf\xe9\r\n". The line ending is now
pinned so the test covers the encoding alone, matching the sibling
test that already pins "\r\n" and passes on Windows.
Also covers three paths the first round missed, all of which pass both
with and without the change and are there to show it does not break
them: a file in a subdirectory, a symlinked parent directory (the path
is resolved in full, so the temp file lands in the destination's real
directory), and a non-ASCII filename. The last two skip where the
platform cannot create symlinks or represent the name.
Refs #1958
trusted_project_path_patterns is matched against the project's root path itself
(SerenaConfig.is_trusted_project_path, called from Project.is_trusted with project_root).
A '<project root>/**' entry therefore matches only paths below the root and trusts no
project at all, while reading like the natural way to name one project. The template's
example showed only parent-directory globs, so there was nothing to copy for the
single-project case.
Adds the bare-root form to the example, states which path a pattern is matched against,
and pins all three cases in tests so the documented example cannot silently go stale.
Requested by a maintainer in #2001.
Configuration endpoint:
* ResponseConfigOverview: agent_interface, language_backend and facades (None in tool mode)
* facades report each facade and its methods with their enabled state (SerenaReplEntrypoint.get_facade_availability_info)
Dashboard UI:
* new rows "Interface" and "Backend"; "Languages" is shown for the LSP backend only
* new collapsible section "Active Functions" listing the facades and their methods, dimming disabled ones
* _activate_project: a project's backend override switches the backend in REPL mode (background
modes, facades, prompt parameters and backend initialisation are recomputed upon activation)
* the tool interface keeps rejecting the mismatch (fixed toolset); the error now mentions the REPL
as a workaround
* single-project sessions: in the REPL tool's description (fixed at registration)
* otherwise: in the project activation message (facades depend on the activated project)
* REPL tool description wording adjusted (listing location, notebook semantics)
Peripheral changes:
* is_tool_function_available: in REPL mode, active tools without facade counterparts count as available
* subject line must cover the entire change (abstract if necessary)
* details as concise bullet items, grouped by topic where the change spans several topics
Availability check:
* SerenaAgent.is_tool_function_available(tool_class): whether a tool's functionality is available
to the LLM (tool mode: tool active; REPL mode: an enabled facade method corresponds to the tool)
* replaces the name-based tool_is_active/tool_is_exposed (activation message, dashboard, onboarding)
Prompts:
* activation message names functions via PromptParams.get_function_name (interface-specific names)
Introduce the ext facade with list_projects and project_context: within
`with s.ext.project_context(name):`, the facades operate read-only on the given external project.
The agent's active project is switched temporarily, and methods marked with
`@facade_method(uses_project_server=True)` (all lsp methods) are executed in the project server when
the LSP backend is active (with JetBrains, the IDE serves all projects, so they run locally). The
project server gains a /call_facade_method route executing a facade method on its own REPL facades
and returning the pickled result; arguments are transferred as JSON. Editing methods are refused,
and contexts cannot be nested. This replaces the query_project tools in the REPL.
Result objects are now self-contained (picklable): renderers no longer hold the agent (only the
default length limit; the constructor interface is unchanged), LSP results carry eagerly retrieved
symbol info and reference contexts instead of renderers accessing the language server or project.
Code is now executed at module level in the session's namespace (statements executed, a trailing
expression evaluated as the result), replacing the function wrapper with global declarations that
persistence had required. This yields notebook semantics throughout: top-level bindings persist by
construction, functions resolve names from the session namespace, and `return` is no longer used
(a top-level `return` yields a syntax error with a hint). Tool description and tests updated.
Code executions of a session now share a namespace (SerenaSession.repl_namespace), which serves as
the globals of the executions: variables, functions, classes and imports bound at the top level of
the submitted code persist across calls, like the cells of a notebook. The code is wrapped in a
function at the AST level (preserving line numbers for error reporting), with the top-level names
declared global. The entrypoint `s` is re-bound before every execution, such that persisted
functions always use the current entrypoint (e.g. after a project switch). `s.vars()` lists the
persisted items, `s.clear()` removes them; the tool description explains the persistence.
The namespace is tied to the session's lifetime; the session registry additionally evicts sessions
which have been idle for too long, releasing their namespaces.
Only the three declarations carrying member curation remain (LanguageServerSymbol, ReplacementOccurrence, MatchedConsecutiveLines); the discovered type sets are unchanged.
Facades now discover all user-defined classes reachable through the annotations of their methods and
of the members of reachable types (transitively) and make them documentable via s.info, without the
types having to be declared. Explicit ReferencedType declarations serve curation only (member
whitelists, flags). TypedDicts are documented with their keys (e.g. the JetBrains symbol DTOs and
the LSP diagnostic structures), enums with their members. SymbolKind no longer needs declaring.
Referenced types are now also discovered through the parameter annotations of type members and
facade methods (not only through return types), such that e.g. SymbolKind (a parameter type of
LanguageServerSymbol.iter_ancestors) is documented along with LanguageServerSymbol. Enums are
documented with their members and values. The lsp facade declares SymbolKind as a referenced type.
Type documentation now transitively includes the declared types a type's members reference, such
that e.g. LspSymbolCollection's documentation includes LanguageServerSymbol. To avoid repeating widely
shared types, the REPL tracks per session which types have been documented: a contained type
documented earlier in the session is only pointed to (an explicit request always yields it in full).
Since MCP provides no reliable session identification and clients keep a stdio server across
conversations, the REPL's session identity is supplied by the LLM: the instructions (system prompt)
establish a SerenaSession and state its id, and the REPL tool takes a required session_id parameter.
Sessions are managed by a SessionRegistry (creation on demand, LRU eviction). Tools which may be
called before the instructions have been read (activate_project, initial_instructions) are unchanged.
The REPL overview (tier 0, part of the tool description) now names the result type of methods
returning objects that can be processed in code (e.g. `find_symbol -> LspSymbolCollection`), such
that type documentation can be requested alongside a facade in one turn.
The facade description (tier 1) documents the common operations in full, summarises methods marked
as niche (new flag of facade_method; not yet set anywhere) with a pointer to their documentation,
and lists result types by name only. Type documentation (tier 2) is provided on request only;
provide_info_with_facade remains available but is no longer set for any type.
The REPL tool description states this protocol, in particular that facade descriptions do not
include result types and that type documentation should only be requested for processing results in
code.
Facades declare the types their methods return or contain as ReferencedTypes (with an optional
whitelist of members for curation, e.g. for LanguageServerSymbol, and a flag for inclusion in the
facade description). Type documentation is generated from class docstrings, attribute annotations,
properties and method signatures/docstrings, and is available via s.info("<facade>.<Type>") or by
bare type name; method documentation points to the documentation of a referenced return type.
Annotations are rendered without module paths, such that the names in signatures are the names by
which types can be looked up. s.info accepts several items at once, reporting unknown items inline.
All APIs declare their result types; result classes declare their attributes at class level.
Introduce ConfigApi, exposing Serena's configuration and session state: get_current_config and
open_dashboard (both non-editing). The corresponding tools delegate to the API via ConfigApiMixin.
As on the tool side, opening the dashboard is only offered if the dashboard is enabled and not
opened automatically; the condition is shared (_is_dashboard_openable) and applied to the API scope
via a NamedApiInclusionDefinition, the API-side counterpart of NamedToolInclusionDefinition.
