diff --git a/DOCKER.md b/DOCKER.md index 08348182..bb77040c 100644 --- a/DOCKER.md +++ b/DOCKER.md @@ -27,6 +27,8 @@ special settings for Docker usage: ``` # Disable the GUI log window since it's not supported in Docker gui_log_window: False +# Listen on all interfaces for the web dashboard to be accessible from outside the container +web_dashboard_listen_address: 0.0.0.0 # Disable opening the web dashboard on launch (not possible within the container) web_dashboard_open_on_launch: False ``` diff --git a/Dockerfile b/Dockerfile index e58d821b..1dccaf40 100644 --- a/Dockerfile +++ b/Dockerfile @@ -64,6 +64,7 @@ ENV SERENA_HOME=/workspaces/serena/config RUN mkdir -p $SERENA_HOME RUN cp src/serena/resources/serena_config.template.yml $SERENA_HOME/serena_config.yml RUN sed -i 's/^gui_log_window: .*/gui_log_window: False/' $SERENA_HOME/serena_config.yml +RUN sed -i 's/^web_dashboard_listen_address: .*/web_dashboard_listen_address: 0.0.0.0/' $SERENA_HOME/serena_config.yml RUN sed -i 's/^web_dashboard_open_on_launch: .*/web_dashboard_open_on_launch: False/' $SERENA_HOME/serena_config.yml # Create virtual environment and install dependencies diff --git a/docs/02-usage/030_clients.md b/docs/02-usage/030_clients.md index b757934b..e1e0b6aa 100644 --- a/docs/02-usage/030_clients.md +++ b/docs/02-usage/030_clients.md @@ -75,7 +75,8 @@ object, e.g. Serena is a great way to make Claude Code both cheaper and more powerful! -From your project directory, add serena with a command like this, +**Per-Project Configuration.** To add the Serena MCP server to the current project in the current directory, +use this command: ```shell claude mcp add serena -- uvx --from git+https://github.com/oraios/serena serena start-mcp-server --context claude-code --project "$(pwd)" @@ -88,18 +89,31 @@ Note: that Serena is configured to work on the current project from the get-go, following Claude Code's mode of operation. -Alternatively, use `--project-from-cwd` for user-level configuration that works across all projects: +**Global Configuration**. Alternatively, use `--project-from-cwd` for user-level configuration that works across all projects: ```shell claude mcp add --scope user serena -- start-mcp-server --context=claude-code --project-from-cwd ``` -This searches up from the current directory for `.serena/project.yml` or `.git` markers, -falling back to the current directory if neither is found. This makes it suitable for a -single global MCP configuration. The `--project-from-cwd` option is intended for CLI-based agents -(like Claude Code, Gemina or Codex) that invoke Serena from within project directories. +Whenever you start Claude Code, Serena will search up from the current directory for `.serena/project.yml` or `.git` markers, +activating the current directory as the project if neither is found. +This mechanism makes it suitable for a single global MCP configuration. -Be sure to use at least `v1.0.52` of Claude Code (as earlier versions do not read MCP server system prompts upon startup). +**Maximum Token Efficiency.** To maximize token efficiency, you may want to use Claude Code's +*on-demand tool loading* feature, which is supported since at least v2.0.74 of Claude Code. +This feature avoids sending all tool descriptions to Claude upon startup, thus saving tokens. +Instead, Claude will search for tools as needed (but there are no guarantees that it will +search optimally, of course). +To enable this feature, set the environment variable `ENABLE_TOOL_SEARCH=true`. +Depending on your shell, you can also set this on a per-session basis, e.g. using +```shell +ENABLE_TOOL_SEARCH=true claude +``` +in bash/zsh, or using +```shell +set ENABLE_TOOL_SEARCH=true && claude +``` +in Windows CMD to launch Claude Code. ## VSCode diff --git a/docs/02-usage/040_workflow.md b/docs/02-usage/040_workflow.md index 35afb55e..4544b25b 100644 --- a/docs/02-usage/040_workflow.md +++ b/docs/02-usage/040_workflow.md @@ -55,18 +55,18 @@ Indexing has to be called only once. During regular usage, Serena will automatic Project activation makes Serena aware of the project you want to work with. You can either choose to do this - * while in a conversation, by telling the model to activate a project, e.g., + * while in a conversation, by telling the LLM to activate a project, e.g., * "Activate the project /path/to/my_project" (for first-time activation with auto-creation) * "Activate the project my_project" Note that this option requires the `activate_project` tool to be active, - which it isn't in the (default version) of context `ide-assistant` if a project is provided at startup. - (The tool is deactivated because we assume that in the ide-assistant context the user will only work on the open project and have + which it isn't in single-project [contexts](contexts) like `ide` or `claude-code` *if* a project is provided at startup. + (The tool is deactivated, because we assume that in these contexts, user will only work on the single, open project and have no need to switch it.) * when the MCP server starts, by passing the project path or name as a command-line argument - (e.g. when working on a fixed project in `ide-assistant` mode): `--project ` + (e.g. when using a single-project mode like `ide` or `claude-code`): `--project ` ## Onboarding & Memories diff --git a/docs/02-usage/050_configuration.md b/docs/02-usage/050_configuration.md index ad384aaa..1a62f6b0 100644 --- a/docs/02-usage/050_configuration.md +++ b/docs/02-usage/050_configuration.md @@ -8,14 +8,15 @@ Serena is configured in four places: 1. The `serena_config.yml` for general settings that apply to all clients and projects. It is located in your user directory under `.serena/serena_config.yml`. - If you do not explicitly create the file, it will be auto-generated when you first run Serena. + The file will be auto-generated when you first run Serena. You can edit it directly or use ```shell - uvx --from git+https://github.com/oraios/serena serena config edit + config edit ``` - (or use the `--directory` command version). + where `` is [your way of running Serena](020_running). + The configuration file can also be accessed through [Serena's dashboard](060_dashboard). 2. In the arguments passed to the `start-mcp-server` in your client's config (see below), which will apply to all sessions started by the respective client. In particular, the [context](contexts) parameter should be set appropriately for Serena to be best adjusted to existing tools and capabilities of your client. @@ -55,6 +56,12 @@ Choose the context that best matches the type of integration you are using. Find the concrete definitions of the above contexts [here](https://github.com/oraios/serena/tree/main/src/serena/resources/config/contexts). +Note that the contexts `ide` and `claude-code` are **single-project contexts** (defining `single_project: true`). +For such contexts, if a project is provided at startup, the set of tools is limited to those required by the project's +concrete configuration, and other tools are excluded completely, allowing the set of tools to be minimal. +Tools explicitly disabled by the project will not be available at all. Since changing the active project +ceases to be a relevant operation in this case, the project activation tool is disabled. + When launching Serena, specify the context using `--context `. Note that for cases where parameter lists are specified (e.g. Claude Desktop), you must add two parameters to the list. diff --git a/pyproject.toml b/pyproject.toml index 7c44e907..086dd9c2 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -25,7 +25,7 @@ dependencies = [ "pydantic>=2.10.6", "types-pyyaml>=6.0.12.20241230", "pyyaml>=6.0.2", - "ruamel.yaml>=0.18.0", + "ruamel.yaml==0.18.14", "jinja2>=3.1.6", "dotenv>=0.9.9", "pathspec>=0.12.1", diff --git a/src/serena/agent.py b/src/serena/agent.py index a43ec287..547c8a5e 100644 --- a/src/serena/agent.py +++ b/src/serena/agent.py @@ -107,26 +107,35 @@ class ToolSet: registry = ToolRegistry() tool_names = set(self._tool_names) for definition in tool_inclusion_definitions: - included_tools = [] - excluded_tools = [] - for included_tool in definition.included_optional_tools: - included_tool = get_updated_tool_name(included_tool) - if not registry.is_valid_tool_name(included_tool): - raise ValueError(f"Invalid tool name '{included_tool}' provided for inclusion") - if included_tool not in tool_names: - tool_names.add(included_tool) - included_tools.append(included_tool) - for excluded_tool in definition.excluded_tools: - excluded_tool = get_updated_tool_name(excluded_tool) - if not registry.is_valid_tool_name(excluded_tool): - raise ValueError(f"Invalid tool name '{excluded_tool}' provided for exclusion") - if excluded_tool in tool_names: - tool_names.remove(excluded_tool) - excluded_tools.append(excluded_tool) - if included_tools: - log.info(f"{definition} included {len(included_tools)} tools: {', '.join(included_tools)}") - if excluded_tools: - log.info(f"{definition} excluded {len(excluded_tools)} tools: {', '.join(excluded_tools)}") + if definition.is_fixed_tool_set(): + tool_names = set() + for fixed_tool in definition.fixed_tools: + fixed_tool = get_updated_tool_name(fixed_tool) + if not registry.is_valid_tool_name(fixed_tool): + raise ValueError(f"Invalid tool name '{fixed_tool}' provided for fixed tool set") + tool_names.add(fixed_tool) + log.info(f"{definition} defined a fixed tool set with {len(tool_names)} tools: {', '.join(tool_names)}") + else: + included_tools = [] + excluded_tools = [] + for included_tool in definition.included_optional_tools: + included_tool = get_updated_tool_name(included_tool) + if not registry.is_valid_tool_name(included_tool): + raise ValueError(f"Invalid tool name '{included_tool}' provided for inclusion") + if included_tool not in tool_names: + tool_names.add(included_tool) + included_tools.append(included_tool) + for excluded_tool in definition.excluded_tools: + excluded_tool = get_updated_tool_name(excluded_tool) + if not registry.is_valid_tool_name(excluded_tool): + raise ValueError(f"Invalid tool name '{excluded_tool}' provided for exclusion") + if excluded_tool in tool_names: + tool_names.remove(excluded_tool) + excluded_tools.append(excluded_tool) + if included_tools: + log.info(f"{definition} included {len(included_tools)} tools: {', '.join(included_tools)}") + if excluded_tools: + log.info(f"{definition} excluded {len(excluded_tools)} tools: {', '.join(excluded_tools)}") return ToolSet(tool_names) def without_editing_tools(self) -> "ToolSet": diff --git a/src/serena/cli.py b/src/serena/cli.py index bc4af3e6..217cb409 100644 --- a/src/serena/cli.py +++ b/src/serena/cli.py @@ -27,7 +27,7 @@ from serena.constants import ( SERENAS_OWN_CONTEXT_YAMLS_DIR, SERENAS_OWN_MODE_YAMLS_DIR, ) -from serena.mcp import SerenaMCPFactory, SerenaMCPFactorySingleProcess +from serena.mcp import SerenaMCPFactory from serena.project import Project from serena.tools import FindReferencingSymbolsTool, FindSymbolTool, GetSymbolsOverviewTool, SearchForPatternTool, ToolRegistry from serena.util.logging import MemoryLogHandler @@ -237,7 +237,7 @@ class TopLevelCommands(AutoRegisteringGroup): log.info("Auto-detected project root: %s", project) project_file = project_file_arg or project - factory = SerenaMCPFactorySingleProcess(context=context, project=project_file, memory_log_handler=memory_log_handler) + factory = SerenaMCPFactory(context=context, project=project_file, memory_log_handler=memory_log_handler) server = factory.create_mcp_server( host=host, port=port, diff --git a/src/serena/config/serena_config.py b/src/serena/config/serena_config.py index aa8ec355..7933c214 100644 --- a/src/serena/config/serena_config.py +++ b/src/serena/config/serena_config.py @@ -5,7 +5,7 @@ The Serena Model Context Protocol (MCP) Server import dataclasses import os import shutil -from collections.abc import Iterable +from collections.abc import Sequence from copy import deepcopy from dataclasses import dataclass, field from datetime import datetime @@ -95,8 +95,32 @@ class SerenaPaths: @dataclass class ToolInclusionDefinition: - excluded_tools: Iterable[str] = () - included_optional_tools: Iterable[str] = () + """ + Defines which tools to include/exclude in Serena's operation. + This can mean either + * defining exclusions/inclusions to apply to an existing set of tools [incremental mode], or + * defining a fixed set of tools to use [fixed mode]. + """ + + excluded_tools: Sequence[str] = () + """ + the names of tools to exclude from use [incremental mode] + """ + included_optional_tools: Sequence[str] = () + """ + the names of optional tools to include [incremental mode] + """ + fixed_tools: Sequence[str] = () + """ + the names of tools to use as a fixed set of tools [fixed mode] + """ + + def is_fixed_tool_set(self) -> bool: + num_fixed = len(self.fixed_tools) + num_incremental = len(self.excluded_tools) + len(self.included_optional_tools) + if num_fixed > 0 and num_incremental > 0: + raise ValueError("Cannot use both fixed_tools and excluded_tools/included_optional_tools at the same time.") + return num_fixed > 0 class SerenaConfigError(Exception): diff --git a/src/serena/mcp.py b/src/serena/mcp.py index f4e6646f..2335ffd1 100644 --- a/src/serena/mcp.py +++ b/src/serena/mcp.py @@ -3,7 +3,6 @@ The Serena Model Context Protocol (MCP) Server """ import sys -from abc import abstractmethod from collections.abc import AsyncIterator, Iterator, Sequence from contextlib import asynccontextmanager from copy import deepcopy @@ -49,15 +48,22 @@ class SerenaMCPRequestContext: class SerenaMCPFactory: - def __init__(self, context: str = DEFAULT_CONTEXT, project: str | None = None): + """ + Factory for the creation of the Serena MCP server with an associated SerenaAgent. + """ + + def __init__(self, context: str = DEFAULT_CONTEXT, project: str | None = None, memory_log_handler: MemoryLogHandler | None = None): """ :param context: The context name or path to context file :param project: Either an absolute path to the project directory or a name of an already registered project. If the project passed here hasn't been registered yet, it will be registered automatically and can be activated by its name afterward. + :param memory_log_handler: the in-memory log handler to use for the agent's logging """ self.context = SerenaAgentContext.load(context) self.project = project + self.agent: SerenaAgent | None = None + self.memory_log_handler = memory_log_handler @staticmethod def _sanitize_for_openai_tools(schema: dict) -> dict: @@ -229,9 +235,9 @@ class SerenaMCPFactory: title=None, ) - @abstractmethod def _iter_tools(self) -> Iterator[Tool]: - pass + assert self.agent is not None + yield from self.agent.get_exposed_tool_instances() # noinspection PyProtectedMember def _set_mcp_tools(self, mcp: FastMCP, openai_tool_compatible: bool = False) -> None: @@ -243,9 +249,13 @@ class SerenaMCPFactory: mcp._tool_manager._tools[tool.get_name()] = mcp_tool log.info(f"Starting MCP server with {len(mcp._tool_manager._tools)} tools: {list(mcp._tool_manager._tools.keys())}") - @abstractmethod - def _instantiate_agent(self, serena_config: SerenaConfig, modes: list[SerenaAgentMode]) -> None: - pass + def _create_serena_agent(self, serena_config: SerenaConfig, modes: list[SerenaAgentMode]) -> SerenaAgent: + return SerenaAgent( + project=self.project, serena_config=serena_config, context=self.context, modes=modes, memory_log_handler=self.memory_log_handler + ) + + def _create_default_serena_config(self) -> SerenaConfig: + return SerenaConfig.from_config_file() def create_mcp_server( self, @@ -275,7 +285,7 @@ class SerenaMCPFactory: :param tool_timeout: Timeout in seconds for tool execution. If not specified, will take the value from the serena configuration. """ try: - config = SerenaConfig.from_config_file() + config = self._create_default_serena_config() # update configuration with the provided parameters if enable_web_dashboard is not None: @@ -293,7 +303,7 @@ class SerenaMCPFactory: config.language_backend = language_backend modes_instances = [SerenaAgentMode.load(mode) for mode in modes] - self._instantiate_agent(config, modes_instances) + self.agent = self._create_serena_agent(config, modes_instances) except Exception as e: show_fatal_exception_safe(e) @@ -308,49 +318,13 @@ class SerenaMCPFactory: return mcp @asynccontextmanager - @abstractmethod async def server_lifespan(self, mcp_server: FastMCP) -> AsyncIterator[None]: """Manage server startup and shutdown lifecycle.""" - yield None # ensures MyPy understands we yield None - - @abstractmethod - def _get_initial_instructions(self) -> str: - pass - - -class SerenaMCPFactorySingleProcess(SerenaMCPFactory): - """ - MCP server factory where the SerenaAgent and its language server run in the same process as the MCP server - """ - - def __init__(self, context: str = DEFAULT_CONTEXT, project: str | None = None, memory_log_handler: MemoryLogHandler | None = None): - """ - :param context: The context name or path to context file - :param project: Either an absolute path to the project directory or a name of an already registered project. - If the project passed here hasn't been registered yet, it will be registered automatically and can be activated by its name - afterward. - """ - super().__init__(context=context, project=project) - self.agent: SerenaAgent | None = None - self.memory_log_handler = memory_log_handler - - def _instantiate_agent(self, serena_config: SerenaConfig, modes: list[SerenaAgentMode]) -> None: - self.agent = SerenaAgent( - project=self.project, serena_config=serena_config, context=self.context, modes=modes, memory_log_handler=self.memory_log_handler - ) - - def _iter_tools(self) -> Iterator[Tool]: - assert self.agent is not None - yield from self.agent.get_exposed_tool_instances() - - def _get_initial_instructions(self) -> str: - assert self.agent is not None - # we don't use the tool (which at the time of writing calls this method), since the tool may be disabled by the config - return self.agent.create_system_prompt() - - @asynccontextmanager - async def server_lifespan(self, mcp_server: FastMCP) -> AsyncIterator[None]: openai_tool_compatible = self.context.name in ["chatgpt", "codex", "oaicompat-agent"] self._set_mcp_tools(mcp_server, openai_tool_compatible=openai_tool_compatible) log.info("MCP server lifetime setup complete") yield + + def _get_initial_instructions(self) -> str: + assert self.agent is not None + return self.agent.create_system_prompt() diff --git a/src/serena/tools/jetbrains_plugin_client.py b/src/serena/tools/jetbrains_plugin_client.py index 37d10a4d..0ed9333b 100644 --- a/src/serena/tools/jetbrains_plugin_client.py +++ b/src/serena/tools/jetbrains_plugin_client.py @@ -41,9 +41,13 @@ class JetBrainsPluginClient(ToStringMixin): """ BASE_PORT = 0x5EA2 + PLUGIN_REQUEST_TIMEOUT = 300 + """ + the timeout used for request handling within the plugin (a constant in the plugin) + """ last_port: int | None = None - def __init__(self, port: int, timeout: int = 30): + def __init__(self, port: int, timeout: int = PLUGIN_REQUEST_TIMEOUT): self.base_url = f"http://127.0.0.1:{port}" self.timeout = timeout self.session = requests.Session() diff --git a/src/serena/tools/tools_base.py b/src/serena/tools/tools_base.py index 879f8184..16f3f517 100644 --- a/src/serena/tools/tools_base.py +++ b/src/serena/tools/tools_base.py @@ -349,12 +349,15 @@ class RegisteredTool: tool_name: str +tool_packages = ["serena.tools"] + + @singleton class ToolRegistry: def __init__(self) -> None: self._tool_dict: dict[str, RegisteredTool] = {} for cls in iter_subclasses(Tool): - if not cls.__module__.startswith("serena.tools"): + if not any(cls.__module__.startswith(pkg) for pkg in tool_packages): continue is_optional = issubclass(cls, ToolMarkerOptional) name = cls.get_name_from_cls() diff --git a/test/serena/test_tool_parameter_types.py b/test/serena/test_tool_parameter_types.py index ec714be4..4fdf0dc8 100644 --- a/test/serena/test_tool_parameter_types.py +++ b/test/serena/test_tool_parameter_types.py @@ -3,7 +3,7 @@ import logging import pytest from serena.config.serena_config import SerenaConfig -from serena.mcp import SerenaMCPFactorySingleProcess +from serena.mcp import SerenaMCPFactory from serena.tools.tools_base import ToolRegistry @@ -16,9 +16,9 @@ def test_all_tool_parameters_have_type(context): cfg = SerenaConfig(gui_log_window_enabled=False, web_dashboard=False, log_level=logging.ERROR) registry = ToolRegistry() cfg.included_optional_tools = tuple(registry.get_tool_names_optional()) - factory = SerenaMCPFactorySingleProcess(context=context) + factory = SerenaMCPFactory(context=context) # Initialize the agent so that the tools are available - factory._instantiate_agent(cfg, []) + factory.agent = factory._create_serena_agent(cfg, []) tools = list(factory._iter_tools()) for tool in tools: diff --git a/uv.lock b/uv.lock index 19393a69..e8083592 100644 --- a/uv.lock +++ b/uv.lock @@ -2014,7 +2014,7 @@ requires-dist = [ { name = "python-dotenv", specifier = ">=1.0.0,<2" }, { name = "pyyaml", specifier = ">=6.0.2" }, { name = "requests", specifier = ">=2.32.3,<3" }, - { name = "ruamel-yaml", specifier = ">=0.18.0" }, + { name = "ruamel-yaml", specifier = "==0.18.14" }, { name = "ruff", marker = "extra == 'dev'", specifier = ">=0.0.285" }, { name = "sensai-utils", specifier = ">=1.5.0" }, { name = "sphinx", marker = "extra == 'dev'", specifier = ">=7,<8" },