tiennm99/serena
1
0 Fork 0
mirror of https://github.com/tiennm99/serena.git synced 2026-10-13 14:16:21 +00:00
Code Issues Packages Projects Releases Wiki Activity

Merge branch 'main' into fix/ts-coverage-dirname-ignore

Browse Source
This commit is contained in:
Dr. Dominik Jain authored and GitHub committed 2026-07-21 13:29:54 +02:00
2 parents c34be023d0 9b710a7193
commit 0c96c39a90
298 files changed
+13756 -3291
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

.github/ISSUE_TEMPLATE/issue--bug--performance-problem--question-.md
+9 -16
View File
@@ -7,22 +7,15 @@ assignees: ''
---
I have:
Preconditions:
- [ ] read the relevant parts of the documentation and verified that the issue cannot be solved by adjusting configuration
- [ ] understood that the Serena Dashboard can be disabled through the config
- [ ] understood that, by default, a client session will start a separate instance of a Serena server.
- [ ] understood that, for multi-agent setups, the Streamable HTTP/SSE mode should be used.
- [ ] understood that non-project files are ignored using either .gitignore or the corresponding setting in `.serena/project.yml`
- [ ] looked for similar issues and discussions, including closed ones
- [ ] made sure it's an actual issue, not a question (use GitHub Discussions instead).
- [ ] I have made sure it's an actual issue, not a question (use [GitHub Discussions](https://github.com/oraios/serena/discussions) instead).
- [ ] I have consulted the [user guide](https://oraios.github.io/serena/02-usage/) and verified that the issue cannot be resolved by adjusting configuration/following recommended workflows.
- [ ] I have looked for similar issues and discussions, including closed ones.
If you have encountered an actual issue:
Issue details:
- If using language servers (not the JetBrains plugin),
- [ ] I performed `<uv invocation> serena project health-check`
- [ ] I indexed the project as described in the documentation
- [ ] I added sufficient explanation of my setup: the MCP client, the OS, the programming language(s), any config adjustments or relevant project specifics
- [ ] I explained how the issue arose and, where possible, added instructions on how to reproduce it
- [ ] If the issue happens on an open-source project, I have added the link
- [ ] I provided a meaningful title and description
- [ ] I have provided a meaningful title and description.
- [ ] I have explained how the issue arose and, where possible, added instructions on how to reproduce it.
- [ ] I have added details on my setup: Serena version, MCP client, OS, the programming language(s), relevant configuration adjustments and project specifics.
- [ ] If the issue relates to an application of Serena to an open-source project, I have added the link.
.github/pull_request_template.md
+4
View File
@@ -0,0 +1,4 @@
## Checklist
- [ ] This PR follows the guidelines in `CONTRIBUTING.md` regarding the scope of PRs.
- [ ] For changes that add features or fix problems, I have added an entry to `CHANGELOG.md`, which concisely describes the change.
.github/workflows/pytest.yml
+377 -236
View File
@@ -1,3 +1,42 @@
# =============================================================================
# Test CI -- strategy & rationale (details inline at each step)
# =============================================================================
# BATCHING / PARALLELISM
# Tests run as parallel matrix jobs over (os x batch). They are split into 5
# marker-based batches (jvm, native, other-langs, niche, catch-all). We batch by
# pytest MARKER, not pytest-xdist (`-n`): tests share fixtures, language-server
# processes and on-disk resources, so in-process parallelism would race them.
# Each batch installs ONLY the toolchains its languages need (every install step
# is gated on matrix.batch), keeping jobs lean. catch-all is the negation of all
# named marker groups, so every test lands in exactly one batch and the partition
# cannot desync when a language is added.
#
# OS GATING
# Default is the full (os x batch) cross-product. `niche` (heavy/slow toolchains:
# lean, ocaml, nix, julia, R, perl) is ubuntu-only via matrix.exclude -- not worth
# the Windows/macOS cost. An OS-specific *language* inside an otherwise all-OS
# batch (swift -> macOS, haskell -> Linux only) is skipped one level down by the
# central conftest guard, NOT by the matrix.
#
# CONCURRENCY LIMITS (GitHub free tier)
# Hard caps: 20 concurrent jobs total and 5 concurrent macOS jobs. These bound the
# batch count -- the current matrix is 13 jobs (4 of them macOS), inside both
# limits; adding batches/OSes must keep respecting these ceilings or jobs queue.
# `concurrency: cancel-in-progress` also kills superseded runs on the same ref to
# free capacity quickly.
#
# CACHING
# Per-batch language-server cache keyed by os+batch (language-servers-<os>-<batch>),
# plus dedicated caches for the uv venv (keyed on uv.lock), swiftly, ZLS, FPC and
# the Go build/bin. Most niche toolchains restore from cache, so that batch is fast
# despite its size.
#
# TESTING / SKIPS
# All language skip logic is centralized in test/conftest.py
# (`language_tests_enabled()`): a test is skipped when its toolchain is not expected
# on the current OS/CI. Lint and type-check are batch-independent and run once per
# OS (catch-all only), not in every batch.
# =============================================================================
name: Tests
on:
@@ -15,32 +54,55 @@ concurrency:
jobs:
cpu:
name: Tests on ${{ matrix.os }}
name: ${{ matrix.batch }} (${{ matrix.os }})
runs-on: ${{ matrix.os }}
# Backstop against hung jobs: without this, GitHub's default limit is 6 hours (observed in
# run 29064865197, where a hung job burned the full 6h). Healthy runs peak at ~16 minutes
# warm-cache; 60 leaves ample headroom for cache-cold runs.
timeout-minutes: 60
# Marker groups are defined once here and combined in the "Compute marker expression" step.
# The catch-all batch is derived as the negation of all named groups, so adding a language to
# any group automatically removes it from catch-all -- the partition can never desync, and no
# test can fall through to "no batch". To add/rebalance: edit a MARKERS_* var, and (for a new
# group) add a case branch below + the batch name to the matrix list.
env:
MARKERS_JVM: "java or csharp or fsharp or kotlin or scala or groovy or clojure or bsl"
MARKERS_NATIVE: "cpp or rust or go or zig or pascal or swift" # swift is macOS-only (skips elsewhere via the guard)
# other-langs: grab-bag + scripting languages + haskell + qml. Haskell (GHC/HLS) and qml (qmlls) are
# installed only on Linux -- too slow/expensive to build on the macOS runner (Haskell), or not worth
# the cost of a cross-OS Qt pull (qmlls) -- and skip on Windows+macOS via the central conftest guard.
MARKERS_OTHER_LANGS: "ruby or php or lua or luau or bash or powershell or elixir or erlang or dart or haxe or haskell or terraform or rego or ansible or yaml or toml or markdown or latex or crystal or cue or fortran or ada or matlab or systemverilog or hlsl or msl or al or qml"
# niche: slow, mostly-cached toolchains. ocaml + haskell are the two slowest -- kept in separate
# batches (ocaml here, haskell in other-langs). nix + perl skip on Windows via the guard.
MARKERS_NICHE: "julia or r or perl or lean4 or nix or ocaml"
strategy:
fail-fast: false
matrix:
os: [ubuntu-latest, windows-latest, macos-latest]
python-version: ["3.11"]
# 5 batches, each installs ONLY the toolchains its languages need (steps gated by matrix.batch).
# Node/Python/uv stay in every batch (cheap, needed for the venv + auto-installed LSes).
# catch-all = web + python + the core framework.
batch: [jvm, native, other-langs, niche, catch-all]
# --- OS gating: which batch runs on which OS -----------------------------------------
# Default is the full cross-product (every batch on all three OSes); the `exclude` entries
# below carve out the exceptions. An OS-specific *language* inside an otherwise all-OS batch
# is handled one level down by the central conftest guard (swift -> macOS only; haskell ->
# not Windows), NOT here -- excludes drop a WHOLE batch from an OS.
# jvm, native, other-langs, catch-all -> ubuntu, windows, macOS
# niche -> ubuntu only (heavy/slow toolchains -- lean, ocaml,
# nix, julia, R, perl -- not worth the Win/macOS cost)
exclude:
- { os: windows-latest, batch: niche }
- { os: macos-latest, batch: niche }
steps:
- uses: actions/checkout@v4
- name: Free disk space
if: runner.os == 'Linux'
run: |
df -h
sudo rm -rf /usr/local/lib/android
sudo rm -rf /usr/share/dotnet
sudo rm -rf /opt/ghc
sudo rm -rf /opt/hostedtoolcache
sudo apt-get clean
sudo apt-get autoremove -y
docker system prune -af || true
df -h
- name: Set up Python ${{ matrix.python-version }}
uses: actions/setup-python@v4
with:
python-version: "${{ matrix.python-version }}"
- uses: actions/setup-go@v5
if: matrix.batch == 'native'
with:
go-version: ">=1.17.0"
- name: Set up Node.js
@@ -54,8 +116,10 @@ jobs:
mkdir -p $HOME/.cache/go-build
mkdir -p $HOME/go/bin
- name: Install uv
shell: bash
run: curl -LsSf https://astral.sh/uv/install.sh | sh
# Use the official action (not `curl | sh`): it adds uv to GITHUB_PATH (the real OS PATH),
# so the Python language server's subprocess `shutil.which("uv"/"uvx")` finds it on Windows.
# The bash installer only puts uv on the Git-Bash MSYS PATH, invisible to Windows processes.
uses: astral-sh/setup-uv@v5
- name: Cache uv virtualenv
id: cache-uv
uses: actions/cache@v3
@@ -75,12 +139,15 @@ jobs:
shell: bash
run: uv pip list
- name: Check formatting
# Lint is batch-independent; run it only once per OS instead of in every batch.
if: matrix.batch == 'catch-all'
shell: bash
run: uv run poe lint
# Add Go bin directory to PATH for this workflow
# GITHUB_PATH is a special file that GitHub Actions uses to modify PATH
# Writing to this file adds the directory to the PATH for subsequent steps
- name: Cache Go binaries
if: matrix.batch == 'native'
id: cache-go-binaries
uses: actions/cache@v3
with:
@@ -89,15 +156,9 @@ jobs:
~/.cache/go-build
key: go-binaries-${{ runner.os }}-gopls-latest
- name: Install gopls
if: steps.cache-go-binaries.outputs.cache-hit != 'true'
if: matrix.batch == 'native' && steps.cache-go-binaries.outputs.cache-hit != 'true'
shell: bash
run: go install golang.org/x/tools/gopls@latest
- name: Set up Elixir
if: runner.os != 'Windows'
uses: erlef/setup-beam@v1
with:
elixir-version: "1.19.3"
otp-version: "28"
# Erlang currently not tested in CI, random hangings on macos, always hangs on ubuntu
# In local tests, erlang seems to work though
# - name: Install Erlang Language Server
@@ -113,10 +174,12 @@ jobs:
# # Ensure erlang_ls is in PATH
# echo "$HOME/.local/bin" >> $GITHUB_PATH
- name: Install clojure tools
if: matrix.batch == 'jvm'
uses: DeLaGuardo/setup-clojure@13.4
with:
cli: latest
- name: Install ccls (C/C++ Language Server)
if: matrix.batch == 'native'
shell: bash
run: |
if [[ "${{ runner.os }}" == "Linux" ]]; then
@@ -135,18 +198,22 @@ jobs:
exit 1
fi
- name: Setup Java (for JVM based languages)
if: matrix.batch == 'jvm'
uses: actions/setup-java@v4
with:
distribution: 'temurin'
java-version: '21'
- name: Setup .NET SDK (for F# and C# languages)
if: matrix.batch == 'jvm'
uses: actions/setup-dotnet@v4
with:
dotnet-version: '10.0.x'
- name: List .NET runtimes
if: matrix.batch == 'jvm'
shell: bash
run: dotnet --list-runtimes
- name: Install Terraform
if: matrix.batch == 'other-langs'
uses: hashicorp/setup-terraform@v3
with:
terraform_version: "1.5.0"
@@ -157,148 +224,87 @@ jobs:
# Installation of swift with the action screws with installation of ruby on macOS for some reason
# We can try again when version 3 of the action is released, where they will also use swiftly
# Until then, we use custom code to install swift. Sourcekit-lsp is installed automatically with swift
- name: Cache Swift toolchain (swiftly)
if: runner.os == 'macOS' && matrix.batch == 'native'
id: cache-swiftly
uses: actions/cache@v3
with:
# macOS keeps the toolchain (incl. sourcekit-lsp) under ~/Library/Developer/Toolchains,
# NOT under ~/.swiftly -- both must be cached or the toolchain is lost on restore.
path: |
~/.swiftly
~/Library/Developer/Toolchains
key: swiftly-${{ runner.os }}-6.1.2-v2
- name: Install Swift with swiftly (macOS)
if: runner.os == 'macOS'
if: runner.os == 'macOS' && matrix.batch == 'native'
run: |
echo "=== Installing swiftly on macOS ==="
curl -O https://download.swift.org/swiftly/darwin/swiftly.pkg && \
installer -pkg swiftly.pkg -target CurrentUserHomeDirectory && \
~/.swiftly/bin/swiftly init --quiet-shell-followup && \
. "${SWIFTLY_HOME_DIR:-$HOME/.swiftly}/env.sh" && \
set -e
# Install swiftly itself only if its binary isn't already present (e.g. restored from cache).
if [ ! -x "$HOME/.swiftly/bin/swiftly" ]; then
echo "=== Installing swiftly ==="
curl -O https://download.swift.org/swiftly/darwin/swiftly.pkg
installer -pkg swiftly.pkg -target CurrentUserHomeDirectory
"$HOME/.swiftly/bin/swiftly" init --quiet-shell-followup
fi
. "${SWIFTLY_HOME_DIR:-$HOME/.swiftly}/env.sh"
hash -r
swiftly install --use 6.1.2
swiftly use 6.1.2
echo "~/.swiftly/bin" >> $GITHUB_PATH
echo "Swiftly installed successfully"
# Verify sourcekit-lsp is working before proceeding
echo "=== Verifying sourcekit-lsp installation ==="
which sourcekit-lsp || echo "Warning: sourcekit-lsp not found in PATH"
sourcekit-lsp --help || echo "Warning: sourcekit-lsp not responding"
- name: Install Swift with swiftly (Ubuntu)
if: runner.os == 'Linux'
run: |
echo "=== Installing swiftly on Ubuntu ==="
# Install dependencies BEFORE Swift to avoid exit code 1
sudo apt-get update
sudo apt-get -y install libcurl4-openssl-dev
curl -O https://download.swift.org/swiftly/linux/swiftly-$(uname -m).tar.gz && \
tar zxf swiftly-$(uname -m).tar.gz && \
./swiftly init --quiet-shell-followup && \
. "${SWIFTLY_HOME_DIR:-$HOME/.local/share/swiftly}/env.sh" && \
hash -r
swiftly install --use 6.1.2
swiftly use 6.1.2
echo "=== Adding Swift toolchain to PATH ==="
echo "$HOME/.local/share/swiftly/bin" >> $GITHUB_PATH
echo "Swiftly installed successfully!"
# Verify sourcekit-lsp is working before proceeding
echo "=== Verifying sourcekit-lsp installation ==="
which sourcekit-lsp || echo "Warning: sourcekit-lsp not found in PATH"
sourcekit-lsp --help || echo "Warning: sourcekit-lsp not responding"
# Always run install: it downloads on a cache miss and is a fast no-op when the toolchain is
# genuinely present from cache. No `--use` -- that triggers the interactive .swift-version prompt.
swiftly install 6.1.2
# Locate sourcekit-lsp directly (macOS toolchains live under ~/Library/Developer/Toolchains)
# and put its directory on PATH -- avoids `swiftly use`, which prompts/hangs in CI.
SK="$(find "$HOME/Library/Developer/Toolchains" "$HOME/.swiftly" -name sourcekit-lsp 2>/dev/null | head -1)"
if [ -z "$SK" ]; then
echo "ERROR: sourcekit-lsp not found after Swift install. Searched:"
ls -la "$HOME/Library/Developer/Toolchains" 2>/dev/null || echo " (no ~/Library/Developer/Toolchains)"
ls -la "$HOME/.swiftly" 2>/dev/null || echo " (no ~/.swiftly)"
exit 1
fi
echo "sourcekit-lsp: $SK"
dirname "$SK" >> "$GITHUB_PATH"
- name: Install Ruby
if: matrix.batch == 'other-langs'
uses: ruby/setup-ruby@v1
with:
ruby-version: '3.4'
- name: Install Ruby language server
if: matrix.batch == 'other-langs'
shell: bash
run: gem install ruby-lsp
- name: Install OCaml and opam
uses: ocaml/setup-ocaml@v3
with:
ocaml-compiler: ${{ runner.os == 'Windows' && '4.14' || '5.3.x' }}
dune-cache: true
opam-repositories: |
${{ runner.os == 'Windows' && 'opam-repository-mingw: https://github.com/ocaml-opam/opam-repository-mingw.git#sunset' || '' }}
default: https://github.com/ocaml/opam-repository.git
- name: Install OCaml packages
shell: bash
run: |
if [ "$RUNNER_OS" = "Windows" ]; then
opam install -y dune ocaml-lsp-server
else
# Require ocaml-lsp-server >= 1.23.0 for cross-file reference support
opam install -y dune 'ocaml-lsp-server>=1.23.0'
fi
- name: Install R
uses: r-lib/actions/setup-r@v2
with:
r-version: '4.4.2'
use-public-rspm: true
- name: Install R sysdeps (libuv for fs package)
if: runner.os == 'Linux'
shell: bash
run: sudo apt-get update && sudo apt-get install -y libuv1-dev
- name: Install R language server
shell: bash
run: |
Rscript -e "install.packages('languageserver', repos='https://cloud.r-project.org')"
- name: Set up Julia
uses: julia-actions/setup-julia@v2
with:
version: '1.12.6'
- name: Install Julia LanguageServer
shell: bash
run: julia -e 'using Pkg; Pkg.add(Pkg.PackageSpec(name="LanguageServer", version="5.0.0"))'
- name: Setup Haskell toolchain
if: runner.os != 'Windows'
uses: haskell/ghcup-setup@v1
with:
ghc: '9.12.2'
cabal: '3.10.3.0'
hls: '2.11.0.0'
- name: Verify Haskell tools
if: runner.os != 'Windows'
run: |
echo "Verifying installed Haskell tools:"
which ghc && ghc --version
which cabal && cabal --version
# HLS verification - non-blocking in case of version incompatibility
if command -v haskell-language-server-wrapper &>/dev/null; then
echo "Found haskell-language-server-wrapper"
haskell-language-server-wrapper --version || echo "WARNING: HLS wrapper found but version check failed"
elif command -v haskell-language-server &>/dev/null; then
echo "Found haskell-language-server"
haskell-language-server --version || echo "WARNING: HLS found but version check failed"
else
echo "WARNING: HLS not found (may be incompatible with GHC 9.12.2)"
echo "This is not a critical error - tests will use HLS if available at runtime"
fi
shell: bash
- name: Pre-build Haskell test project for HLS
if: runner.os != 'Windows'
run: |
cd test/resources/repos/haskell/test_repo
cabal update
cabal build --only-dependencies
cabal build
echo "Haskell test project built successfully"
shell: bash
- name: Install Zig
if: matrix.batch == 'native'
uses: mlugg/setup-zig@v2
with:
version: 0.14.1
- name: Cache ZLS (Zig Language Server)
if: matrix.batch == 'native'
id: cache-zls
uses: actions/cache@v3
with:
path: ~/.zls
key: zls-${{ runner.os }}-0.14.0
- name: Install ZLS (Zig Language Server)
if: matrix.batch == 'native'
shell: bash
run: |
if [[ "${{ runner.os }}" == "Linux" ]]; then
wget https://github.com/zigtools/zls/releases/download/0.14.0/zls-x86_64-linux.tar.xz
tar -xf zls-x86_64-linux.tar.xz
sudo mv zls /usr/local/bin/
rm zls-x86_64-linux.tar.xz
elif [[ "${{ runner.os }}" == "macOS" ]]; then
wget https://github.com/zigtools/zls/releases/download/0.14.0/zls-x86_64-macos.tar.xz
tar -xf zls-x86_64-macos.tar.xz
sudo mv zls /usr/local/bin/
rm zls-x86_64-macos.tar.xz
elif [[ "${{ runner.os }}" == "Windows" ]]; then
curl -L -o zls.zip https://github.com/zigtools/zls/releases/download/0.14.0/zls-x86_64-windows.zip
unzip -o zls.zip
mkdir -p "$HOME/bin"
mv zls.exe "$HOME/bin/"
echo "$HOME/bin" >> $GITHUB_PATH
rm zls.zip
ZLS_DIR="$HOME/.zls"
if [[ "${{ steps.cache-zls.outputs.cache-hit }}" != "true" ]]; then
mkdir -p "$ZLS_DIR"
if [[ "${{ runner.os }}" == "Linux" ]]; then
wget -O /tmp/zls.tar.xz https://github.com/zigtools/zls/releases/download/0.14.0/zls-x86_64-linux.tar.xz
tar -xf /tmp/zls.tar.xz -C "$ZLS_DIR" && rm /tmp/zls.tar.xz
elif [[ "${{ runner.os }}" == "macOS" ]]; then
wget -O /tmp/zls.tar.xz https://github.com/zigtools/zls/releases/download/0.14.0/zls-x86_64-macos.tar.xz
tar -xf /tmp/zls.tar.xz -C "$ZLS_DIR" && rm /tmp/zls.tar.xz
elif [[ "${{ runner.os }}" == "Windows" ]]; then
curl -L -o zls.zip https://github.com/zigtools/zls/releases/download/0.14.0/zls-x86_64-windows.zip
unzip -o zls.zip -d "$ZLS_DIR" && rm zls.zip
fi
chmod +x "$ZLS_DIR/zls" 2>/dev/null || true
fi
echo "$ZLS_DIR" >> "$GITHUB_PATH"
- name: Install verible-verilog-ls (SystemVerilog Language Server)
if: matrix.batch == 'other-langs'
shell: bash
run: |
VERIBLE_VERSION="v0.0-4051-g9fdb4057"
@@ -329,6 +335,7 @@ jobs:
echo "WARNING: verible-verilog-ls not found in PATH"
fi
- name: Install Lua Language Server
if: matrix.batch == 'other-langs'
shell: bash
run: |
LUA_LS_VERSION="3.15.0"
@@ -373,61 +380,23 @@ jobs:
echo "$LUA_LS_DIR/bin" >> $GITHUB_PATH
rm lua-ls.zip
fi
- name: Install Perl::LanguageServer
if: runner.os != 'Windows'
shell: bash
run: |
if [[ "${{ runner.os }}" == "Linux" ]]; then
sudo apt-get update
sudo apt-get install -y cpanminus build-essential libanyevent-perl libio-aio-perl
elif [[ "${{ runner.os }}" == "macOS" ]]; then
brew install cpanminus
fi
PERL_MM_USE_DEFAULT=1 cpanm --notest --force Perl::LanguageServer
# Set up Perl local::lib environment for subsequent steps
echo "PERL5LIB=$HOME/perl5/lib/perl5${PERL5LIB:+:${PERL5LIB}}" >> $GITHUB_ENV
echo "PERL_LOCAL_LIB_ROOT=$HOME/perl5${PERL_LOCAL_LIB_ROOT:+:${PERL_LOCAL_LIB_ROOT}}" >> $GITHUB_ENV
echo "PERL_MB_OPT=--install_base \"$HOME/perl5\"" >> $GITHUB_ENV
echo "PERL_MM_OPT=INSTALL_BASE=$HOME/perl5" >> $GITHUB_ENV
echo "$HOME/perl5/bin" >> $GITHUB_PATH
- name: Install ansible-core and ansible-lint (for Ansible language server tests)
if: matrix.batch == 'other-langs'
shell: bash
run: uv run pip install ansible-core ansible-lint
- name: Install Haxe
if: matrix.batch == 'other-langs'
# The Haxe LS binary (server.js) is auto-downloaded and runs on Node.js,
# but it delegates to the Haxe compiler for code analysis at runtime.
uses: krdlab/setup-haxe@v2
with:
haxe-version: 4.3.7
- name: Install Elm
if: matrix.batch == 'catch-all'
shell: bash
run: npm install -g elm@0.19.1-6
- name: Install Nix
if: runner.os != 'Windows' # Nix doesn't support Windows natively
uses: cachix/install-nix-action@v30
with:
nix_path: nixpkgs=channel:nixos-unstable
- name: Install nixd (Nix Language Server)
if: runner.os != 'Windows' # Skip on Windows since Nix isn't available
shell: bash
run: |
# Install nixd using nix
nix profile install github:nix-community/nixd
# Verify nixd is installed and working
if ! command -v nixd &> /dev/null; then
echo "nixd installation failed or not in PATH"
exit 1
fi
echo "$HOME/.nix-profile/bin" >> $GITHUB_PATH
- name: Verify Nix package build
if: runner.os != 'Windows' # Nix only supported on Linux/macOS
shell: bash
run: |
# Verify the flake builds successfully
nix build --no-link
- name: Install Regal (Rego Language Server)
if: matrix.batch == 'other-langs'
shell: bash
run: |
REGAL_VERSION="0.39.0"
@@ -454,35 +423,61 @@ jobs:
mv regal.exe "$HOME/bin/"
echo "$HOME/bin" >> $GITHUB_PATH
fi
- name: Install Free Pascal Compiler
- name: Install qmlls (QML Language Server)
# qmlls needs Qt 6.6+ for textDocument/documentSymbol (the feature these tests assert); Ubuntu
# noble's apt only has Qt 6.4.2, whose qmlls returns -32601 "method not found" for it. So install
# the official standalone build (TheQtCompanyRnD/qmlls-workflow, linked from the Qt docs) -- an 8 MB
# single binary built from recent Qt dev. Restricted to Linux (a cross-OS Qt pull isn't worth it);
# QML skips on Windows/macOS via the conftest guard. libodbc2 is its only non-bundled runtime dep.
if: matrix.batch == 'other-langs' && runner.os == 'Linux'
shell: bash
run: |
QMLLS_VERSION="0.7"
QMLLS_DIR="$HOME/.qmlls"
mkdir -p "$QMLLS_DIR"
curl -fsSL -o /tmp/qmlls.zip "https://github.com/TheQtCompanyRnD/qmlls-workflow/releases/download/${QMLLS_VERSION}/qmllanguageserver-linux-x64-${QMLLS_VERSION}.zip"
unzip -o /tmp/qmlls.zip -d "$QMLLS_DIR" >/dev/null
rm /tmp/qmlls.zip
chmod +x "$QMLLS_DIR/qmlls"
# libodbc.so.2 is qmlls's only non-bundled runtime dep; install only if absent on the image.
ldconfig -p | grep -q 'libodbc.so.2' || { sudo apt-get update && sudo apt-get install -y libodbc2; }
# Fail loudly if any shared lib is still unresolved (e.g. a missing runtime dep).
missing="$(ldd "$QMLLS_DIR/qmlls" | grep 'not found' || true)"
[ -z "$missing" ] || { echo "ERROR: qmlls has unresolved runtime libs:"; echo "$missing"; exit 1; }
echo "$QMLLS_DIR" >> "$GITHUB_PATH"
- name: Cache Free Pascal Compiler
if: matrix.batch == 'native'
id: cache-fpc
uses: actions/cache@v3
with:
path: |
~/fpc
~/fpcsrc
key: fpc-${{ runner.os }}-3.2.2
- name: Install Free Pascal Compiler
if: matrix.batch == 'native'
shell: bash
run: |
# Only the downloads are gated on the cache; the PP/FPCDIR/PATH detection always runs
# (it works the same on freshly-downloaded or cache-restored ~/fpc + ~/fpcsrc).
FPC_CACHE_HIT="${{ steps.cache-fpc.outputs.cache-hit }}"
FPC_VERSION="3.2.2"
if [[ "${{ runner.os }}" == "Linux" ]]; then
sudo apt-get update
sudo apt-get install -y fpc fpc-source
# Set environment variables for pasls
echo "PP=/usr/bin/fpc" >> $GITHUB_ENV
# Find FPC source directory (version may vary) - remove trailing slash
FPCDIR=$(ls -d /usr/share/fpcsrc/*/ 2>/dev/null | head -1 | sed 's:/$::')
if [[ -z "$FPCDIR" ]]; then
FPCDIR="/usr/share/fpcsrc"
fi
[[ -z "$FPCDIR" ]] && FPCDIR="/usr/share/fpcsrc"
echo "FPCDIR=$FPCDIR" >> $GITHUB_ENV
echo "=== FPC source directory structure ==="
ls -la "$FPCDIR" || echo "FPCDIR not found"
ls -la "$FPCDIR/rtl" 2>/dev/null || echo "rtl subdirectory not found"
elif [[ "${{ runner.os }}" == "macOS" ]]; then
brew install fpc
# Download FPC source from SourceForge (fpc-src-laz cask is incompatible with ARM64)
FPC_VERSION="3.2.2"
curl -L -o fpc-source.tar.gz "https://sourceforge.net/projects/freepascal/files/Source/${FPC_VERSION}/fpc-${FPC_VERSION}.source.tar.gz/download"
mkdir -p "$HOME/fpcsrc"
tar -xzf fpc-source.tar.gz -C "$HOME/fpcsrc"
rm fpc-source.tar.gz
# Check extracted directory structure (might be nested)
echo "=== Extracted FPC source structure ==="
ls -la "$HOME/fpcsrc"
# Find the actual FPC source root (contains rtl, packages, etc.)
command -v fpc >/dev/null || brew install fpc
if [[ "$FPC_CACHE_HIT" != "true" ]]; then
# FPC source from SourceForge (fpc-src-laz cask is incompatible with ARM64)
curl -L -o fpc-source.tar.gz "https://sourceforge.net/projects/freepascal/files/Source/${FPC_VERSION}/fpc-${FPC_VERSION}.source.tar.gz/download"
mkdir -p "$HOME/fpcsrc"
tar -xzf fpc-source.tar.gz -C "$HOME/fpcsrc"
rm fpc-source.tar.gz
fi
if [[ -d "$HOME/fpcsrc/fpc-${FPC_VERSION}/rtl" ]]; then
FPCDIR="$HOME/fpcsrc/fpc-${FPC_VERSION}"
elif [[ -d "$HOME/fpcsrc/fpc-${FPC_VERSION}/fpc-${FPC_VERSION}/rtl" ]]; then
@@ -492,34 +487,26 @@ jobs:
fi
echo "PP=$(which fpc)" >> $GITHUB_ENV
echo "FPCDIR=$FPCDIR" >> $GITHUB_ENV
echo "=== FPC source directory ==="
ls -la "$FPCDIR" || echo "FPCDIR not found"
elif [[ "${{ runner.os }}" == "Windows" ]]; then
FPC_VERSION="3.2.2"
# Download freepascal-ootb (includes FPC compiler)
curl -L -o fpc-ootb.zip https://github.com/fredvs/freepascal-ootb/releases/download/${FPC_VERSION}/fpc-ootb-322-x86_64-win64.zip
mkdir -p "$HOME/fpc"
unzip -q fpc-ootb.zip -d "$HOME/fpc"
rm fpc-ootb.zip
# Download FPC source from SourceForge (fpc-ootb only has compiled units, not source)
curl -L -o fpc-source.zip "https://sourceforge.net/projects/freepascal/files/Source/${FPC_VERSION}/fpc-${FPC_VERSION}.source.zip/download"
mkdir -p "$HOME/fpcsrc"
unzip -q fpc-source.zip -d "$HOME/fpcsrc"
rm fpc-source.zip
# Find fpc executable (fpc-ootb uses fpc-ootb.exe as the compiler)
echo "=== FPC directory structure ==="
find "$HOME/fpc" -name "*.exe" -type f 2>/dev/null | head -10
if [[ "$FPC_CACHE_HIT" != "true" ]]; then
# freepascal-ootb (compiler) + source (ootb ships compiled units only, not source)
curl -L -o fpc-ootb.zip https://github.com/fredvs/freepascal-ootb/releases/download/${FPC_VERSION}/fpc-ootb-322-x86_64-win64.zip
mkdir -p "$HOME/fpc"
unzip -q fpc-ootb.zip -d "$HOME/fpc"
rm fpc-ootb.zip
curl -L -o fpc-source.zip "https://sourceforge.net/projects/freepascal/files/Source/${FPC_VERSION}/fpc-${FPC_VERSION}.source.zip/download"
mkdir -p "$HOME/fpcsrc"
unzip -q fpc-source.zip -d "$HOME/fpcsrc"
rm fpc-source.zip
fi
FPC_EXE=$(find "$HOME/fpc" -name "fpc-ootb-64.exe" -type f 2>/dev/null | head -1)
echo "Found FPC executable: $FPC_EXE"
echo "Found FPC source dir: $HOME/fpcsrc/fpc-${FPC_VERSION}"
# Set environment variables for pasls
echo "PP=$FPC_EXE" >> $GITHUB_ENV
echo "FPCDIR=$HOME/fpcsrc/fpc-${FPC_VERSION}" >> $GITHUB_ENV
# Add FPC bin directory to PATH
FPC_BIN_DIR=$(dirname "$FPC_EXE")
echo "$FPC_BIN_DIR" >> $GITHUB_PATH
echo "$(dirname "$FPC_EXE")" >> $GITHUB_PATH
fi
- name: Verify FPC installation
if: matrix.batch == 'native'
shell: bash
run: |
echo "=== Environment variables ==="
@@ -560,11 +547,154 @@ jobs:
echo "ERROR: FPCDIR does not exist: $FPCDIR"
exit 1
fi
# --- Slow / heavy language toolchains, each isolated in its own batch (OS gating via matrix excludes) ---
- name: Setup Haskell toolchain
if: matrix.batch == 'other-langs' && runner.os == 'Linux'
uses: haskell/ghcup-setup@v1
with:
ghc: '9.12.2'
cabal: '3.10.3.0'
hls: '2.11.0.0'
- name: Verify Haskell tools
if: matrix.batch == 'other-langs' && runner.os == 'Linux'
shell: bash
run: |
which ghc && ghc --version
which cabal && cabal --version
if command -v haskell-language-server-wrapper &>/dev/null; then
haskell-language-server-wrapper --version || echo "WARNING: HLS wrapper version check failed"
else
echo "WARNING: haskell-language-server-wrapper not found (may be incompatible with GHC 9.12.2)"
fi
- name: Pre-build Haskell test project for HLS
if: matrix.batch == 'other-langs' && runner.os == 'Linux'
shell: bash
run: |
cd test/resources/repos/haskell/test_repo
cabal update
cabal build --only-dependencies
cabal build
- name: Install OCaml and opam
if: matrix.batch == 'niche'
uses: ocaml/setup-ocaml@v3
with:
ocaml-compiler: ${{ runner.os == 'Windows' && '4.14' || '5.3.x' }}
dune-cache: true
opam-repositories: |
${{ runner.os == 'Windows' && 'opam-repository-mingw: https://github.com/ocaml-opam/opam-repository-mingw.git#sunset' || '' }}
default: https://github.com/ocaml/opam-repository.git
- name: Install OCaml packages
if: matrix.batch == 'niche'
shell: bash
run: |
if [ "$RUNNER_OS" = "Windows" ]; then
opam install -y dune ocaml-lsp-server
else
opam install -y dune 'ocaml-lsp-server>=1.23.0'
fi
- name: Install Nix
if: matrix.batch == 'niche' && runner.os != 'Windows'
uses: cachix/install-nix-action@v30
with:
nix_path: nixpkgs=channel:nixos-unstable
# Authenticate any GitHub fetch Nix makes during evaluation/tests with the runner's
# GITHUB_TOKEN (raises the unauthenticated 60 req/hr-per-IP API limit to ~1000/hr).
# nixd itself is installed from the channel below, so this is just insurance.
extra_nix_config: |
access-tokens = github.com=${{ secrets.GITHUB_TOKEN }}
- name: Install nixd (Nix Language Server)
if: matrix.batch == 'niche' && runner.os != 'Windows'
shell: bash
run: |
# Install nixd from the nixos-unstable channel (set as <nixpkgs> on NIX_PATH via nix_path
# above) using classic nix-env. `-f '<nixpkgs>'` resolves the channel tarball from
# nixos.org and selects the `nixd` attribute -- it never touches the GitHub API. (The flake
# form `nix profile install github:nix-community/nixd` resolves an unpinned HEAD through
# api.github.com/.../commits/HEAD, which GitHub 429-throttles on CI even with a token;
# and `nix-env -iA nixpkgs.nixd` fails here because no channel named `nixpkgs` is registered.)
nix-env -f '<nixpkgs>' -iA nixd
if ! command -v nixd &> /dev/null; then
echo "nixd installation failed or not in PATH"; exit 1
fi
echo "$HOME/.nix-profile/bin" >> $GITHUB_PATH
- name: Build Lean 4 test project
if: matrix.batch == 'niche'
uses: leanprover/lean-action@v1
with:
lake-package-directory: test/resources/repos/lean4/test_repo
# --- niche batch: R, Julia, Perl ---
- name: Install R
if: matrix.batch == 'niche'
uses: r-lib/actions/setup-r@v2
with:
r-version: '4.4.2'
use-public-rspm: true
- name: Cache R packages (languageserver + its deps)
if: matrix.batch == 'niche'
id: cache-r
uses: actions/cache@v3
with:
# setup-r exports R_LIBS_USER as $RUNNER_TEMP/Library (a stable path) and re-exports it on
# every run -- including cache hits -- so R finds the restored packages with no extra env.
# Bump the trailing -vN (or r-version) to invalidate after an R/OS-image upgrade.
path: ${{ env.R_LIBS_USER }}
key: r-pkgs-${{ runner.os }}-R4.4.2-languageserver-v1
- name: Install R sysdeps (libuv for fs package)
# Always run (NOT gated on the cache): the cached, compiled `fs` package links libuv at
# runtime, so the shared lib must be present even when the build below is skipped.
if: matrix.batch == 'niche' && runner.os == 'Linux'
shell: bash
run: sudo apt-get update && sudo apt-get install -y libuv1-dev
- name: Install R language server
if: matrix.batch == 'niche' && steps.cache-r.outputs.cache-hit != 'true'
shell: bash
run: Rscript -e "install.packages('languageserver', repos='https://cloud.r-project.org')"
- name: Set up Julia
if: matrix.batch == 'niche'
uses: julia-actions/setup-julia@v2
with:
version: '1.12.6'
- name: Cache Julia depot (packages + precompiled)
# Caches the depot (~/.julia: packages, artifacts, compiled, ...) with an auto-managed key,
# so the LanguageServer Pkg.add below is a near-no-op (already installed + precompiled) on
# warm runs. The action handles save/restore itself -- no cache-hit gating needed.
if: matrix.batch == 'niche'
uses: julia-actions/cache@v2
- name: Install Julia LanguageServer
if: matrix.batch == 'niche'
shell: bash
run: julia -e 'using Pkg; Pkg.add(Pkg.PackageSpec(name="LanguageServer", version="5.0.0"))'
- name: Cache Perl::LanguageServer (~/perl5)
if: matrix.batch == 'niche' && runner.os != 'Windows'
id: cache-perl
uses: actions/cache@v3
with:
path: ~/perl5
key: perl-ls-${{ runner.os }}-v1
- name: Install Perl::LanguageServer
if: matrix.batch == 'niche' && runner.os != 'Windows'
shell: bash
run: |
# System deps are cheap and needed at runtime (the cached LS links libanyevent/libio-aio),
# so install them always; the slow cpanm build (XS compile of Coro etc.) runs only on a miss.
if [[ "${{ runner.os }}" == "Linux" ]]; then
sudo apt-get update
sudo apt-get install -y cpanminus build-essential libanyevent-perl libio-aio-perl
elif [[ "${{ runner.os }}" == "macOS" ]]; then
brew install cpanminus
fi
if [[ "${{ steps.cache-perl.outputs.cache-hit }}" != "true" ]]; then
# --local-lib pins the install to ~/perl5 (matching the PERL5LIB exports below), so the
# cached directory is exactly what `perl -MPerl::LanguageServer` loads on a restore.
PERL_MM_USE_DEFAULT=1 cpanm --notest --force --local-lib="$HOME/perl5" Perl::LanguageServer
fi
echo "PERL5LIB=$HOME/perl5/lib/perl5${PERL5LIB:+:${PERL5LIB}}" >> $GITHUB_ENV
echo "PERL_LOCAL_LIB_ROOT=$HOME/perl5${PERL_LOCAL_LIB_ROOT:+:${PERL_LOCAL_LIB_ROOT}}" >> $GITHUB_ENV
echo "PERL_MB_OPT=--install_base \"$HOME/perl5\"" >> $GITHUB_ENV
echo "PERL_MM_OPT=INSTALL_BASE=$HOME/perl5" >> $GITHUB_ENV
echo "$HOME/perl5/bin" >> $GITHUB_PATH
- name: Install npm deps for Angular test repo
if: matrix.batch == 'catch-all'
shell: bash
run: |
if [ -d test/resources/repos/angular/test_repo ]; then
@@ -576,17 +706,28 @@ jobs:
uses: actions/cache@v3
with:
path: ~/.serena/language_servers/static
key: language-servers-${{ runner.os }}-v1
key: language-servers-${{ runner.os }}-${{ matrix.batch }}-v1
restore-keys: |
language-servers-${{ runner.os }}-
- name: Report free disk space
if: runner.os == 'Linux'
language-servers-${{ runner.os }}-${{ matrix.batch }}-
- name: Compute marker expression for this batch
id: markers
shell: bash
run: |
echo "Free disk space before tests:"
df -h
case "${{ matrix.batch }}" in
jvm) expr="$MARKERS_JVM" ;;
native) expr="$MARKERS_NATIVE" ;;
other-langs) expr="$MARKERS_OTHER_LANGS" ;;
niche) expr="$MARKERS_NICHE" ;;
catch-all) expr="not ($MARKERS_JVM or $MARKERS_NATIVE or $MARKERS_OTHER_LANGS or $MARKERS_NICHE)" ;;
*) echo "Unknown batch: ${{ matrix.batch }}" >&2; exit 1 ;;
esac
echo "Marker expression: $expr"
echo "expr=$expr" >> "$GITHUB_OUTPUT"
- name: Test with pytest
shell: bash
run: uv run poe test -q --tb=short
- name: Type-checking with mypy
run: uv run poe test -m "${{ steps.markers.outputs.expr }}" -q --tb=short
- name: Type-checking with ty
# Type-checking is batch-independent; run it only once per OS instead of in every batch.
if: matrix.batch == 'catch-all'
shell: bash
run: uv run poe type-check
.serena/memories/adding_new_language_support_guide.md
+44 -17
View File
@@ -33,51 +33,78 @@ To implement a new language server using the DependencyProvider pattern:
```
The resource dir that is passed is the directory in which installed dependencies should be stored!
**Base Classes:**
**Base Classes** (choose the most specific one that fits):
- **`LanguageServerDependencyProviderSinglePath`** - For language servers with a single core dependency (e.g., an executable or JAR file)
- Provides automatic support for the `ls_path` custom setting, allowing users to override the core dependency path (if they have it installed it themselves)
- **`LanguageServerDependencyProviderUvx`** - For language servers distributed as a PyPI package, run on demand
via `uvx` / `uv x` (no installation step to implement)
- Simply instantiate it with the package name, pinned default version, entrypoint (console script) and optional `extra_args`;
the version can be overridden by the user via the configured `version_setting_key` custom setting
- Reference implementation: `PyrightServer`
- **`LanguageServerDependencyProviderBaseCommand`** - For the common case where the
launch command is constructed from a *base command* (which the user can override via custom settings; handled generically)
- Implement `_create_default_base_command()` to return the default base command (executable + args), downloading/installing
dependencies beforehand if necessary
- Implement `_create_launch_command_from_base_command(base_command)` to add any further arguments, producing the
final launch command
- **`LanguageServerDependencyProviderSinglePath`** - Alternative to inheriting from `...BaseCommand` directly for the case
of a single core dependency (e.g., an executable or JAR file); use when the single path is not used directly as a base command
otherwise inherit from `...BaseCommand` directly
- Implement `_get_or_install_core_dependency()` to return the path to the core dependency, downloading/installing it automatically if necessary
- Implement `_create_launch_command(core_path)` to build the full command from the core path
- Reference implementations: `TypeScriptLanguageServer`, `Intelephense`, `ClojureLSP`, `ClangdLanguageServer`, `PyrightServer`
- Reference implementations: `TypeScriptLanguageServer`, `Intelephense`, `ClojureLSP`, `ClangdLanguageServer`
- **`LanguageServerDependencyProvider`** - The base class, which can be directly inherited from for complex cases with multiple dependencies or custom setup
- Implement `create_launch_command()` directly
- **`LanguageServerDependencyProvider`** - The root base class, for complex cases with multiple dependencies or custom setup
- Implement `create_launch_command()` directly (note: no automatic support for user-level launch command overrides in this case)
- Reference implementations: `EclipseJDTLS`, `CSharpLanguageServer`, `MatlabLanguageServer`
**Implementation Pointers::**
- When returning the command, prefer the list-based representation for robustness
- Override `create_launch_command_env` if the launch command needs environment variables to be set (defaults to `{}` in the base implementation)
You should look at at least one existing implementation of each base class to understand how they work.
### 1.2 LSP Initialization
Override initialization methods if needed:
Override `_create_base_initialize_params` to provide server-specific initialization
parameters. The common keys — `processId`, `rootPath`, `rootUri`, `clientInfo` and
`workspaceFolders` — are set centrally by the `InitializeParamsBuilder` (see
`src/solidlsp/initialize_params.py`), so your override MUST NOT set them. Just return
the server-specific settings (typically `capabilities` and `initializationOptions`):
```python
def _get_initialize_params(self) -> InitializeParams:
"""Return language-specific initialization parameters."""
def _create_base_initialize_params(self) -> dict:
"""Return language-specific initialization parameters (server-specific keys only)."""
return {
"processId": os.getpid(),
"rootUri": PathUtils.path_to_uri(self.repository_root_path),
"capabilities": {
# Language-specific capabilities
}
},
# "initializationOptions": {...}, # if the server needs them
}
def _start_server(self):
"""Start the language server with custom handlers."""
# Set up notification handlers
self.server.on_notification("window/logMessage", self._handle_log_message)
# Start server and initialize
# Start server and initialize. Do NOT call _create_base_initialize_params directly;
# _create_initialize_params() wraps it with the builder to add the common keys.
self.server.start()
init_response = self.server.send.initialize(self._get_initialize_params())
init_response = self.server.send.initialize(self._create_initialize_params())
self.server.notify.initialized({})
```
Notes:
- The builder resolves `workspaceFolders` from the language server config (indexed
folders + `ls_additional_workspace_folders`); don't build the folder list yourself.
- To send a folder list nested inside `initializationOptions` (some servers, e.g.
`EclipseJDTLS`/`KotlinLanguageServer`, need this), set it explicitly there — only the
*top-level* `workspaceFolders` is builder-managed.
- To suppress the top-level `workspaceFolders` entirely, override
`_create_initialize_params_builder` and construct `DefaultInitializeParamsBuilder`
with `set_workspace_folders=False`.
After `_start_server` returns, the language server should be fully operational.
If the server requires that one waits for certain notifications or responses before being ready, implement that logic here.
For an example, see `EclipseJDTLS._start_server`.
.serena/memories/conventions.md
+11 -5
View File
@@ -13,14 +13,20 @@
- `ruff format` runs on `src scripts test`; same set for `ruff check`.
- mccabe complexity cap: 20.
## Typing (mypy strict)
- `disallow_untyped_defs = true`, `disallow_incomplete_defs = true`, `strict_equality`, `strict_optional`, `warn_unreachable`, `no_implicit_optional`.
- Excluded paths: `^build/`, `^docs/`, `^test/resources/`.
- Test files relax `disallow_untyped_defs` (still type-checked otherwise).
## Typing (ty)
- Type checker is **ty** (Astral), configured under `[tool.ty]` in `pyproject.toml`.
## Tests
- Language-server tests are pytest-marker-gated (one marker per language; see `pyproject.toml` `[tool.pytest.ini_options].markers`). Default `poe test` runs unmarked tests + whatever `PYTEST_MARKERS` selects.
- Snapshot tests use **syrupy** with custom `--snapshot-patch-pycharm-diff` plugin (auto-added via `addopts`).
## Tool descriptions (LLM-facing)
- To change how a tool is described to the model, edit the Tool class's `apply()` **docstring** (in
`src/serena/tools/*.py`) — `make_mcp_tool` parses the docstring body + `:param:` lines into the MCP tool
description/schema. Do NOT use `tool_description_overrides` in context ymls (e.g. `claude-code.yml`); the
docstring is the single source of truth and overrides drift. Keep `:param:` lines accurate.
## Memories
- Follow `mem:memory_maintenance` for any new/updated memory in `.serena/memories/`.
- Durable knowledge goes in memories/docs (not the assistant's auto-memory); see the monorepo top-level
`CLAUDE.md` for the cross-repo conventions (no benchmark-tuning of agent-facing strings, etc.).
.serena/memories/creating_pull_requests.md
+11
View File
@@ -0,0 +1,11 @@
# Creating Pull Requests
* Make sure the instructions in `CONTRIBUTING.md` regarding the scope of PRs
are followed.
* For changes that add features or fix problems,
add an entry to `CHANGELOG.md` which *concisely* describes the change.
Note that details/background information on the change should be in the commit
message, not in the changelog.
Make sure to follow the existing format and to add your change to the
appropriate section.
.serena/memories/serena_repository_structure.md
+1 -1
View File
@@ -104,7 +104,7 @@ The project uses modern Python tooling:
- **uv** for fast dependency resolution and virtual environments
- **pytest** for testing with language-specific markers (`@pytest.mark.csharp`)
- **ruff** for linting and formatting
- **mypy** for type checking
- **ty** (Astral) for type checking
## Build and Development
.serena/memories/suggested_commands.md
+1 -1
View File
@@ -6,7 +6,7 @@ Run via `uv run poe <task>` (or `poe <task>` inside the activated venv). Poe exe
- `poe test` — pytest on `test/` (per-language tests are marker-gated; pass `-m <marker>` to enable).
- `poe lint` — ruff format-check + ruff check (no fixes).
- `poe format` — ruff `--fix` then `ruff format` (mutates files).
- `poe type-check` — mypy on `src/serena`, `src/solidlsp`, and `test/` (test pass disables `no-untyped-def`).
- `poe type-check` — ty on `src/serena`, `src/solidlsp`, and `test/` (test pass relaxes pytest/mock-noisy rules via `[[tool.ty.overrides]]`).
- Single test file: `uv run pytest test/path/to/test_x.py -vv`. Language-gated: add `-m python` etc.
## Docs
.serena/memories/task_completion.md
+1 -1
View File
@@ -3,7 +3,7 @@
After any code change in `src/` or `test/`, run:
1. `uv run poe format` — applies ruff fixes + formatting (mutates files).
2. `uv run poe type-check` — mypy strict on `src/serena`, `src/solidlsp`, `test/`.
2. `uv run poe type-check` — ty on `src/serena`, `src/solidlsp`, `test/`.
3. `uv run poe test` — pytest on affected files or affected languages, using `-m` markers.
If prompt templates changed: `uv run python scripts/gen_prompt_factory.py` (regenerates `src/serena/generated/generated_prompt_factory.py`; use `uv run poe format` and commit the result).
.serena/memories/tech_stack.md
+1 -1
View File
@@ -5,6 +5,6 @@
- Build backend: `hatchling`. Packages: `src/serena`, `src/interprompt`, `src/solidlsp`.
- Task runner: **poethepoet** (`poe <task>`). Poe executor is `simple` (does NOT shell out via uv) — avoids env recreation while MCP server is running.
- Key runtime deps: `mcp`, `flask` (dashboard), `pydantic`, `pygls` + `lsprotocol` (LSP), `anthropic`, `jinja2`, `ruamel.yaml`, `pywebview`/`pystray` (GUI), `pythonnet` (Windows only).
- Dev deps: `ruff` (lint+format), `mypy` (strict), `pytest` (+ `pytest-xdist`, `pytest-timeout`, `syrupy` snapshots), `sphinx`/`jupyter-book` for docs.
- Dev deps: `ruff` (lint+format), `ty` (Astral type checker; replaced mypy), `pytest` (+ `pytest-xdist`, `pytest-timeout`, `syrupy` snapshots), `sphinx`/`jupyter-book` for docs.
- Optional extras: `agno` (Agno agent integration), `google` (gemini).
- LSP client core lives under `src/solidlsp/`; one subdir per supported language server under `language_servers/`.
.serena/project.yml
+58 -25
View File
@@ -1,26 +1,31 @@
# the name by which the project can be referenced within Serena
# the name by which the project can be referenced within Serena/when chatting with the LLM.
project_name: "serena"
# list of languages for which language servers are started; choose from:
# al ansible bash clojure cpp
# cpp_ccls crystal csharp csharp_omnisharp dart
# elixir elm erlang fortran fsharp
# list of languages for which language servers are started (LSP backend only); choose from:
# ada al angular ansible bash
# bsl clojure cpp cpp_ccls crystal
# csharp csharp_omnisharp cue dart elixir
# elm erlang fortran fsharp gdscript
# go groovy haskell haxe hlsl
# java json julia kotlin lean4
# lua luau markdown matlab msl
# nix ocaml pascal perl php
# php_phpactor powershell python python_jedi python_ty
# r rego ruby ruby_solargraph rust
# scala solidity swift systemverilog terraform
# toml typescript typescript_vts vue yaml
# zig
# (This list may be outdated. For the current list, see values of Language enum here:
# https://github.com/oraios/serena/blob/main/src/solidlsp/ls_config.py
# For some languages, there are alternative language servers, e.g. csharp_omnisharp, ruby_solargraph.)
# html java json julia kotlin
# latex lean4 lua luau markdown
# matlab msl nix ocaml pascal
# perl php php_phpactor php_phpantom powershell
# python python_jedi python_pyrefly python_ty r
# rego ruby ruby_solargraph rust scala
# scss solidity svelte swift systemverilog
# terraform toml typescript typescript_vts vue
# yaml zig
# (This list may be outdated; generated with scripts/print_language_list.py;
# For the current list, see values of Language enum here:
# https://github.com/oraios/serena/blob/main/src/solidlsp/ls_config.py)
# For some languages, there are alternative language servers, e.g. csharp_omnisharp, ruby_solargraph.)
# Note:
# - For C, use cpp
# - For JavaScript, use typescript
# - For Angular projects, use angular (subsumes typescript+html; requires `npm install` in the project root)
# - For Svelte projects, use svelte (subsumes typescript/javascript for .svelte projects; requires npm)
# - For SCSS / Sass / plain CSS, use scss (some-sass-language-server handles all three)
# - For Free Pascal/Lazarus, use pascal
# Special requirements:
# Some languages require additional setup/installations.
@@ -74,6 +79,9 @@ initial_prompt: |
When describing parameters, methods/functions and classes, you use a precise style, where the initial (elliptical) phrase
clearly defines *what* it is. Any details then follow in subsequent sentences.
Pull requests:
Read `mem:creating_pull_requests` when asked to participate in the creation of a pull request.
# the encoding used by text files in the project
# For a list of possible encodings, see https://docs.python.org/3.11/library/codecs.html#standard-encodings
encoding: utf-8
@@ -133,8 +141,8 @@ ignored_memory_patterns: []
# advanced configuration option allowing to configure language server-specific options.
# Maps the language key to the options.
# Have a look at the docstring of the constructors of the LS implementations within solidlsp (e.g., for C# or PHP) to see which options are available.
# No documentation on options means no options are available.
# The settings are considered only if the project is trusted (see global configuration to define trusted projects).
# See https://oraios.github.io/serena/02-usage/050_configuration.html#language-server-specific-settings
ls_specific_settings: {}
# list of mode names to be activated additionally for this project, e.g. ["query-projects"]
@@ -142,13 +150,38 @@ ls_specific_settings: {}
# See https://oraios.github.io/serena/02-usage/050_configuration.html#modes
added_modes:
# list of additional workspace folder paths for cross-package reference support (e.g. in monorepos).
# optional shell command to run before the language backend (LSP or JetBrains) is initialised.
# the command runs in the project root directory and is only executed if the project is trusted
# (see trusted_project_path_patterns in the global configuration).
# serena waits for the command to exit: a non-zero exit code is logged as an error but does not
# abort activation. a per-project timeout (activation_command_timeout, default 180s) is the safety
# backstop for non-terminating commands; on expiry the process is killed and activation continues.
# example: activation_command: "npx nx run-many -t build"
activation_command:
# maximum time in seconds to wait for activation_command to complete before killing it (default 180s).
# must be a positive number.
activation_command_timeout: 180.0
# list of workspace folder paths (LSP backend only).
# These folders will be used to build up Serena's symbol index.
# Paths must be within the project root and should thus be relative to the project root.
# Furthermore, the paths should not be filtered by ignore settings.
# Default setting: The entire project root folder (".") is considered.
# In (large) monorepos, this can be used to index only subfolders of the project root, e.g.
# ls_workspace_folders:
# - "./subproject1"
# - "./subproject2"
ls_workspace_folders:
- "."
# list of additional workspace folder paths for cross-package reference support.
# Paths can be absolute or relative to the project root.
# Each folder is registered as an LSP workspace folder, enabling language servers to discover
# symbols and references across package boundaries.
# Currently supported for: TypeScript.
# symbols and references across package boundaries, but these folders are not indexed by Serena,
# i.e. the respective symbols will not be found using Serena's symbol search tool.
# Example:
# additional_workspace_folders:
# - ../sibling-package
# - ../shared-lib
additional_workspace_folders: []
# - "../sibling-package"
# - "../shared-lib"
ls_additional_workspace_folders: []
CHANGELOG.md
+199 -7
View File
@@ -2,22 +2,203 @@
Status of the `main` branch. Changes prior to the next official version change will appear here.
* General:
- Fix: `FileUtils.read_file`'s `charset_normalizer` fallback (used when a file cannot be decoded with
the project's configured `encoding`) decoded the raw bytes directly and therefore skipped the
universal-newline translation that the primary read path applies. CR characters from disk thus
reached Serena's in-memory file contents, where the rest of the code assumes LF-normalized text and
the `line_ending` setting is meant to be the single point of line-ending translation on write. The
fallback now normalizes line endings to LF, consistently with the primary path.
- Fix: a symbol whose LSP range ended exactly one line past EOF, at column 0 (the convention for
a range covering whole lines through the end of the file), raised `IndexError` in
`SymbolBody.get_text`. That one well-defined case is now corrected to end at the actual last
line; any other out-of-range end position now raises `InvalidTextLocationError` instead,
rather than guessing at a body that could be wrong #1498
* Language Servers:
- `typescript_vts`: Add `initialization_options` setting in `ls_specific_settings.typescript_vts`.
The dict is forwarded to vtsls via `initializationOptions`, `workspace/didChangeConfiguration`,
and `workspace/configuration` pulls. Enables Yarn PnP setups with `typescript.tsdk` pointing
at the Yarn-generated SDK.
- `SvelteLanguageServer`: Fix diagnostics requests for TypeScript/JavaScript files incorrectly being
processed by the Svelte LS instead of the TypeScript LS.
- Improve quoting of arguments in shell executions
- Fix: Properly differentiate between raw and high-level symbol cache fingerprints, avoiding unnecessary
invalidations of the raw cache when only the derived high-level representation changes
- Fix: Language servers were not notified of external file system changes, causing some
symbolic operations (such as `find_referencing_symbols`) to report stale information.
An explicit file system polling mechanism is now used to detect changes prior to the affected
tool executions.
- Fix: Order of ignore patterns passed to language servers was not respected #1729
- Improve uv-based language server launch command compatibility: Use the more widely supported
`uv tool run` instead of `uv x` #1721
- Java (JDT-LS): add `runtimes` to `ls_specific_settings.java`, a list of extra JRE/JDK entries
(`name`, `path`, optional `default`/`sources`/`javadoc`) passed through to JDT-LS's
`java.configuration.runtimes`. Fixes silently broken JDK type resolution (`java.lang.Object`
and other JDK types reported as "cannot be resolved") for projects whose source/target level
exceeds the bundled JDK 21 JRE JDT-LS registers by default; configured runtimes extend rather
than replace that bundled default. #1478
- `gopls`: Fix `replace_symbol_body` corrupting single `type`/`var`/`const` declarations by
duplicating the leading keyword (e.g. `type Foo` becoming `type type Foo`). gopls reports the
symbol range of such declarations starting at the identifier rather than the keyword (unlike
`func` declarations); the range is now extended to include the keyword so the body and the
replacement range stay consistent.
- `typescript` / `typescript_vts`: No longer ignore directories named `coverage`. This was intended to skip
coverage-report output, but matched by bare dirname and so also hid legitimate source directories named
`coverage` (e.g. `src/routes/coverage/`) from symbol tools. Generated report dirs are already covered by
gitignore. Fixes #1523.
* Tools:
- Fix: `search_for_pattern` marked one line too many as matched whenever a match ended with a line
break, because the match's exclusive end index was mapped to a line number directly and therefore
resolved to the start of the following line. The line the match ends on is now determined correctly,
which also keeps `context_lines_after` aligned.
- `safe_delete`: Add heuristic to delete superfluous empty lines after a deletion
- `search_for_pattern`: on overflow, the shortening chain now emits each match's first line (full when
it fits, otherwise truncated with a trailing '...' and a note) before falling back to bare line
numbers, so agents can pick the right match without re-reading files. #1640
* Language Servers:
- PHP: treat `.phtml` files as PHP sources by default (all PHP language servers) #1710
- PHP/Intelephense: expose `file_filter` via `ls_specific_settings["php"]`, so that additional
extensions containing PHP sources (e.g. Drupal's `.module` / `.install` / `.inc` / `.theme`)
become visible to the symbol tools and are indexed by the language server #1710
- Fix: an LSP `ContentModified` (-32801) response was surfaced as a hard `SolidLSPException`
instead of being retried, as the spec expects for requests a client declared it will reissue.
`send_request` now retries such responses for methods declared via a server's
`retryOnContentModified` capability; rust-analyzer declares `textDocument/hover`, fixing the
flaky `test_find_symbol[rust_add_function]` on windows-latest. #1724
* JetBrains:
- Allow external files from dependencies (specified via references like "<ext:FileUtil.class|472e0a13>") to be
- read via `ReadFileTool`
- searched via `SearchForPatternTool`
- used in `JetBrainsFindDeclarationTool`
when using plugin version 2023.3.3+
* Dashboard:
- Fix: the "Last Execution" panel stayed on "Loading..." forever when there was no logged execution
(e.g. a fresh server / no tool run yet), because `loadLastExecution()` only rendered the panel when
the backend returned a non-null execution; the empty state is now rendered via the existing
`displayLastExecution(null)` path, mirroring the other panels #1713
* Dependencies:
- Bump `mcp` from 1.27.0 to 1.28.1
- Bump `anthropic` from 0.59.0 to 0.117.0
# v1.6.0 (2026-07-16)
* General:
- Speed up MCP startup when auto-creating projects at startup (e.g. using `--project-from-cwd`) by
determining the project's languages in a background thread #1683
- Fix: in `glob_to_regex` / `search_text(is_glob=True)`, a `?` wildcard matched two characters instead
of one (it emitted `..` rather than `.`); it now matches a single character, consistent with the
`?` semantics documented for `glob_match` and the `test_??.py` example in `search_text`'s docstring.
- Add notion of trusted projects via new global configuration setting `trusted_project_path_patterns`.
Current effects:
- `ls_specific_settings` defined in project configurations will only be applied for trusted projects
- `activation_command` (and `activation_command_timeout`) defined in project configurations will only
be executed for trusted projects: an optional shell command run in the project root before the
language backend initialises (e.g. to generate source files a language server needs to index).
Exit code is the primary completion signal; `activation_command_timeout` (default 180s) is a safety
backstop — on expiry the process is killed and activation continues. Failures and timeouts are
logged but do not abort activation.
- Fix: context or mode argument referencing a known name (e.g. `--context anitgravity`) could result in
incorrect file access if a corresponding local file existed (e.g. `./antigravity` binary);
file access is now guarded with path detection (file ending or path separator must be present)
- Adjust prompt generation mechanism to use newly introduced tool name mapping `tool_names`, allowing
prompts to directly use tool names that match the active language backend (and removing the need
for additional prompts that explain tool name differences)
- Improve quoting/escaping of arguments in shell executions on Windows (via `oslex` dependency)
- Fix: a registered project whose root directory was deleted while Serena was already running could break
`activate_project`/project lookup, raising `FileNotFoundError` in `RegisteredProject.matches_root_path`
- Update prompts/instructions: Serena instructions manual, modes (editing, interactive)
- Allow structured tool output to be configured on a per-context basis, disabling it for Claude Code
(which does not correctly unpack structured output) #1042
- Fix: Project-specific filtering of files for source files ignored the language backend.
The check is really only possible for LSP.
- Fix: File system permission errors during gitignore scanning were not caught #1624
- During project creation, language composition percentages are now computed relative to the total number
of recognised source files instead of all files, i.e. unrecognised files are ignored in the percentage
computation.
- Consider all LSP-compliant line endings ("\n", "\r\n" and "\r") in `TextUtils`, noting that
only "\n" appears in files read by `FileUtils.read_file` (Serena's default reading mechanism)
- Fix: Apply consistent line splitting across tools, uniformly applying the LSP splitting semantics;
affects `search_text` used by `search_for_pattern` tool (reported in #1684)
- Fix in `TextUtils` (used by editors): Deleting up to the end of the file, referencing the line one past
the end of the file (particularly for files with no newline at end of file) raised `InvalidTextLocationError`
instead of accepting the deletion (change in `TextUtils.delete_text_between_positions`,
which now accepts the end position similar to `insert_text_at_position`).
- Fix: glob pattern expansion in `expand_braces` did not terminate with empty or unbalanced braces #1690
* CLI:
- Fix `--project-from-cwd` hijacking git worktrees nested under a Serena project. `find_project_root`
now walks up in a single pass so the nearest project boundary wins (either a `.serena/project.yml`
or a `.git`, including worktree/submodule pointer files), instead of preferring an ancestor's
`.serena/project.yml` over a closer `.git`. This previously bound CLI agents (Claude Code, Codex,
Gemini) launched from inside a worktree to the parent repo, causing stale reads and misdirected edits.
- Fix: CLI flags on `start-mcp-server` could incorrectly be saved to the global configuration file if the
list of projects was modified (triggering a save of the configuration with transient overrides applied)
* Tools:
- New tool: `replace_in_files`
- `get_symbols_overview`, `jet_brains_get_symbols_overview`: Improved default for `depth` parameter
- Add tool parameter alias support, adding `name_path` as an alias for `name_path_pattern` in `find_symbol` tools
- Allow `query_project` tool to access read-only tools that are not enabled in the current configuration
- Make tool call errors surface explicitly as errors at the MCP protocol level
* Language Servers:
- Fix: `SolidLanguageServer.is_ignored_path` raised `FileNotFoundError` for paths that are not present
on disk, crashing symbol tools when a language server reported locations of generated files
(e.g. JDTLS reporting Lombok-generated classes under `target/classes`). Missing paths are now
classified by the usual ignore rules, and symbol locations resolving to non-existent files are skipped.
- Solidity: fix diagnostics intermittently coming back empty on slow or cold environments (macOS/Windows CI).
- Perl: expose `file_filter` and `ignore_dirs` via `ls_specific_settings["perl"]`, so projects with
non-standard extensions (e.g. `.cgi`, `.psgi`) can make those files visible to Perl::LanguageServer.
Configured extensions are also synced into the Perl source-file matcher (now a cached singleton),
keeping `find_symbol` / symbol indexing consistent with the LS; the matcher is reset on every
language server activation so one project's reconfiguration does not leak into the next.
`FilenameMatcher` gains `add_extensions` / `reset` methods. Defaults are unchanged. #1449
- C/C++ (clangd): improve support and documentation for Unreal Engine 5 projects.
- HLSL (`shader-language-server`): pass `--locked` to `cargo install` when building from source
on macOS (and in the manual-install instructions), honoring the crate's packaged `Cargo.lock`.
Without it, fresh dependency resolution pulled in shader-sense 1.4.0, which no longer compiles
against the pinned shader_language_server 1.3.1, breaking the macOS CI job.
- `typescript_vts`: Add `initialization_options` setting in `ls_specific_settings.typescript_vts`.
Enables Yarn PnP setups with `typescript.tsdk` pointing at the Yarn-generated SDK.
- TypeScript/VTS: disable automatic typing acquisition during initialization (no network
downloads at startup) and replace the fixed 2-second cross-file reference wait with
event-based `$/progress` indexing tracking (configurable `indexing_timeout`, default 30s)
- C#: minor fixes in Omnisharp and Roslyn that prevented startup on some systems #1617
- `SvelteLanguageServer`: Fix diagnostics requests for TypeScript/JavaScript files incorrectly being
processed by the Svelte LS instead of the TypeScript LS.
- `SvelteLanguageServer`: Fix document-symbol requests for TypeScript/JavaScript files returning empty
results in svelte-only mode (`languages: [svelte]`. #1552
- Svelte + TypeScript: make companion TypeScript server raise on readiness and indexing timeouts (instead
of silent "proceeding anyway"), so that partial indexing surfaces as a clear failure instead of flaky/wrong
cross-file results; add configurable timeouts (`server_ready_timeout`, `indexing_timeout`)
and overridable timeout hooks (base TS stays permissive). Svelte test fixture now uses `npm ci` +
committed `package-lock.json`.
- `JuliaLanguageServer`: Fix the stdio MCP server exiting right after `initialize` ("tools fetch failed")
when `julia` is enabled. #1577
- `Java`: invalidate JDTLS workspace cache when Java import settings change #1576
- `Java`: use `JAVA_HOME` for Gradle import when `use_system_java_home` is enabled and `gradle_java_home`
is unset. #1657
- `Java`: stop hard-ignoring directories named `target`/`build`/`bin`/`out`/`classes`/`dist`/`lib` in
`EclipseJDTLS`. These are all valid Java package identifiers, so ignoring them by name hid legitimate
source from the symbol tools even when they were not gitignored. Removed the hardcoded override; real
build output is already excluded via `.gitignore`. #1645
- Improve quoting of arguments in shell executions
- Add **LaTeX** support (experimental) via [texlab](https://github.com/latex-lsp/texlab).
- Add **QML** support via Qt's [`qmlls`](https://doc.qt.io/qt-6/qtqml-tool-qmlls.html) language
server (requires Qt 6 with `qmlls`/`qmlls6` on PATH). #1381
- PHP: add support for PHPantom as alternative to the already supported PHP LS #1554.
- Add new launch command customization options: `ls_args`, `ls_extra_args` and `ls_base_cmd`
- Add new configuration option `ls_workspace_folders` to allow indexed source folders to be specified
explicitly. In monorepos, this allows the set of indexed folders to be restricted to a subset of
the repository. #1627
- Rename configuration option `additional_workspace_folders` to `ls_additional_workspace_folders`
and support the option across all language servers (previously limited to TypeScript).
- `Pyright`: bump timeout for waiting for initial analysis from 5s to 60s.
* JetBrains:
- Add configuration option `jetbrains_launch_command`, allowing Serena to spawn IDE instances automatically
upon project activation
- Fix: `jet_brains_list_inspections` failed when only default parameters were used #1615
- Fix: `jet_brains_run_inspections` returned incorrectly transformed data (malforming snake-case conversion applied to all keys)
* Dashboard:
- Make list of trusted hosts configurable, fixing host validation introduced in v1.5.2 allowing only
@@ -28,6 +209,17 @@ Status of the `main` branch. Changes prior to the next official version change w
- Tray manager: Add NixOS-support for AppIndicator-based trays (e.g., most Wayland-trays) to the package in flake.nix.
- Fix: Wait for the subprocess that opens the browser window, preventing zombie processes #1488
* Hooks:
- Handle tool_input passed as string gracefully instead of failing (Copilot CLI sends strings).
* Memories:
- Make memory iteration follow symbolic links
- Fix: a memory name that was absolute (e.g. `/etc/cron.d/backdoor`) or contained empty path
segments could write outside the `memories` folder.
* Dependencies:
- Add dependency `oslex`
# v1.5.3 (2026-05-26)
Add meta-data for the GitHub MCP registry
CONTRIBUTING.md
+6
View File
@@ -18,6 +18,12 @@ Every PR should cover a single logical change or a set of closely related change
See the corresponding [memory](.serena/memories/adding_new_language_support_guide.md).
## Submitting Pull Requests
Before submitting a PR, be sure to document your relevant changes (i.e. new features, fixes) in `CHANGELOG.md`.
Use a concise style and add your change to the appropriate section
("Language Servers", "Tools", "JetBrains", "CLI", "Memories", "Dashboard", "Hooks", "General").
## Python Environment Setup
You can install a virtual environment with the required as follows
README.md
+1 -1
View File
@@ -110,7 +110,7 @@ Serena incorporates a powerful abstraction layer for the integration of language
The underlying language servers are typically open-source projects or at least freely available for use.
When using Serena's language server backend, we provide **support for over 40 programming languages**, including
Ada / SPARK, AL, Angular, Ansible, Bash, BSL, C#, C/C++, Clojure, Crystal, CUE, Dart, Elixir, Elm, Erlang, Fortran, F#, GDScript, GLSL, Go, Groovy, Haskell, Haxe, HLSL, HTML, Java, JavaScript, JSON, Julia, Kotlin, Lean 4, Lua, Luau, Markdown, MATLAB, mSL, Nix, OCaml, Perl, PHP, PowerShell, Python, R, Ruby, Rust, Scala, SCSS / Sass / CSS, Solidity, Svelte, Swift, TOML, TypeScript, WGSL, YAML, and Zig.
Ada / SPARK, AL, Angular, Ansible, Bash, BSL, C#, C/C++, Clojure, Crystal, CUE, Dart, Elixir, Elm, Erlang, Fortran, F#, GDScript, GLSL, Go, Groovy, Haskell, Haxe, HLSL, HTML, Java, JavaScript, JSON, Julia, Kotlin, LaTeX, Lean 4, Lua, Luau, Markdown, MATLAB, mSL, Nix, OCaml, Perl, PHP, PowerShell, Python, R, Ruby, Rust, Scala, SCSS / Sass / CSS, Solidity, Svelte, Swift, TOML, TypeScript, WGSL, YAML, and Zig.
### The Serena JetBrains Plugin
compose.yaml
+2 -2
View File
@@ -10,8 +10,8 @@ services:
dockerfile: Dockerfile
target: production
ports:
- "${SERENA_PORT:-9121}:9121" # MCP server port
- "${SERENA_DASHBOARD_PORT:-24282}:24282" # Dashboard port (default 0x5EDA = 24282)
- "127.0.0.1:${SERENA_PORT:-9121}:9121"
- "127.0.0.1:${SERENA_DASHBOARD_PORT:-24282}:24282"
environment:
- SERENA_DOCKER=1
command:
docs/01-about/020_programming-languages.md
+15 -2
View File
@@ -55,7 +55,8 @@ Some languages require additional installations or setup steps, as noted.
* **C/C++**
(by default, uses the clangd language server (language `cpp`) but we also support ccls (language `cpp_ccls`);
for best results, provide a `compile_commands.json` at the repository root;
see the [C/C++ Setup Guide](../03-special-guides/cpp_setup) for details.)
see the [C/C++ Setup Guide](../03-special-guides/cpp_setup) for details;
for Unreal Engine 5 projects, see the [Unreal Engine Setup Guide](../03-special-guides/unreal_engine_setup_guide_for_serena).)
* **Clojure**
* **Crystal**
(requires [Crystalline](https://github.com/elbywan/crystalline) language server to be installed and available on PATH;
@@ -97,6 +98,10 @@ Some languages require additional installations or setup steps, as noted.
* **Julia**
* **Kotlin**
(uses the pre-alpha [official kotlin LS](https://github.com/Kotlin/kotlin-lsp), some issues may appear)
* **LaTeX**
(experimental; must be explicitly enabled via language `latex`; uses [texlab](https://github.com/latex-lsp/texlab),
auto-downloaded as a SHA-256-verified prebuilt binary; supports `.tex`, `.bib`, `.sty`, and `.cls` files; texlab is
GPL-3.0 and runs as a separate downloaded process)
* **Lean 4**
(requires `lean` and `lake` installed via [elan](https://github.com/leanprover/elan); uses the built-in Lean 4 LSP;
the project must be a Lake project with `lake build` run before use)
@@ -117,8 +122,16 @@ Some languages require additional installations or setup steps, as noted.
(requires installation of Perl::LanguageServer)
* **PHP**
(by default, uses the Intelephense language server (language `php`), set `INTELEPHENSE_LICENSE_KEY` environment variable for premium features;
we also support [Phpactor](https://github.com/phpactor/phpactor) (language `php_phpactor`), which requires PHP 8.1+)
we also support [Phpactor](https://github.com/phpactor/phpactor) (language `php_phpactor`), which requires PHP 8.1+;
and the experimental [PHPantom](https://github.com/PHPantom-dev/phpantom_lsp) backend (language `php_phpantom`)
* **Python**
(by default, uses [Pyright](https://github.com/microsoft/pyright) (language `python`);
alternatives: [ty](https://github.com/astral-sh/ty) (language `python_ty`),
[pyrefly](https://github.com/facebook/pyrefly) (language `python_pyrefly`),
[Jedi](https://github.com/palotas/jedi-language-server) (language `python_jedi`);
ty and pyrefly require `uv`/`uvx` in PATH)
* **QML**
(requires Qt 6, provides `qmlls` or `qmlls6` on PATH; see the [Qt qmlls documentation](https://doc.qt.io/qt-6/qtqml-tool-qmlls.html))
* **R**
(requires installation of the `languageserver` R package)
* **Ruby**
docs/02-usage/020_running.md
+4 -2
View File
@@ -92,7 +92,9 @@ Some useful options include:
* `--project <path|name>`: specify the project to work on by name or path.
* `--project-from-cwd`: auto-detect the project from current working directory
(looking for a directory containing `.serena/project.yml` or `.git` in parent directories and activating the containing directory as the project root, if any).
(walking up the parent directories and activating the nearest one that contains either `.serena/project.yml`
or `.git`, if any). The nearest boundary wins, so a git worktree nested under another Serena project resolves
to the worktree itself rather than the ancestor project.
This option is intended for CLI-based agents like Claude Code, Gemini and Codex, which are typically started from within the project directory
and which do not change directories during their operation.
* `--language-backend JetBrains`: use the Serena JetBrains Plugin as the language backend (overriding the default backend configured in the central configuration)
@@ -239,7 +241,7 @@ docker run --rm -i --network host -v /path/to/your/projects:/workspaces/projects
This command mounts your projects into the container under `/workspaces/projects`, so when working with projects,
you need to refer to them using the respective path (e.g. `/workspaces/projects/my-project`).
Alternatively, you may use Docker compose with the `compose.yml` file provided in the repository.
Alternatively, you may use Docker compose. Adjust the file `compose.yml`, which is provided in the repository, according to your needs.
See our [advanced Docker usage](https://github.com/oraios/serena/blob/main/DOCKER.md) documentation for more detailed instructions, configuration options, and limitations.
:::{note}
docs/02-usage/030_clients.md
+91 -3
View File
@@ -397,7 +397,7 @@ Add the `serena` MCP server configuration
"command": "serena",
"args": [
"start-mcp-server",
"--context=claude-desktop"
"--context=desktop-app"
]
}
}
@@ -525,6 +525,93 @@ You will have to prompt Antigravity's agent to "Activate the current project usi
Unlike VSCode, Antigravity does not currently support including the working directory in the MCP configuration.
Also, the current client will be shown as `none` in Serena's dashboard (Antigravity currently does not fully support the MCP specifications). This is not a problem, all tools will work as expected.
## CodeBuddy
Serena provides native support for CodeBuddy, a CLI coding agent that shares a similar architecture with Claude Code.
To set up the Serena MCP server for CodeBuddy, simply run:
serena setup codebuddy
### Manual Setup
**Global Configuration**. To add the Serena MCP server for all your projects, use the user-level configuration of CodeBuddy and the `--project-from-cwd` flag:
```bash
codebuddy mcp add --scope user serena -- serena start-mcp-server --context codebuddy --project-from-cwd
```
**Project-Level Configuration**. To add the Serena MCP server for a single project only:
```bash
codebuddy mcp add serena -- serena start-mcp-server --context codebuddy --project "$(pwd)"
```
Confirm that CodeBuddy is connected to Serena by running the `/mcp` command and reconnecting if necessary.
### Hooks
CodeBuddy supports the same hook system as Claude Code. To set up hooks, add the following to your CodeBuddy settings file (`.codebuddy/settings.json` in your project directory, or `~/.codebuddy/settings.json` globally):
```json
{
"hooks": {
"PostToolUse": [
{
"matcher": "",
"hooks": [
{
"type": "command",
"command": "serena-hooks remind --client=codebuddy"
}
]
}
],
"Notification": [
{
"matcher": "",
"hooks": [
{
"type": "command",
"command": "serena-hooks auto-approve --client=codebuddy"
}
]
}
],
"SessionStart": [
{
"matcher": "",
"hooks": [
{
"type": "command",
"command": "serena-hooks activate --client=codebuddy"
}
]
}
],
"Stop": [
{
"matcher": "",
"hooks": [
{
"type": "command",
"command": "serena-hooks cleanup --client=codebuddy"
}
]
}
]
}
}
```
### Hook Descriptions
- **`remind`**: Remind the agent to use Serena's tools instead of built-in `grep` and `read` tools.
- **`activate`**: Prompt the agent to activate the project at session start and read Serena's instructions.
- **`cleanup`**: Clean up hook session data when the session ends.
- **`auto-approve`**: Auto-approve Serena tool calls whenever CodeBuddy is in a permissive
permission mode (`acceptEdits` or `auto`), so blanket approvals cover Serena's destructive
tools (e.g. `replace_symbol_body`, `rename_symbol`) instead of prompting on every call.
## Other Clients
For other clients, follow the [general instructions](#clients-general-instructions) above to set up Serena as an MCP server.
@@ -536,8 +623,9 @@ There are many terminal-based coding assistants that support MCP servers, such a
* [Gemini-CLI](https://github.com/google-gemini/gemini-cli),
* [Qwen3-Coder](https://github.com/QwenLM/Qwen3-Coder),
* [rovodev](https://community.atlassian.com/forums/Rovo-for-Software-Teams-Beta/Introducing-Rovo-Dev-CLI-AI-Powered-Development-in-your-terminal/ba-p/3043623),
* [OpenHands CLI](https://docs.all-hands.dev/usage/how-to/cli-mode) and
* [opencode](https://github.com/sst/opencode).
* [OpenHands CLI](https://docs.all-hands.dev/usage/how-to/cli-mode),
* [opencode](https://github.com/sst/opencode) and
* [CodeBuddy-Code](https://www.codebuddy.cn/cli/).
They generally benefit from the symbolic tools provided by Serena. You might want to customize some aspects of Serena
by writing your own context, modes or prompts to adjust it to the client's respective internal capabilities (and your general workflow).
docs/02-usage/040_workflow.md
+8 -34
View File
@@ -13,7 +13,7 @@ setting up a project with Serena typically involves the following steps:
4. **Working on coding tasks**: Using Serena to help you with actual coding tasks in the project
(project-creation-indexing)=
## Project Creation & Indexing
## Project Creation
Project creation is the process of defining fundamental project settings that are relevant to Serena's operation.
@@ -38,7 +38,7 @@ To explicitly create a project, use the following command while in the project d
* You can immediately index the project after creation with `--index`.
(project-config)=
#### Project Configuration
## Project Configuration
After creation, you can adjust the project settings in the generated `.serena/project.yml` file
within the project directory.
@@ -51,8 +51,9 @@ The file allows you to configure ...
* the encoding used in source files
* ignore rules
* write access
* [additional workspace folders](additional-workspace-folders) for cross-package reference support in monorepos
* the list of workspace folders to be processed by language servers (when using the LSP backend)
* an initial prompt that shall be passed to the LLM whenever the project is activated
* a shell command to run upon project activation (prior to language backend initialisation)
* the set of tools and modes to use for the project
* and some other settings.
@@ -60,44 +61,17 @@ For detailed information on the parameters and possible settings, see the
[template file](https://github.com/oraios/serena/blob/main/src/serena/resources/project.template.yml).
:::{note}
Many settings in project.yml *extend* or *override* settings in the global configuration file `serena_config.yml`.
Many settings in project.yml *extend* or *override* settings in [Serena's global configuration](global-config).
So use the project configuration specifically for aspects that apply only to the particular project.
:::
**Local Overrides**. The project.yml file is intended to be versioned together with the project.
**Local Overrides**. The `project.yml` file is intended to be versioned together with the project.
You can specify local overrides for the settings in a `project.local.yml` file in the same directory
(which, by default, is ignored by git).
Any keys defined therein will override the respective key in `project.yml`.
(additional-workspace-folders)=
#### Additional Workspace Folders (Cross-Package References)
In monorepos or multi-package setups, Serena's language server normally only sees symbols within the
project root. To enable cross-package references (e.g. `find_referencing_symbols` discovering usages
in sibling packages), configure `additional_workspace_folders` in your `project.yml`:
```yaml
additional_workspace_folders:
- ../shared-lib
- ../api-client
- /absolute/path/to/another-package
```
Paths can be absolute or relative to the project root. Each folder is registered as an LSP workspace
folder, and the language server will discover symbols and references across all listed packages.
**Currently supported for:** TypeScript. Other language servers will raise an error if this setting
is used with them. Support for additional languages can be added by implementing the
`_find_representative_source_file()` method in the respective language server class.
:::{note}
Each additional workspace folder adds startup time, as the language server needs to index the
additional projects. For large monorepos, consider listing only the packages you actively need
cross-references for.
:::
(indexing)=
### Indexing
## Indexing
:::{note}
Indexing is not a relevant operation when using the JetBrains plugin, as indexing is handled by the IDE.
@@ -109,7 +83,7 @@ that requires symbol information.
While in the project directory, run this command:
<serena> project index
serena project index
Indexing has to be called only once. During regular usage, Serena will automatically update the index whenever files change.
docs/02-usage/050_configuration.md
+129 -22
View File
@@ -7,7 +7,7 @@ You can disable tools, change Serena's fundamental instructions
(what we denote as the `system_prompt`), adjust the output of tools that just provide a prompt,
and even adjust tool descriptions.
Serena is configured in using a multi-layered approach:
Serena is configured using a multi-layered approach:
* **global configuration** (`serena_config.yml`, see below)
* **project configuration** (`project.yml`, see [Project Configuration](project-config))
@@ -27,10 +27,11 @@ Some of the configurable settings include:
this can also be [overridden per project](per-project-language-backend)
* UI settings affecting the [Serena Dashboard and GUI tool](060_dashboard.md)
* the set of tools to enable/disable by default
* the set of modes to use by default
* the set of [modes](modes) to use by default
* tool execution parameters (timeout, max. answer length)
* global ignore rules
* logging settings
* the set of trusted project paths
* advanced settings specific to individual language servers (see [below](ls-specific-settings))
The global configuration settings apply to all projects.
@@ -199,9 +200,12 @@ Most users will not need to adjust these settings.
:::
Under the key `ls_specific_settings` in `serena_config.yml`, you can you pass global per-language,
language server-specific configuration. You can use the same key in the project configuration files (`project.yml`
language server-specific configuration.
You can use the same key in the project configuration files (`project.yml`
and `project.local.yml` ) to override or extend the global settings for a specific project.
The settings are merged on top-level, meaning that project-level settings for a language will replace global settings for the same language.
The settings are merged on top-level, meaning that project-level settings for a language will replace global settings for the same language.
Note: Project-level settings are considered only for *trusted projects* (which are defined in the [global configuration](global-config)).
Structure:
@@ -212,31 +216,69 @@ ls_specific_settings:
```
(override-ls-path)=
#### Overriding the Language Server Path
#### Customizing the Language Server Launch Command
Most of Serena's language servers, particularly those that use a single core path for the language server (e.g. the main executable),
support overriding that path via the `ls_path` setting.
Therefore, if you have installed the language server yourself and want to use your installation
instead of Serena's managed installation, you can set the `ls_path` setting as follows:
Most of Serena's language servers construct the command that launches the language server process from
a *base command* or a *core dependency*.
For these language servers, the following settings can be used to customize the launch command:
| Setting | Description |
|--------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| `ls_path` (string) or `ls_base_cmd` (list) | overrides the path of the language server's core dependency (`ls_path`), e.g. its executable or a JAR file, or a base command for its execution (`ls_base_cmd`), e.g. `["npx", "-y", "/my/local/package"]`. Use this if you have installed the language server yourself and want Serena to use your installation instead of its managed installation. |
| `ls_args` (list) | overrides the internal command construction completely and simply adds `ls_args` to the base command |
| `ls_extra_args` (list) | a list of additional arguments to append to the launch command |
* If you set `ls_args`, the internal command construction (which may do more than to append arguments to a base command) is bypassed.
You can define the full launch command by providing both `ls_path`/`ls_base_cmd` and `ls_args`.
* If `ls_args` is not set, the internal command construction (which sets default arguments) is applied, and you can use `ls_path` or `ls_base_cmd` to override the path of the core dependency/the base command.
* `ls_extra_args` is always appended to the end of the launch command.
Example:
```yaml
ls_specific_settings:
<language>:
ls_path: "/path/to/language-server"
ls_extra_args: ["--log-level=debug"]
```
This is supported by all language servers deriving their dependency provider from `LanguageServerDependencyProviderSinglePath`,
and by some additional wrappers that explicitly expose `ls_path`.
These settings are supported by all language servers whose dependency provider derives from
`LanguageServerDependencyProviderBaseCommand`, and `ls_path` is additionally exposed by some implementations explicitly.
Common examples include: `ansible`, `bash`, `bsl`, `clojure`, `cpp`, `cpp_ccls`, `hlsl`, `html`, `kotlin`, `lean4`, `luau`, `markdown`, `php`,
`php_phpactor`, `python`, `rust`, `scss`, `solidity`, `systemverilog`, `toml`, `typescript`, and `yaml`.
Note: `angular` does **not** support `ls_path` — the Angular language server is part of a multi-process orchestration
(`ngserver` plus a companion TypeScript language server with the `@angular/language-service` plugin and an HTML
companion) where the dependency layout matters; use the version overrides documented in the Angular section below
to pin specific releases of the bundled stack.
If `ls_path` is set, Serena's managed download or install is bypassed for that language server.
In that case, any server-specific version or registry settings do not apply.
If a language server supports `ls_path`, setting it bypasses Serena's managed download or install for that server.
In that case, any server-specific version or registry settings only apply when `ls_path` is not set.
(override-init-options)=
#### Overriding Language Server Initialization Options
When Serena starts a language server, it sends a set of `initializationOptions` as part of the
Language Server Protocol `initialize` request. These options are constructed internally and are
tailored to each language server. In some cases, you may want to override or extend these options,
e.g. to enable a feature or to adjust a behavior that is specific to your setup.
Under the key `initializationOptions` within a language's `ls_specific_settings`, you can provide a
dictionary of options that is applied on top of the internally constructed `initializationOptions`.
The values are combined at the top level only: for each top-level key you define, your value
replaces the original value for that key exactly as given (there is no recursive/deep merge of
nested dictionaries). Internally constructed keys that you do not define are left unchanged.
* If Serena constructs `initializationOptions` for the language server, each top-level key you
provide replaces the internally constructed value for that same key, while all other internally
constructed keys are retained.
* If Serena does not construct any `initializationOptions` for the language server, your custom
options are used as-is.
Example:
```yaml
ls_specific_settings:
<language>:
initializationOptions:
someFeature:
enabled: true
```
#### AL
@@ -653,8 +695,9 @@ The following settings are supported for the Java language server:
| `maven_user_settings` | `~/.m2/settings.xml` | Path to Maven `settings.xml` |
| `gradle_user_home` | `~/.gradle` | Path to Gradle user home directory |
| `gradle_wrapper_enabled` | `false` | Use the project's Gradle wrapper (`gradlew`) instead of the bundled Gradle distribution. Enable this for projects with custom plugins or repositories. |
| `gradle_java_home` | `null` | Path to the JDK used by Gradle. When unset, Gradle uses the bundled JRE. |
| `use_system_java_home` | `false` | Use the system's `JAVA_HOME` environment variable for JDTLS itself. Enable this if your project requires a specific JDK vendor or version for Gradle's JDK checks. |
| `gradle_java_home` | `null` | Path to the JDK used by Gradle. When unset, Gradle uses `JAVA_HOME` if `use_system_java_home` is enabled and `JAVA_HOME` is set; otherwise it falls back to Serena's bundled JRE. |
| `use_system_java_home` | `false` | Use the system's `JAVA_HOME` environment variable for JDTLS itself and, when `gradle_java_home` is unset, Gradle import. Enable this if your project requires a specific JDK vendor or version for Gradle's JDK checks. |
| `runtimes` | `[]` | Extra JRE/JDK entries registered with JDT-LS via `java.configuration.runtimes`. Use this when a project's source/target level exceeds the JDK JDT-LS itself runs on (currently JDK 21 in default vscode-java VSIX mode). Each entry is a mapping with required `name` (e.g. `JavaSE-25`, matching the `JavaSE-NN` container the build tool requests) and `path` (JDK/JRE home directory; must exist), plus optional `default`, `sources`, and `javadoc` (passed through to JDT-LS). Entries extend rather than replace the bundled `JavaSE-21` runtime; an entry that reuses the `JavaSE-21` name overrides the bundled one. Changing this setting invalidates the JDTLS workspace hash so a fresh import is performed. |
| `gradle_version` | `8.14.2` | (vscode-java mode only) Override the Gradle distribution version Serena downloads by default. |
| `vscode_java_version` | `1.54.0-923` | (vscode-java mode only) Override the bundled `vscode-java` runtime bundle version Serena downloads by default. |
| `intellicode_version` | `1.2.30` | (vscode-java mode only) Override the IntelliCode VSIX version Serena downloads by default. |
@@ -674,6 +717,10 @@ Notes:
- In upstream-jdtls mode the `gradle_version`, `vscode_java_version`, `intellicode_version`,
`intellicode_xmx`, `intellicode_xms` settings are silently ignored — they only apply to the
vscode-java VSIX mode.
- Without `runtimes`, JDT-LS only knows about the bundled `JavaSE-21` JRE. Projects that request a newer
container (e.g. `sourceCompatibility = JavaVersion.VERSION_25`) then fail to resolve JDK types such as
`java.lang.Object`. Register the matching installed JDK via `runtimes` instead of symlinking over Serena's
bundled JRE directory.
Example: upstream-jdtls mode (offline / corporate network):
@@ -694,6 +741,19 @@ ls_specific_settings:
use_system_java_home: true
```
Example: register an additional JDK for a project targeting a newer Java version:
```yaml
ls_specific_settings:
java:
runtimes:
- name: JavaSE-21
path: /usr/lib/jvm/java-21-openjdk
- name: JavaSE-25
path: /home/user/Java/jdk25
default: true
```
#### Kotlin
Serena uses [JetBrains' Kotlin Language Server](https://github.com/Kotlin/kotlin-lsp) for Kotlin support.
@@ -829,6 +889,34 @@ Notes:
- Use the FPC compiler driver (`fpc`/`fpc.exe`), not backend compilers like `ppc386.exe`.
- These settings are passed as environment variables to the pasls process.
#### Perl
Serena uses [Perl::LanguageServer](https://metacpan.org/pod/Perl::LanguageServer) for Perl support. Install Perl and the server with `cpanm Perl::LanguageServer`; Linux and macOS only (the server does not run on Windows).
Perl::LanguageServer only indexes files whose extension is in its `perl.fileFilter` and skips directories listed in `perl.ignoreDirs`. Both are exposed below so projects with non-standard extensions (e.g. `.cgi` / `.psgi` web handlers) can make those files visible (#1449).
**Configuration:**
Configure the language server via `ls_specific_settings.perl` in `serena_config.yml`:
| Setting | Default | Description |
| -------------- | ------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------- |
| `file_filter` | `[".pm", ".pl", ".t"]` | File extensions (with leading dot) that Perl::LanguageServer should index, e.g. `[".pm", ".pl", ".t", ".cgi"]`. |
| `ignore_dirs` | `[".git", ".svn", "blib", "local", ".carton", "vendor", "_build", "cover_db"]` | Directory names Perl::LanguageServer should skip when indexing. |
Example configuration:
```yaml
ls_specific_settings:
perl:
file_filter: [".pm", ".pl", ".t", ".cgi", ".psgi"]
ignore_dirs: [".git", "blib", "local", "vendor", "cover_db"]
```
Notes:
- Extensions added via `file_filter` are also synced into Serena's Perl source-file matcher, so `find_symbol` and symbol indexing treat the same files as the language server. Defaults are unchanged when these keys are omitted.
- The matcher is reset on every language server activation, so one project's `file_filter` does not leak into another.
#### PHP (`Intelephense`)
Serena uses Intelephense for the `php` language key.
@@ -843,6 +931,19 @@ Supported settings:
| `ignore_vendor` | `true` | Ignore directories named `vendor` while indexing the project. |
| `maxFileSize` | unset | Forwarded as `intelephense.files.maxSize` in `initializationOptions`. |
| `maxMemory` | unset | Forwarded as `intelephense.maxMemory` in `initializationOptions`. |
| `file_filter` | unset | Additional file extensions (with leading dot) to treat as PHP sources, e.g. `[".module", ".install"]`; added to the defaults `.php` / `.phtml` (#1710). |
Example configuration making Drupal source files visible to the symbol tools:
```yaml
ls_specific_settings:
php:
file_filter: [".module", ".install", ".inc", ".theme", ".profile", ".engine"]
```
Notes:
- Extensions added via `file_filter` are synced into Serena's PHP source-file matcher and pushed to Intelephense as `intelephense.files.associations` globs at startup, so `find_symbol` and the language server treat the same files as PHP sources.
- The matcher is reset on every language server activation, so one project's `file_filter` does not leak into another.
#### PHP (`Phpactor`)
@@ -988,9 +1089,8 @@ Supported settings:
| `typescript_version` | `5.9.3` | Override the bundled `typescript` npm package version Serena installs when `ls_path` is not set. |
| `typescript_language_server_version` | `5.1.3` | Override the bundled `typescript-language-server` npm package version Serena installs when `ls_path` is not set. |
| `npm_registry` | `null` | Override the npm registry Serena uses for the managed install. |
TypeScript supports [additional workspace folders](additional-workspace-folders) for cross-package
reference discovery. Configure `additional_workspace_folders` in `project.yml` to enable this feature.
| `indexing_timeout` | `30.0` | Timeout in seconds for waiting on tsserver's `$/progress` project-indexing signal (both at startup and before the first cross-file reference query). If indexing does not complete within this window, Serena logs a warning and proceeds anyway. Increase it for very large projects. |
| `server_ready_timeout` | `10.0` | Timeout in seconds for waiting on the server-ready signal after initialization. If the signal does not arrive within this window, Serena logs a message and proceeds anyway. |
#### Svelte
@@ -1008,8 +1108,15 @@ Supported settings:
| `typescript_language_server_version` | `5.1.3` (falls back to `ls_specific_settings.typescript.typescript_language_server_version`) | Override the `typescript-language-server` npm package version for the companion server. |
| `typescript_svelte_plugin_version` | `0.3.52` | Override the `typescript-svelte-plugin` npm package version used for `.svelte`-aware TS resolution. |
| `npm_registry` | `null` | Override the npm registry Serena uses for all managed installs. |
| `indexing_timeout` | `120.0` (falls back to `ls_specific_settings.typescript.indexing_timeout`) | Timeout in seconds for the companion TS server to finish indexing `.svelte` files. On timeout, startup fails with a diagnostic indexing-state summary instead of serving cross-file results from a partially indexed program. |
| `initialization_options_configuration` | `{}` | Deep-merge overrides for any of the ten plugin configuration sections (`svelte`, `prettier`, `emmet`, `typescript`, `javascript`, `js/ts`, `css`, `less`, `scss`, `html`). |
Unlike the plain TypeScript server, the companion is strict about readiness: it raises on server-ready and
indexing timeouts instead of proceeding with a cold or partially indexed program (which would silently degrade
cross-file renames and references). The companion reads `server_ready_timeout` and `indexing_timeout` from
`ls_specific_settings.typescript` with raised defaults (30s and 120s respectively); `ls_specific_settings.svelte.indexing_timeout`
takes precedence for the `.svelte`-file indexing wait.
All four packages are tracked via a version file; changing any version setting triggers a clean reinstall.
#### TypeScript via `vtsls`
docs/02-usage/999_additional-usage.md
+2
View File
@@ -14,3 +14,5 @@ having persisted the plan in a memory or dedicated file.
[git-worktree](https://git-scm.com/docs/git-worktree) can be an excellent way to parallelize your work. More on this in [Anthropic: Run parallel Claude Code sessions with Git worktrees](https://docs.claude.com/en/docs/claude-code/common-workflows#run-parallel-claude-code-sessions-with-git-worktrees).
Be sure to add the `.serena` folder to version control, such that your project-specific settings and memories are available across worktrees.
When you launch a CLI agent from inside a worktree using `--project-from-cwd`, Serena activates the worktree itself, even if the worktree lives under another Serena project (for example `<repo>/.claude/worktrees/<name>`, where Claude Code creates them natively). The nearest project boundary wins: the worktree's own `.git` pointer file takes precedence over an ancestor's `.serena/project.yml`, so file operations always resolve against the correct working tree.
docs/03-special-guides/cpp_setup.md
+5
View File
@@ -100,3 +100,8 @@ the language server is restarted.
- Clangd official documentation: https://clangd.llvm.org/
- Clangd project setup: https://clangd.llvm.org/installation#project-setup
- CCLS repository: https://github.com/MaskRay/ccls
## Unreal Engine projects
For Unreal Engine 5 projects (reflection macros, UnrealBuildTool), see the
[Unreal Engine Setup Guide](unreal_engine_setup_guide_for_serena.md).
docs/03-special-guides/unreal_engine_setup_guide_for_serena.md
+157
View File
@@ -0,0 +1,157 @@
# Unreal Engine Setup Guide
This guide explains how to prepare an Unreal Engine 5 C++ project so that Serena's
clangd-based C/C++ support can provide full code intelligence: symbol search,
cross-file references, and symbol-level editing in your hand-written sources.
UE game code uses a macro-based reflection layer (`UCLASS`, `UFUNCTION`, `UPROPERTY`,
`GENERATED_BODY`) and engine types (`TArray`, `TMap`). clangd handles all of this,
provided it receives the compiler flags for your project via a `compile_commands.json`
at the project root. Unreal's build system (UnrealBuildTool) does not produce this
file by default; this guide shows how to obtain it.
---
## Prerequisites
- An Unreal Engine 5 C++ project that has been **built at least once** (the build
generates the `*.generated.h` headers that your sources include).
- No additional language server: Serena downloads clangd automatically.
- clangd never compiles your code. The compilation database is only a list of flags.
---
## Getting a compilation database
If clangd starts in a project that has a `.uproject` file but no usable
`compile_commands.json`, Serena fails the language server startup with an error that
inlines the command below, because clangd cannot resolve engine headers or reflection
macros without the database. Pick one of the following routes to create it.
None of these routes change how you build. You still compile with MSVC. A clang toolchain,
where a route uses one, only generates the database that clangd reads.
### Route 1 (recommended): UnrealBuildTool's clang database
<Engine>\Binaries\DotNET\UnrealBuildTool\UnrealBuildTool.exe -mode=GenerateClangDatabase -project="<YourProject>.uproject" <YourProject>Editor Win64 Development -OutputDir="<YourProject's directory>"
This emits clang-native commands. Each entry carries the system-include paths, so clangd
resolves system and engine headers without guessing.
It needs a clang toolchain. Install one from the Visual Studio Installer:
**Modify > Individual Components > "C++ Clang tools for Windows"**. This is not a separate
LLVM download. UnrealBuildTool auto-detects it at `VC\Tools\Llvm\x64`.
- UnrealBuildTool searches for clang in this order: `C:\Program Files\LLVM`, then the
`LLVM_PATH` environment variable, then the VS-bundled `VC\Tools\Llvm\x64`, then AutoSDK.
A standalone LLVM install wins if present. Remove it to fall back to the version-matched VS one.
- UE 5.7 expects clang in roughly the 18.1.8 to 20.1.8 range, and the VS component is
matched. A newer standalone (e.g. 22.x) still generates the database but prints
`Clang compiler version ... is not a preferred version`.
- `-OutputDir` is required. Without it the file lands in the engine root.
- Build the editor target once before generating, so the `*.generated.h` headers exist.
After a build you can add `-NoExecCodeGenActions` to skip redundant codegen.
### Route 2 (no clang toolchain): MSVC database
If you cannot add the clang component, append `-Compiler=VisualStudio2022`:
<Engine>\Binaries\DotNET\UnrealBuildTool\UnrealBuildTool.exe -mode=GenerateClangDatabase -project="<YourProject>.uproject" <YourProject>Editor Win64 Development -Compiler=VisualStudio2022 -OutputDir="<YourProject's directory>"
This produces an MSVC `cl.exe` database and needs no extra toolchain. The entries omit
system-include paths, which MSVC reads from the `INCLUDE` environment variable rather than
the database, so clangd may not resolve standard headers like `<vector>` and logs
`Failed to compile ... index may be incomplete` per file. Symbol tools still work, since
clangd indexes through those errors. To stop them from truncating symbol trees, add a
`.clangd` at the project root (see Troubleshooting):
CompileFlags:
Add: [-D_ALLOW_COMPILER_AND_STL_VERSION_MISMATCH, -ferror-limit=200]
### Route 3 (situational): VSCode project files
UnrealBuildTool's VSCode generator emits per-project compile commands:
<Engine>\Build\BatchFiles\Build.bat -projectfiles -project="<YourProject>.uproject" -game -VSCode
On an installed engine (Launcher/Fab) with a project outside the engine tree, this writes
an empty array. UnrealBuildTool treats the project as foreign and emits no commands. Verify
the output is a non-empty JSON array before relying on it:
<YourProject>\.vscode\compileCommands_Default.json
If it contains entries, copy or symlink it to the project root as `compile_commands.json`.
If it is `[]`, use Route 1 or Route 2. The generator works reliably only for source-built
or in-tree engines, or when refreshed from inside the editor via
**Tools > Refresh Visual Studio Code Project**.
### Notes for Rider users
Building the project in Rider never produces `compile_commands.json`. `-VSCode` is a
project-file generator, not an editor mode, and the `.vscode/` directory it creates is
safe to delete afterwards. Use Route 1.
### When to regenerate
Regenerate only when you add a module or change a `*.Build.cs` (new files or compiler
flags). clangd watches `compile_commands.json` and reloads automatically, so routine edits
need no regeneration. If Serena failed to start because the database was missing or empty,
reconnect or restart the MCP after creating it. Serena reads the database only at startup.
---
## Build artifact directories
Generated reflection code (`*.gen.cpp`, `*.generated.h`) legitimately references your
functions, so symbol results could otherwise include hits inside `Intermediate/`. When a
`.uproject` file is present at the project root, the clangd and ccls language servers skip
UE's build and cache directories (`Binaries`, `DerivedDataCache`, `Intermediate`, `Saved`)
during indexing. This is automatic and needs no configuration.
To exclude further paths, add them to `ignored_paths` in your project's
`.serena/project.yml`.
---
## Known behavior
- **`GENERATED_BODY()` and `__LINE__`:** the macro expands using its line number.
After editing lines above it, clangd may report stale-macro diagnostics until the
next build regenerates headers. Symbol operations keep working, since clangd is
designed to operate on code with errors.
- **First index:** large projects take a few minutes to index once; afterwards
results are incremental. The index cache is kept under `.serena/.cache` inside
the project.
- **New `UFUNCTION`/`UCLASS` declarations** need a build before their generated
headers exist.
- **Symbol searches on large projects:** prefer passing `relative_path` to
`find_symbol`. An unscoped search visits every translation unit, and UE
files are expensive to parse because each pulls in large engine headers.
- **clangd index logs (MSVC database):** an `Indexed ... (N symbols)` line is a success.
A following `Failed to compile ... index may be incomplete` means clangd hit errors
while parsing that file but indexed it anyway. Raise `-ferror-limit` via `.clangd` if
symbol trees look truncated (see Troubleshooting).
---
## Troubleshooting
Extra flags are easiest to add via a `.clangd` file at the project root, e.g.:
CompileFlags:
Add: [-D_ALLOW_COMPILER_AND_STL_VERSION_MISMATCH, -ferror-limit=200]
- **`STL1000: Unexpected compiler version` errors:** recent MSVC STL headers
assert a minimum Clang version that may be newer than Serena's bundled clangd.
Defining `_ALLOW_COMPILER_AND_STL_VERSION_MISMATCH` (see above) silences the
check; clangd only parses, so the mismatch is harmless.
- **Truncated symbol trees / symbols missing below a certain line:** clangd
aborts a file's parse after ~20 errors by default, which discards everything
declared after that point. Raising the limit with `-ferror-limit=200` keeps
the symbol tree intact even when diagnostics are noisy (common right after
edits, before the next UE build regenerates headers).
- **Stale results after changing the compilation database:** clangd's index
shards in `.serena/.cache` were built with the old flags. Delete that cache
directory and let the project re-index.
---
## Verifying the setup
After activating the project in Serena, a symbol overview of any `UCLASS` header
should list the class with its `UFUNCTION` methods and `UPROPERTY` fields, and
references to a method should resolve to your `Source/` files only.
news/20260716.html
+45
View File
@@ -0,0 +1,45 @@
<div class="news-item">
<h3>Serena v1.6.0</h3>
<p class="date">July 16, 2026</p>
<p>
A lot has happened since the last news item for v1.3.0. Here are the highlights:
</p>
<p>
<b>Extended Memory System.</b>
Memories can now reference each other via the <code>mem:&lt;name&gt;</code> convention (renames propagate automatically),
and a new <code>serena memories</code> CLI command group supports listing, reading, writing and integrity checking.
See the <a target="_blank" href="https://oraios.github.io/serena/02-usage/045_memories.html">memories documentation</a> for details.
</p>
<p>
<b>Trusted Projects &amp; Activation Commands.</b>
The new global setting <code>trusted_project_path_patterns</code> introduces the notion of trusted projects,
which may define an <code>activation_command</code> that is run before the language backend initialises
(e.g. to generate sources a language server needs) as well as language server-specific settings.
</p>
<p>
<b>Expanded Language Support.</b>
New languages for the LSP backend: GDScript (Godot Engine), CUE, QML, and LaTeX (experimental);
PHPantom was added as an alternative PHP language server.
</p>
<p>
<b>New Tools &amp; Configurability.</b>
The new <code>replace_in_files</code> tool applies pattern replacements across multiple files in one call.
Language server launching and indexing became more configurable,
and the JetBrains backend can now auto-launch IDE instances upon project activation
via <code>jetbrains_launch_command</code>.
</p>
<p>
<b>Robustness &amp; Fixes.</b>
A multitude of fixes and improvements across the board, including many language servers
(Java, TypeScript, Svelte, C#, Julia, Perl, and more), the dashboard,
the CLI (git worktree handling), and general token efficiency.
</p>
<p>
See our <a target="_blank" href="https://github.com/oraios/serena/blob/main/CHANGELOG.md">change log</a> for the full list of changes.
</p>
<p>
<b>Update Now.</b>
If you are using the uv tool installation of Serena, upgrade as follows:<br>
uv tool upgrade serena-agent
</p>
</div>
news/20260718.html
+19
View File
@@ -0,0 +1,19 @@
<div class="news-item">
<h3>Trusted Projects</h3>
<p class="date">July 18, 2026</p>
<p>
Serena v1.6.0 introduced the notion of <i>trusted projects</i>.<br>
In particular, only trusted projects are allowed to run activation commands and configure language server-specific
settings at the project level.
</p>
<p>
To not break existing workflows, existing Serena configurations were migrated to trust all projects
by default (whereas new installations will have no trusted projects by default).
</p>
<p>
<b>Configure trusted projects!</b>
If you sometimes activate projects from untrusted sources, we highly recommend that you adjust
the <code>trusted_project_path_patterns</code> setting in
<a href="https://oraios.github.io/serena/02-usage/050_configuration.html#global-configuration" target="_blank">Serena's global configuration</a>.
</p>
</div>
pyproject.toml
+62 -39
View File
@@ -4,7 +4,7 @@ requires = ["hatchling"]
[project]
name = "serena-agent"
version = "1.5.4.dev0"
version = "1.6.1.dev0"
description = "A powerful MCP toolkit for coding, providing semantic retrieval and editing capabilities - the IDE for your agent"
authors = [{ name = "Oraios AI", email = "info@oraios-ai.de" }]
readme = "README.md"
@@ -20,7 +20,7 @@ dependencies = [
"requests==2.33.0",
"overrides==7.7.0",
"python-dotenv==1.2.2",
"mcp==1.27.0",
"mcp==1.28.1",
"flask==3.1.3", # bumped from 3.1.1 for CVE fix (also fixes werkzeug alert)
"sensai-utils==1.5.0",
"pydantic==2.12.5",
@@ -35,7 +35,7 @@ dependencies = [
"joblib==1.5.1",
"tqdm==4.67.1",
"tiktoken==0.12.0",
"anthropic==0.59.0",
"anthropic==0.117.0",
"beautifulsoup4==4.14.2",
"pywebview==6.2",
"pystray==0.19.5",
@@ -45,12 +45,13 @@ dependencies = [
# Exact pins because uvx installs from git, ignoring the lock file.
"urllib3==2.7.0",
"werkzeug==3.1.7",
"starlette==1.0.0",
"python-multipart==0.0.27",
"starlette==1.3.1",
"python-multipart==0.0.31",
"filelock==3.25.2",
"cryptography==46.0.7",
"cryptography==48.0.1",
"regex==2026.2.28",
"pythonnet==3.1.0 ; sys_platform == 'win32'",
"oslex==2.0.0",
]
@@ -72,7 +73,7 @@ text = "MIT"
dev = [
"jinja2==3.1.6",
# In version 1.0.4 we get a NoneType error related to some config conversion (yml_analytics is None and should be a list)
"mypy==1.17.0",
"ty==0.0.24",
"poethepoet==0.36.0",
"pytest==8.4.1",
"pytest-xdist==3.8.0",
@@ -91,12 +92,12 @@ dev = [
"pytest-timeout==2.4.0",
# Transitive dev deps pinned for security (dependabot alerts).
# Exact pins because uvx installs from git, ignoring the lock file.
"tornado==6.5.5",
"tornado==6.5.7",
"wheel==0.46.3",
"pyasn1==0.6.3",
"PyJWT==2.12.0",
]
agno = ["agno==2.5.10", "sqlalchemy==2.0.41"] # agno bumped for session state overwrite CVE
agno = ["agno==2.6.6", "sqlalchemy==2.0.41"] # agno bumped for session state overwrite CVE
google = ["google-genai==1.27.0"]
[project.urls]
@@ -108,29 +109,47 @@ packages = ["src/serena", "src/interprompt", "src/solidlsp"]
[tool.doc8]
max-line-length = 1000
[tool.mypy]
allow_redefinition = true
check_untyped_defs = true
disallow_incomplete_defs = true
disallow_untyped_defs = true
ignore_missing_imports = true
no_implicit_optional = true
pretty = true
show_error_codes = true
show_error_context = true
show_traceback = true
strict_equality = true
strict_optional = true
warn_no_return = true
warn_redundant_casts = true
warn_unreachable = true
warn_unused_configs = true
warn_unused_ignores = false
exclude = "^build/|^docs/|^test/resources/"
[tool.ty.environment]
python-version = "3.11"
# Analyze for all platforms rather than defaulting to the OS ty happens to run on. This keeps the
# check deterministic across the CI matrix (Linux/Windows/macOS) and lets platform-conditional stdlib
# members (e.g. subprocess.CREATE_NO_WINDOW, ctypes.windll, pwd) resolve without per-OS type-ignores.
python-platform = "all"
[[tool.mypy.overrides]]
module = "test.*"
disallow_untyped_defs = false # untyped defs are OK in tests
[tool.ty.rules]
# Mirror mypy's ignore_missing_imports=true: optional extras (e.g. agno, google-genai) and
# platform-specific modules (e.g. AppKit on macOS) are not installed in the default dev environment,
# so we do not want unresolvable imports to fail the type check.
unresolved-import = "ignore"
possibly-missing-submodule = "ignore"
[tool.ty.src]
# NOTE: test/resources is intentionally NOT excluded here. This config is also discovered by the ty
# language server (used by the PYTHON_TY diagnostics provider) when it analyzes the sample fixture
# repos under test/resources; excluding them here would suppress the diagnostics those tests rely on.
# test/resources is instead excluded from the `ty check test` CLI task via its --exclude flag.
exclude = ["build/", "docs/"]
[[tool.ty.overrides]]
# Test code is heavily dynamic (pytest fixtures, MagicMock, intentionally loose Optionals). ty models
# pytest's fail/skip helpers and MagicMock far more strictly than mypy did (mypy inferred `Any` for
# these), which produces hundreds of diagnostics that are not real bugs. We therefore relax these
# specific type rules for tests only, while still catching genuinely broken test code (undefined
# names, syntax errors, unused type-ignore comments, etc.). The sample fixture repos under
# test/resources are excluded so the ty language server still reports real diagnostics on them.
include = ["test/**"]
exclude = ["test/resources/"]
[tool.ty.overrides.rules]
invalid-argument-type = "ignore"
invalid-key = "ignore"
invalid-return-type = "ignore"
no-matching-overload = "ignore"
not-subscriptable = "ignore"
parameter-already-assigned = "ignore"
too-many-positional-arguments = "ignore"
unresolved-attribute = "ignore"
unsupported-operator = "ignore"
[tool.poe.env]
PYDEVD_DISABLE_FILE_VALIDATION = "1"
@@ -143,9 +162,7 @@ PYDEVD_DISABLE_FILE_VALIDATION = "1"
type = "simple"
[tool.poe.tasks]
# Uses PYTEST_MARKERS env var for default markers
# For custom markers, one can either adjust the env var or just use -m option in the command line,
# as the second -m option will override the first one.
# For custom markers, use poe test -m ...
test = "pytest test -vv"
_ruff_check = "ruff check src scripts test"
_ruff_format_check = "ruff format --check src scripts test"
@@ -153,9 +170,10 @@ _ruff_fix = "ruff check --fix src scripts test"
_ruff_format = "ruff format src scripts test"
lint = ["_ruff_format_check", "_ruff_check"]
format = ["_ruff_fix", "_ruff_format"]
_mypy_core = "mypy src/serena src/solidlsp"
_mypy_test = "mypy --disable-error-code no-untyped-def test"
type-check = ["_mypy_core", "_mypy_test"]
_ty_core = "ty check src/serena src/solidlsp"
_ty_test = "ty check test --exclude test/resources"
type-check = ["_ty_core", "_ty_test"]
ty = ["type-check"]
# docs
_autogen_docs = "python docs/autogen_docs.py"
_sphinx_build = "sphinx-build -b html docs docs/_build -W --keep-going"
@@ -283,6 +301,7 @@ ignore = [
"UP042", # wants str,Enum -> StrEnum (breaking change)
"PLW0108", # unnecessary lambda (style preference)
"PLC0207", # split vs rsplit optimization (style preference)
"RSE102", # wants to force use of raise with class instead of instance (style preference)
]
unfixable = ["F841", "F601", "F602", "B018"]
extend-fixable = ["F401", "B905", "W291"]
@@ -335,6 +354,7 @@ markers = [
"fsharp: language server running for F#",
"rego: language server running for Rego",
"markdown: language server running for Markdown",
"latex: language server running for LaTeX",
"julia: Julia language server tests",
"fortran: language server running for Fortran",
"haskell: Haskell language server tests",
@@ -358,11 +378,14 @@ markers = [
"scss: language server running for SCSS / Sass / CSS (some-sass-language-server)",
"angular: language server running for Angular (uses @angular/language-server + @angular/language-service tsserver plugin)",
"ada: language server running for Ada / SPARK (uses AdaCore Ada Language Server)",
"qml: language server running for QML (uses Qt's qmlls)",
]
[tool.codespell]
# Ref: https://github.com/codespell-project/codespell#using-a-config-file
skip = '.git*,*.svg,*.lock,*.min.*,*test_memories_manager.py'
# We skip third-party test fixture repositories entirely (they contain
# external package lockfiles, minified code, and arbitrary dependency strings).
skip = '.git*,*.svg,*.lock,*.min.*,*test_memories_manager.py,./test/resources/repos'
check-hidden = true
ignore-regex = '\.\w+'
ignore-words-list = 'paket,EDN,als'
ignore-words-list = 'paket,EDN,als,ue'
src/serena/__init__.py
+1 -1
View File
@@ -1,4 +1,4 @@
__version__ = "1.5.4.dev0"
__version__ = "1.6.1.dev0"
import logging
src/serena/agent.py
+146 -57
View File
@@ -63,6 +63,7 @@ from serena.util.inspection import iter_subclasses
from serena.util.logging import MemoryLogHandler
from solidlsp.ls_config import Language
from solidlsp.util import subprocess_util
from solidlsp.util.subprocess_util import terminate_process_tree_with_kill_fallback
if TYPE_CHECKING:
from serena.gui_log_viewer import GuiLogViewer
@@ -210,7 +211,12 @@ class ToolSet:
class ActiveModes:
_mode_instances: dict[str, SerenaAgentMode] = {}
def __init__(self) -> None:
def __init__(self, background_base_modes: Sequence[SerenaAgentMode] | None = None) -> None:
"""
:param background_base_modes: base modes that are always active in the background and will not be removed
by applications of mode selection definitions via method `apply`
"""
self._background_base_modes = background_base_modes or []
self._configured_base_modes: Sequence[str] | None = None
self._configured_default_modes: Sequence[str] | None = None
self._added_modes: set[str] = set()
@@ -220,7 +226,7 @@ class ActiveModes:
"""
self._active_mode_names: Sequence[str] = []
"""
the full list of active mode names
the full list of active mode names (not including background base modes)
"""
def apply(self, mode_selection: ModeSelectionDefinition) -> None:
@@ -245,6 +251,9 @@ class ActiveModes:
self._active_mode_names = sorted(set(self._configured_base_modes or []) | self._dynamically_activated_mode_names)
def get_mode_names(self) -> Sequence[str]:
"""
:return: the ordered list of active mode names (not including the non-user-facing background base modes)
"""
return self._active_mode_names
@classmethod
@@ -253,14 +262,22 @@ class ActiveModes:
cls._mode_instances[mode_name] = SerenaAgentMode.load(mode_name)
return cls._mode_instances[mode_name]
def get_modes(self) -> Sequence[SerenaAgentMode]:
return [self.get_mode_instance(mode_name) for mode_name in self._active_mode_names]
def get_modes(self, include_background_base_modes: bool = False) -> Sequence[SerenaAgentMode]:
result: list[SerenaAgentMode] = []
if include_background_base_modes:
result.extend(self._background_base_modes)
result.extend([self.get_mode_instance(mode_name) for mode_name in self._active_mode_names])
return result
def get_dynamically_activated_modes(self) -> Sequence[SerenaAgentMode]:
return [self.get_mode_instance(mode_name) for mode_name in self._dynamically_activated_mode_names]
def get_base_modes(self) -> Sequence[SerenaAgentMode]:
return [self.get_mode_instance(mode_name) for mode_name in self._configured_base_modes or []]
def get_base_modes(self, include_background_base_modes: bool = False) -> Sequence[SerenaAgentMode]:
result: list[SerenaAgentMode] = []
if include_background_base_modes:
result.extend(self._background_base_modes)
result.extend([self.get_mode_instance(mode_name) for mode_name in self._configured_base_modes or []])
return result
class ProjectPromptProvisionStatus:
@@ -555,10 +572,6 @@ class SerenaAgent:
# propagate configuration to other components
self.serena_config.propagate_settings()
# initialise active modes (baseline modes prior to project activation)
self._active_modes: ActiveModes
self._update_active_modes(log_message=False)
# determine registered project to be activated (if any)
registered_project_to_activate: RegisteredProject | None = (
self.serena_config.get_registered_project(project, autoregister=True) if project is not None else None
@@ -628,12 +641,13 @@ class SerenaAgent:
# determine the effective language backend for this session.
# If a startup project is provided and has a per-project override, use it; otherwise use the global config.
# Since we don't want to change the toolset after startup, the language backend cannot be changed within a running Serena session
self._language_backend = self.serena_config.language_backend
if registered_project_to_activate is not None and registered_project_to_activate.project_config.language_backend is not None:
self._language_backend = registered_project_to_activate.project_config.language_backend
log.info(f"Using language backend as configured in project.yml: {self._language_backend.name}")
else:
log.info(f"Using language backend from global configuration: {self._language_backend.name}")
self._language_backend = self.serena_config.determine_language_backend(
project_config=registered_project_to_activate.project_config if registered_project_to_activate is not None else None,
log_choice=True,
)
# create the tool names mapping for prompts
self._prompt_tool_names_mapping = self._create_prompt_tool_names_mapping(self._language_backend)
# create executor for starting the language server and running tools in another thread
# This executor is used to achieve linear task execution
@@ -642,6 +656,10 @@ class SerenaAgent:
# Initialize the prompt factory
self.prompt_factory = SerenaPromptFactory()
# initialise active modes (baseline modes prior to project activation, allowing newly activated modes to be tracked)
self._active_modes: ActiveModes
self._update_active_modes(log_message=False)
# activate the given project (if any), also updating the active modes
# Note: We cannot update the active tools yet, because the base toolset has not been computed yet
# (and its computation depends on the active project)
@@ -653,9 +671,7 @@ class SerenaAgent:
self._update_active_modes()
# determine the base toolset defining the set of exposed tools (which e.g. the MCP shall see),
self._base_toolset = self._create_base_toolset(
self.serena_config, self._language_backend, self._context, self._active_modes, self._active_project
)
self._base_toolset = self._create_base_toolset(self.serena_config, self._context, self._active_modes, self._active_project)
self._exposed_tools = self._base_toolset.to_available_tools(self._all_tools)
log.info(f"Number of exposed tools: {len(self._exposed_tools)}. Exposed tools: {self._exposed_tools.tool_names}")
@@ -719,7 +735,6 @@ class SerenaAgent:
def _create_base_toolset(
cls,
serena_config: SerenaConfig,
language_backend: LanguageBackend,
context: SerenaAgentContext,
modes: ActiveModes,
project: Project | None,
@@ -730,9 +745,9 @@ class SerenaAgent:
* dashboard availability/opening on launch
* Serena config
* the context (which is fixed for the session)
* the optional tools enabled by initial modes
* the base modes (including background base modes like JetBrains mode)
* the optional tools enabled by initial dynamic modes
* single-project mode reductions (if applicable)
* JetBrains mode
"""
# determine whether to include the OpenDashboardTool based on the Serena configuration
tool_inclusion_definitions: list[ToolInclusionDefinition] = []
@@ -751,7 +766,7 @@ class SerenaAgent:
# consider modes
# * base modes: These cannot be changed, so they are fully applied
for base_mode in modes.get_base_modes():
for base_mode in modes.get_base_modes(include_background_base_modes=True):
tool_inclusion_definitions.append(base_mode)
# * dynamically activated modes:
# - When not in a single-project context, these modes can later be turned off,
@@ -788,10 +803,6 @@ class SerenaAgent:
)
tool_inclusion_definitions.append(project.project_config)
# enabled the internal 'jetbrains' mode for the JetBrains backend
if language_backend == LanguageBackend.JETBRAINS:
tool_inclusion_definitions.append(SerenaAgentMode.from_name_internal("jetbrains"))
# compute the resulting tool set
base_toolset = ToolSet.default().apply(*tool_inclusion_definitions)
log.info(f"Number of exposed tools: {len(base_toolset)}")
@@ -912,15 +923,41 @@ class SerenaAgent:
raise ValueError("No active project. Please activate a project first.")
return project
def get_active_modes(self) -> list[SerenaAgentMode]:
def get_active_modes(self) -> ActiveModes:
"""
:return: the list of active modes
:return: the active modes
"""
return list(self._active_modes.get_modes())
return self._active_modes
@staticmethod
def _create_prompt_tool_names_mapping(language_backend: LanguageBackend) -> dict[str, str]:
"""
Creates a mapping from tool names to new tool names, which take into consideration
* legacy tool names, where the name was changed and
* LSP tools which are functionally replaced by other tools due to the active language backend
(e.g. "find_symbol" being replaced by "jet_brains_find_symbol" in JetBrains mode).
The mapping is intended to be used for the generation of prompts, such that prompts can
refer to tool names as `{{ tool_names["find_symbol"] }}`, and the mapping will ensure that
the correct tool name is used in the prompt based on the active language backend.
:return: the mapping from tool names to new tool names
"""
result = dict(ToolSet.LEGACY_TOOL_NAME_MAPPING)
class_replacements = language_backend.get_lsp_tool_class_replacements()
for tool_class in ToolRegistry().get_all_tool_classes():
new_tool_class: type[Tool] = class_replacements.get(tool_class, tool_class)
result[tool_class.get_name_from_cls()] = new_tool_class.get_name_from_cls()
return result
def _format_prompt(self, prompt_template: str) -> str:
template = JinjaTemplate(prompt_template)
return template.render(available_tools=self._exposed_tools.tool_names, available_markers=self._exposed_tools.tool_marker_names)
return template.render(
available_tools=self._exposed_tools.tool_names,
available_markers=self._exposed_tools.tool_marker_names,
tool_names=self._prompt_tool_names_mapping,
)
def create_connection_prompt(self) -> str:
"""
@@ -948,7 +985,7 @@ class SerenaAgent:
# determine modes for which prompts must (still) be provided, excluding modes that were already provided in a
# previously provided project activation message (if any)
relevant_modes = []
for mode in self.get_active_modes():
for mode in self.get_active_modes().get_modes(include_background_base_modes=True):
if mode.has_prompt():
if not self._project_prompt_status.is_mode_prompt_already_provided(mode.name, session_id):
relevant_modes.append(mode)
@@ -960,6 +997,7 @@ class SerenaAgent:
available_tools=available_tools.tool_names,
available_markers=available_markers,
global_memories_list=global_memories_str,
tool_names=self._prompt_tool_names_mapping,
)
# provide the project activation message if it hasn't yet been provided
@@ -1027,7 +1065,10 @@ class SerenaAgent:
Updates the active modes based on the Serena configuration, the active project configuration (if any),
and mode overrides (if any).
"""
self._active_modes = ActiveModes()
background_base_modes = []
if self._language_backend.is_jetbrains():
background_base_modes.append(SerenaAgentMode.from_name_internal("jetbrains"))
self._active_modes = ActiveModes(background_base_modes=background_base_modes)
self._active_modes.apply(self.serena_config)
if self._active_project:
self._active_modes.apply(self._active_project.project_config)
@@ -1159,17 +1200,78 @@ class SerenaAgent:
if update_active_tools:
self._update_active_tools()
def init_language_server_manager() -> None:
# start the language server
def init_project_services() -> None:
self._run_project_activation_command(project)
self._init_active_project_language_backend()
# initialise the project's language backend in the background
self.issue_task(init_project_services)
if self._project_activation_callback is not None:
self._project_activation_callback()
# notify the dashboard manager of the project change (if applicable)
if self._dashboard_manager:
self._dashboard_manager.update_active_project(self._active_project)
return True
@staticmethod
def _run_project_activation_command(project: Project) -> None:
"""
Runs the given project's activation_command (if set and the project is trusted).
Failures are logged.
"""
activation_command = project.project_config.activation_command
if not activation_command:
return
if not project.is_trusted():
log.warning(
f"Project path {project.project_root} is not trusted, ignoring activation_command "
"from project configuration. To trust the project, modify the trusted path patterns "
"in the global configuration."
)
return
timeout = project.project_config.activation_command_timeout
cmd = subprocess_util.convert_shell_cmd(activation_command)
log.info(f"Running activation_command for project '{project.project_name}': {cmd}")
try:
with LogTime("Project activation command", logger=log):
p = subprocess.Popen(
cmd,
shell=True,
cwd=project.project_root,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE,
**subprocess_util.subprocess_kwargs(),
)
try:
_, stderr = p.communicate(timeout=timeout)
if p.returncode != 0:
log.error(f"activation_command for project '{project.project_name}' failed (exit {p.returncode}): {stderr.strip()}")
except subprocess.TimeoutExpired:
log.error(
f"Activation_command for project '{project.project_name}' timed out after "
f"{timeout}s; terminating process and continuing with backend initialisation."
)
terminate_process_tree_with_kill_fallback(p, terminate_timeout=5.0, process_name="activation_command")
except Exception:
log.exception(f"Unexpected error running activation_command for project '{project.project_name}'")
def _init_active_project_language_backend(self) -> None:
"""
Initialises the active project's language backend
"""
project = self._active_project
assert project is not None
# for LSP mode, start the language server manager
if self.get_language_backend().is_lsp():
with LogTime("Language server initialization", logger=log):
self.reset_language_server_manager()
# initialize the language server in the background (if in language server mode)
if self.get_language_backend().is_lsp():
self.issue_task(init_language_server_manager)
def init_jetbrains_ide() -> None:
assert self.serena_config.jetbrains_launch_command is not None
# for JetBrains mode, search for plugin server and spawn IDE (if not found and launch command provided)
elif self.get_language_backend().is_jetbrains():
try:
client = jetbrains_plugin_client.JetBrainsPluginClient.from_project(project, log_warning=False)
log.info("Found Serena JetBrains Plugin server: %s", client)
@@ -1183,19 +1285,6 @@ class SerenaAgent:
if p.returncode != 0:
log.error(f"Failed to launch JetBrains IDE: {stderr.decode('utf-8')}")
# for JetBrains mode, search for plugin server and spawn IDE if not found and enabled
if self.get_language_backend().is_jetbrains():
self.issue_task(init_jetbrains_ide)
if self._project_activation_callback is not None:
self._project_activation_callback()
# notify the dashboard manager of the project change (if applicable)
if self._dashboard_manager:
self._dashboard_manager.update_active_project(self._active_project)
return True
def activate_project_from_path_or_name(
self, project_root_or_name: str, update_active_modes: bool = True, update_active_tools: bool = True
) -> bool:
@@ -1211,7 +1300,7 @@ class SerenaAgent:
if project_instance is not None:
log.info(f"Found registered project '{project_instance.project_name}' at path {project_instance.project_root}")
elif os.path.isdir(project_root_or_name):
project_instance = self.serena_config.add_project_from_path(project_root_or_name)
project_instance = self.serena_config.add_project_from_path(project_root_or_name, asynchronous_autogen=True)
log.info(f"Added new project {project_instance.project_name} for path {project_instance.project_root}")
if project_instance is None:
@@ -1261,7 +1350,7 @@ class SerenaAgent:
result_str += f"Active context: {self._context.name}\n"
# Active modes
active_mode_names = [mode.name for mode in self.get_active_modes()]
active_mode_names = self.get_active_modes().get_mode_names()
result_str += "Active modes: {}\n".format(", ".join(active_mode_names)) + "\n"
# Available but not active modes
@@ -1316,7 +1405,7 @@ class SerenaAgent:
self.issue_task(lambda: self.get_active_project_or_raise().remove_language(language), name=f"RemoveLanguage:{language.value}")
def get_tool(self, tool_class: type[TTool]) -> TTool:
return self._all_tools[tool_class] # type: ignore
return self._all_tools[tool_class]
def print_tool_overview(self) -> None:
ToolRegistry().print_tool_overview(self._active_tools.tools)
src/serena/cli.py
+26 -43
View File
@@ -65,7 +65,16 @@ For details on mode configuration, see
def find_project_root(root: str | Path | None = None) -> str | None:
"""Find project root by walking up from CWD.
Checks for .serena/project.yml first (explicit Serena project), then .git (git root).
Returns the nearest ancestor that is either an explicit Serena project
(contains .serena/project.yml) or a git root (contains .git, which may be a
directory or, for git worktrees and submodules, a pointer file). The nearest
such directory wins; a .serena/project.yml at the same level takes priority
over .git only because they resolve to the same directory.
Walking up with a single pass (rather than searching all levels for
.serena/project.yml first and only then for .git) ensures a git worktree
nested under another Serena project resolves to the worktree itself, instead
of being hijacked by the ancestor project's .serena/project.yml.
:param root: If provided, constrains the search to this directory and below
(acts as a virtual filesystem root). Search stops at this boundary.
@@ -82,14 +91,12 @@ def find_project_root(root: str | Path | None = None) -> str | None:
if boundary is not None and parent == boundary:
return
# First pass: look for .serena
# Single pass: the nearest project boundary wins, whether it is an explicit
# Serena project (.serena/project.yml) or a git root (.git, a dir or a worktree
# pointer file). This keeps a nested git worktree from being hijacked by an
# ancestor's .serena/project.yml.
for directory in ancestors():
if (directory / ".serena" / "project.yml").is_file():
return str(directory)
# Second pass: look for .git
for directory in ancestors():
if (directory / ".git").exists(): # .git can be file (worktree) or dir
if (directory / ".serena" / "project.yml").is_file() or (directory / ".git").exists():
return str(directory)
return None
@@ -177,9 +184,7 @@ class TopLevelCommands(AutoRegisteringGroup):
)
def init(language_backend: Literal["LSP", "JetBrains"] = "LSP") -> None:
click.echo(f"\nSerena version: {serena_version()}\n")
serena_config = SerenaConfig.from_config_file()
serena_config.language_backend = LanguageBackend(language_backend)
serena_config.save()
serena_config = SerenaConfig.init(language_backend=LanguageBackend(language_backend))
click.echo(f"Configuration file: {serena_config.config_file_path}")
click.echo(f"Language backend: {language_backend}")
@@ -390,13 +395,7 @@ class TopLevelCommands(AutoRegisteringGroup):
@click.command(
"print-system-prompt", help="Print the system prompt for a project.", context_settings={"max_content_width": _MAX_CONTENT_WIDTH}
)
@click.argument("project", type=click.Path(exists=True), default=os.getcwd(), required=False)
@click.option(
"--log-level",
type=click.Choice(["DEBUG", "INFO", "WARNING", "ERROR", "CRITICAL"]),
default="WARNING",
help="Log level for prompt generation.",
)
@click.argument("project", type=click.Path(exists=True), default=None, required=False)
@click.option("--only-instructions", is_flag=True, help="Print only the initial instructions, without prefix/postfix.")
@click.option(
"--context", type=str, default=DEFAULT_CONTEXT, show_default=True, help="Built-in context name or path to custom context YAML."
@@ -410,27 +409,19 @@ class TopLevelCommands(AutoRegisteringGroup):
show_default=False,
help=_MODES_EXPLANATION,
)
def print_system_prompt(
project: str, log_level: str, only_instructions: bool, context: str, modes: Sequence[str] | None = None
) -> None:
def print_system_prompt(project: str | None, only_instructions: bool, context: str, modes: Sequence[str] | None = None) -> None:
from serena.agent import SerenaAgent
prefix = "You will receive access to Serena's symbolic tools. Below are instructions for using them, take them into account."
postfix = "You begin by acknowledging that you understood the above instructions and are ready to receive tasks."
lvl = logging.getLevelNamesMapping()[log_level.upper()]
logging.configure(level=lvl)
context_instance = SerenaAgentContext.load(context)
modes_selection_def: ModeSelectionDefinition | None = None
if modes:
modes_selection_def = ModeSelectionDefinition(default_modes=modes)
serena_config = SerenaConfig.from_config_file()
serena_config.web_dashboard = False
print(serena_config.default_modes)
print(serena_config.base_modes)
serena_config = SerenaConfig.from_config_file().with_headless_mode_overrides()
agent = SerenaAgent(
project=os.path.abspath(project),
project=os.path.abspath(project) if project is not None else None,
serena_config=serena_config,
context=context_instance,
modes=modes_selection_def,
@@ -923,10 +914,8 @@ class ProjectCommands(AutoRegisteringGroup):
logging.configure(level=logging.INFO)
project_path = os.path.abspath(project)
serena_config = SerenaConfig.from_config_file()
serena_config = SerenaConfig.from_config_file().with_headless_mode_overrides()
serena_config.language_backend = LanguageBackend.LSP
serena_config.gui_log_window = False
serena_config.web_dashboard = False
proj = Project.load(project_path, serena_config=serena_config)
# Create log file with timestamp
@@ -1108,7 +1097,7 @@ class ToolCommands(AutoRegisteringGroup):
agent = SerenaAgent(
project=None,
serena_config=SerenaConfig(web_dashboard=False, log_level=logging.INFO),
serena_config=SerenaConfig(log_level=logging.INFO).with_headless_mode_overrides(),
context=serena_context,
)
tool = agent.get_tool_by_name(tool_name)
@@ -1135,17 +1124,11 @@ class MemoryCommands(AutoRegisteringGroup):
exist at the configured location). Never auto-creates the ``.serena`` directory —
if no project configuration is found, the user is directed at ``serena project create``.
"""
from serena.project import Project
serena_config = SerenaConfig.from_config_file()
project_path = os.path.abspath(project)
try:
proj = Project.load(project_path, serena_config=serena_config, autogenerate=False)
except FileNotFoundError as e:
raise click.UsageError(
f"{e}\nNo Serena project found at {project_path}. Create one first with:\n serena project create {project_path}"
) from e
return proj.memory_manager
registered_project = serena_config.get_registered_project(project)
if registered_project is None:
raise click.UsageError(f"No Serena project found for '{project}'. Create one first.")
return registered_project.get_project_instance(serena_config).memory_manager
@staticmethod
@click.command(
src/serena/code_editor.py
+27 -3
View File
@@ -10,9 +10,10 @@ from serena.jetbrains.jetbrains_plugin_client import JetBrainsPluginClient
from serena.symbol import JetBrainsSymbol, LanguageServerSymbol, LanguageServerSymbolRetriever, PositionInFile, Symbol
from solidlsp import SolidLanguageServer, ls_types
from solidlsp.ls import LSPFileBuffer
from solidlsp.ls_utils import PathUtils, TextUtils
from solidlsp.ls_utils import PathUtils, TextStepper, TextUtils
from .project import Project
from .util.file_proxy import FileProxy
log = logging.getLogger(__name__)
TSymbol = TypeVar("TSymbol", bound=Symbol)
@@ -78,6 +79,8 @@ class CodeEditor(Generic[TSymbol], ABC):
"""
Context manager for editing a file.
"""
if FileProxy.is_external_path(relative_path):
raise ValueError(f"Cannot edit external file: {relative_path}")
with self._open_file_context(relative_path) as edited_file:
yield edited_file
# save the file
@@ -239,9 +242,31 @@ class CodeEditor(Generic[TSymbol], ABC):
symbol = self._find_unique_symbol(name_path, relative_file_path)
start_pos = symbol.get_body_start_position_or_raise()
end_pos = symbol.get_body_end_position_or_raise()
with self.edited_file_context(relative_file_path) as edited_file:
# do the actual deletion
edited_file.delete_text_between_positions(start_pos, end_pos)
# empty line removal heuristic:
# After the deletion, go to the line where the deleted symbol started,
# check whether it and any subsequent lines are empty, and if so, delete them.
# The empty lines preceding the deleted symbol's body should normally suffice as a separator
# between the symbols before and after the deleted symbol.
start_pos.col = 0
stepper = TextStepper(edited_file.get_contents())
stepper.step_to(start_pos.line, start_pos.col)
apply_empty_line_deletion = False
while stepper.step_line():
line = stepper.get_last_line(with_end=False)
if line.strip() == "":
end_pos.line = stepper.line
end_pos.col = stepper.col
apply_empty_line_deletion = True
else:
break
if apply_empty_line_deletion:
edited_file.delete_text_between_positions(start_pos, end_pos)
@abstractmethod
def rename_symbol(self, name_path: str, relative_path: str, new_name: str) -> str:
pass
@@ -400,8 +425,7 @@ class JetBrainsCodeEditor(CodeEditor[JetBrainsSymbol]):
super().__init__(relative_path)
path = os.path.join(project.project_root, relative_path)
log.info("Editing file: %s", path)
with open(path, encoding=project.project_config.encoding) as f:
self._content = f.read()
self._content = FileProxy.from_project_relative_path(project, relative_path).get_contents()
def get_contents(self) -> str:
return self._content
src/serena/config/client_setup.py
+24 -3
View File
@@ -53,7 +53,7 @@ class ClientSetupHandlerClaudeCode(ClientSetupHandler):
super().__init__("claude-code")
def is_applicable(self) -> bool:
result = execute_shell_command("claude --version")
result = execute_shell_command("claude --version", capture_stderr=True)
return result.return_code == 0 and "Claude" in result.stdout
def get_mcp_server_options(self) -> list[str]:
@@ -78,7 +78,7 @@ class ClientSetupHandlerCodex(ClientSetupHandler):
super().__init__("codex")
def is_applicable(self) -> bool:
result = execute_shell_command("codex --version")
result = execute_shell_command("codex --version", capture_stderr=True)
return result.return_code == 0 and "codex-cli" in result.stdout
def get_mcp_server_options(self) -> list[str]:
@@ -88,4 +88,25 @@ class ClientSetupHandlerCodex(ClientSetupHandler):
return self._run_shell_command(f"codex mcp add serena -- {self.get_mcp_server_command()}")
client_setup_handlers = [ClientSetupHandlerClaudeCode(), ClientSetupHandlerCodex()]
class ClientSetupHandlerCodeBuddy(ClientSetupHandler):
def __init__(self) -> None:
super().__init__("codebuddy")
def is_applicable(self) -> bool:
result = execute_shell_command("codebuddy --version", capture_stderr=True)
return result.return_code == 0
def get_mcp_server_options(self) -> list[str]:
return ["--context=codebuddy", "--project-from-cwd"]
def apply(self) -> bool:
cmd = f"codebuddy mcp add --scope user serena -- {self.get_mcp_server_command()}"
is_success = self._run_shell_command(cmd)
if is_success:
click.echo("\nIMPORTANT: We additionally recommend to set up hooks for CodeBuddy to ensure the best experience.")
click.echo(" Please read the instructions here:")
click.echo(" https://oraios.github.io/serena/02-usage/030_clients.html#codebuddy")
return is_success
client_setup_handlers = [ClientSetupHandlerClaudeCode(), ClientSetupHandlerCodeBuddy(), ClientSetupHandlerCodex()]
src/serena/config/context_mode.py
+24 -18
View File
@@ -26,6 +26,10 @@ if TYPE_CHECKING:
log = logging.getLogger(__name__)
def looks_like_yaml_path(s: str) -> bool:
return os.sep in s or (os.altsep and os.altsep in s) or s.lower().endswith((".yml", ".yaml"))
@dataclass(kw_only=True)
class SerenaAgentMode(ToolInclusionDefinition, ToStringMixin):
"""Represents a mode of operation for the agent, typically read off a YAML file.
@@ -124,16 +128,15 @@ class SerenaAgentMode(ToolInclusionDefinition, ToStringMixin):
@classmethod
def load(cls, name_or_path: str | Path) -> Self:
# Check if it's a file path that exists
path = Path(name_or_path)
if path.exists() and path.is_file():
return cls.from_yaml(name_or_path)
# If it looks like a file path but doesn't exist, raise FileNotFoundError
name_or_path_str = str(name_or_path)
if os.sep in name_or_path_str or (os.altsep and os.altsep in name_or_path_str) or name_or_path_str.endswith((".yml", ".yaml")):
raise FileNotFoundError(f"Mode file not found: {path.resolve()}")
# If it is a path or looks like a path, load from file
if isinstance(name_or_path, Path) or looks_like_yaml_path(str(name_or_path)):
path = Path(name_or_path)
if path.exists() and path.is_file():
return cls.from_yaml(name_or_path)
else:
raise FileNotFoundError(f"Mode file not found: {path.resolve()}")
# load from name
return cls.from_name(str(name_or_path))
def has_prompt(self) -> bool:
@@ -181,6 +184,11 @@ class SerenaAgentContext(ToolInclusionDefinition, ToStringMixin):
The `activate_project` tool will, therefore, be disabled in this case, as project switching is not allowed.
"""
structured_tool_output: bool | None = None
"""
whether to use structured output for tools (None = auto)
"""
def _tostring_includes(self) -> list[str]:
return ["name"]
@@ -239,15 +247,13 @@ class SerenaAgentContext(ToolInclusionDefinition, ToStringMixin):
@classmethod
def load(cls, name_or_path: str | Path) -> Self:
# Check if it's a file path that exists
path = Path(name_or_path)
if path.exists() and path.is_file():
return cls.from_yaml(name_or_path)
# If it looks like a file path but doesn't exist, raise FileNotFoundError
name_or_path_str = str(name_or_path)
if os.sep in name_or_path_str or (os.altsep and os.altsep in name_or_path_str) or name_or_path_str.endswith((".yml", ".yaml")):
raise FileNotFoundError(f"Context file not found: {path.resolve()}")
# If it is a path or looks like a path, load from file
if isinstance(name_or_path, Path) or looks_like_yaml_path(str(name_or_path)):
path = Path(name_or_path)
if path.exists() and path.is_file():
return cls.from_yaml(name_or_path)
else:
raise FileNotFoundError(f"Context file not found: {path.resolve()}")
return cls.from_name(str(name_or_path))
src/serena/config/serena_config.py
+330 -97
View File
@@ -6,6 +6,7 @@ import dataclasses
import os
import re
import shutil
import threading
from collections.abc import Iterator, Sequence
from copy import deepcopy
from dataclasses import dataclass, field
@@ -32,6 +33,7 @@ from serena.constants import (
SERENA_MANAGED_DIR_NAME,
)
from serena.util.inspection import determine_programming_language_composition
from serena.util.text_utils import glob_match
from serena.util.yaml import YamlCommentNormalisation, load_yaml, normalise_yaml_comments, save_yaml, transfer_yaml_comments
from solidlsp.ls_config import Language
@@ -42,6 +44,7 @@ from ..util.dataclass import get_dataclass_default
if TYPE_CHECKING:
from ..project import Project
from ..tools.tools_base import Tool
log = logging.getLogger(__name__)
T = TypeVar("T")
@@ -217,6 +220,28 @@ class LanguageBackend(Enum):
def is_jetbrains(self) -> bool:
return self == LanguageBackend.JETBRAINS
def get_lsp_tool_class_replacements(self) -> "dict[type[Tool], type[Tool]]":
"""
:return: mapping from LSP tool classes to replacement tool classes (functional replacements)
"""
match self:
case LanguageBackend.LSP:
return {}
case LanguageBackend.JETBRAINS:
from ..tools import jetbrains_tools, symbol_tools
return {
symbol_tools.FindSymbolTool: jetbrains_tools.JetBrainsFindSymbolTool,
symbol_tools.GetSymbolsOverviewTool: jetbrains_tools.JetBrainsGetSymbolsOverviewTool,
symbol_tools.FindReferencingSymbolsTool: jetbrains_tools.JetBrainsFindReferencingSymbolsTool,
symbol_tools.FindImplementationsTool: jetbrains_tools.JetBrainsFindImplementationsTool,
symbol_tools.FindDeclarationTool: jetbrains_tools.JetBrainsFindDeclarationTool,
symbol_tools.RenameSymbolTool: jetbrains_tools.JetBrainsRenameTool,
symbol_tools.SafeDeleteSymbol: jetbrains_tools.JetBrainsSafeDeleteTool,
}
case _:
raise NotImplementedError()
class LineEnding(Enum):
"""Line ending convention for file writes."""
@@ -274,33 +299,105 @@ Uses $projectDir and $projectFolderName as placeholders.
"""
class ProjectConfigAutoGenerationMode(Enum):
NONE = "none"
"""
no auto-generation
"""
SYNCHRONOUS = "sync"
"""
synchronous auto-generation, i.e. the configuration is fully generated before returning from the function call
"""
ASYNCHRONOUS = "async"
"""
asynchronous auto-generation, where time-consuming configuration parts (currently only the
list of programming languages) are determined in a background thread and initialised as empty
"""
def is_autogen_enabled(self):
return self != ProjectConfigAutoGenerationMode.NONE
@dataclass(kw_only=True)
class ProjectConfig(SharedConfig, ModeSelectionDefinitionWithAddedModes):
project_name: str
languages: list[Language]
ignored_paths: list[str] = field(default_factory=list)
additional_workspace_folders: list[str] = field(default_factory=list)
ls_workspace_folders: list[str] = field(default_factory=lambda: ["."])
ls_additional_workspace_folders: list[str] = field(default_factory=list)
read_only: bool = False
ignore_all_files_in_gitignore: bool = True
initial_prompt: str = ""
encoding: str = DEFAULT_SOURCE_FILE_ENCODING
activation_command: str | None = None
activation_command_timeout: float = 180.0
# internal fields which are not mapped to/from the configuration file (must start with "_")
_local_override_keys: list[str] = field(default_factory=list)
# class-level constants
# class-level members
SERENA_PROJECT_FILE = "project.yml"
SERENA_LOCAL_PROJECT_FILE = "project.local.yml"
FIELDS_WITHOUT_DEFAULTS = {"project_name", "languages"}
RENAMED_FIELDS = {"additional_workspace_folders": "ls_additional_workspace_folders"}
YAML_COMMENT_NORMALISATION = YamlCommentNormalisation.LEADING
"""
the comment normalisation strategy to use when loading/saving project configuration files.
The template file must match this configuration (i.e. it must use leading comments if this is set to LEADING).
"""
_async_completion_events = {}
"""
maps the object id of a ProjectConfig instance to an event which is set when the asynchronous auto-generation of
the configuration is complete (if applicable).
"""
_save_lock = threading.Lock()
def _tostring_includes(self) -> list[str]:
return ["project_name"]
@classmethod
def _determine_project_languages(cls, project_root: str, interactive: bool) -> list[Language]:
# determine languages automatically
log.info("Determining programming languages used in the project")
language_composition = determine_programming_language_composition(project_root)
log.info("Language composition: %s", language_composition)
if len(language_composition) == 0:
log.warning(
"No source files for supported language servers were found in %s. "
"Creating project with no configured languages. "
"Symbol-related tools (e.g. find_symbol, get_symbols_overview) will not work "
"when using the LSP backend. You can add languages later via the Serena dashboard "
"or by manually editing the project configuration.",
project_root,
)
languages_to_use = []
else:
# sort languages by number of files found
languages_and_percentages = sorted(
language_composition.items(), key=lambda item: (item[1], item[0].get_priority()), reverse=True
)
# find the language with the highest percentage and enable it
top_language_pair = languages_and_percentages[0]
other_language_pairs = languages_and_percentages[1:]
languages_to_use = [top_language_pair[0]]
# if in interactive mode, ask the user which other languages to enable
if len(other_language_pairs) > 0 and interactive:
print(
"Detected and enabled main language '%s' (%.2f%% of source files)." % (top_language_pair[0].value, top_language_pair[1])
)
print(f"Additionally detected {len(other_language_pairs)} other language(s).\n")
print("Note: Enable only languages you need symbolic retrieval/editing capabilities for.")
print(" Additional language servers use resources and some languages may require additional")
print(" system-level installations/configuration (see Serena documentation).")
print("\nWhich additional languages do you want to enable?")
for lang, perc in other_language_pairs:
enable = ask_yes_no("Enable %s (%.2f%% of source files)?" % (lang.value, perc), default=False)
if enable:
languages_to_use.append(lang)
print()
log.info("Using languages: %s", languages_to_use)
return languages_to_use
@classmethod
def autogenerate(
cls,
@@ -310,6 +407,7 @@ class ProjectConfig(SharedConfig, ModeSelectionDefinitionWithAddedModes):
languages: list[Language] | None = None,
save_to_disk: bool = True,
interactive: bool = False,
asynchronous: bool = False,
) -> Self:
"""
Autogenerate a project configuration for a given project root.
@@ -321,8 +419,12 @@ class ProjectConfig(SharedConfig, ModeSelectionDefinitionWithAddedModes):
:param languages: the languages of the project; if None, they will be determined automatically
:param save_to_disk: whether to save the project configuration to disk
:param interactive: whether to run in interactive CLI mode, asking the user for input where appropriate
:param asynchronous: whether to run in asynchronous mode, where time-consuming configuration parts (currently only the
determination of the list of programming languages) are determined in a background thread and initialised as empty
:return: the project configuration
"""
if interactive and asynchronous:
raise ValueError("Cannot use interactive mode with asynchronous auto-generation")
project_root = Path(project_root).resolve()
if not project_root.exists():
raise FileNotFoundError(f"Project root not found: {project_root}")
@@ -330,61 +432,62 @@ class ProjectConfig(SharedConfig, ModeSelectionDefinitionWithAddedModes):
log.info("Project root: %s", project_root)
project_folder_name = project_root.name
project_name = project_name or project_folder_name
use_asynchronous_language_determination = False
if languages is None:
# determine languages automatically
log.info("Determining programming languages used in the project")
language_composition = determine_programming_language_composition(str(project_root))
log.info("Language composition: %s", language_composition)
if len(language_composition) == 0:
log.warning(
"No source files for supported language servers were found in %s. "
"Creating project with no configured languages. "
"Symbol-related tools (e.g. find_symbol, get_symbols_overview) will not work "
"when using the LSP backend. You can add languages later via the Serena dashboard "
"or by manually editing the project configuration.",
project_root,
)
languages_to_use: list[str] = []
if asynchronous:
use_asynchronous_language_determination = True
languages_to_use = [] # temporarily empty, will be determined in background thread
else:
# sort languages by number of files found
languages_and_percentages = sorted(
language_composition.items(), key=lambda item: (item[1], item[0].get_priority()), reverse=True
)
# find the language with the highest percentage and enable it
top_language_pair = languages_and_percentages[0]
other_language_pairs = languages_and_percentages[1:]
languages_to_use = [top_language_pair[0].value]
# if in interactive mode, ask the user which other languages to enable
if len(other_language_pairs) > 0 and interactive:
print(
"Detected and enabled main language '%s' (%.2f%% of source files)."
% (top_language_pair[0].value, top_language_pair[1])
)
print(f"Additionally detected {len(other_language_pairs)} other language(s).\n")
print("Note: Enable only languages you need symbolic retrieval/editing capabilities for.")
print(" Additional language servers use resources and some languages may require additional")
print(" system-level installations/configuration (see Serena documentation).")
print("\nWhich additional languages do you want to enable?")
for lang, perc in other_language_pairs:
enable = ask_yes_no("Enable %s (%.2f%% of source files)?" % (lang.value, perc), default=False)
if enable:
languages_to_use.append(lang.value)
print()
log.info("Using languages: %s", languages_to_use)
determined_languages = cls._determine_project_languages(str(project_root), interactive=interactive)
languages_to_use = [l.value for l in determined_languages]
else:
languages_to_use = [lang.value for lang in languages]
config_with_comments, _ = cls._load_yaml_dict(PROJECT_TEMPLATE_FILE)
config_with_comments["project_name"] = project_name
config_with_comments["languages"] = languages_to_use
project_yml_path = serena_config.get_project_yml_location(str(project_root))
if save_to_disk:
project_yml_path = serena_config.get_project_yml_location(str(project_root))
log.info("Saving project configuration to %s", project_yml_path)
save_yaml(project_yml_path, config_with_comments)
with cls._save_lock:
save_yaml(project_yml_path, config_with_comments)
project_local_yml_path = os.path.join(os.path.dirname(project_yml_path), cls.SERENA_LOCAL_PROJECT_FILE)
shutil.copy(PROJECT_LOCAL_TEMPLATE_FILE, project_local_yml_path)
return cls._from_dict(config_with_comments, local_override_keys=[])
project_config = cls._from_dict(config_with_comments, local_override_keys=[])
# if asynchronous language determination is used, start a background thread which updates and saves the configuration
# and set an event which can be awaited to ensure that the configuration is complete
if use_asynchronous_language_determination:
event = threading.Event()
project_config._async_completion_events[id(project_config)] = event
def async_language_determination():
try:
with LogTime("Asynchronous language determination", logger=log):
project_config.languages = cls._determine_project_languages(str(project_root), interactive=False)
if save_to_disk:
project_config.save(project_yml_path)
finally:
event.set()
threading.Thread(target=async_language_determination, name="project-language-determination", daemon=True).start()
return project_config
def await_asynchronous_completion(self):
"""
Wait for the asynchronous auto-generation of the configuration to complete (if applicable), ensuring
that, in particular, the list of programming languages is complete, which may be determined asynchronously
when first creating the project configuration.
"""
event = ProjectConfig._async_completion_events.get(id(self))
if event is None:
return
if not event.is_set():
log.info("Waiting for asynchronous auto-generation of project configuration to complete ...")
event.wait()
log.info("Asynchronous auto-generation of project configuration completed.")
@classmethod
def default_project_yml_path(cls, project_root: str | Path) -> str:
@@ -417,9 +520,22 @@ class ProjectConfig(SharedConfig, ModeSelectionDefinitionWithAddedModes):
and `was_complete` will always be True.
"""
data = load_yaml(yml_path, comment_normalisation=comment_normalisation)
was_complete = True
# backward compatibility
# NOTE: This must also work for project.local.yml files, which may be highly incomplete
# * handle single "language" field
if "languages" not in data and "language" in data:
data["languages"] = [data["language"]]
del data["language"]
# * handle renamed fields
for old_key, new_key in cls.RENAMED_FIELDS.items():
if old_key in data and new_key not in data:
data[new_key] = data[old_key]
del data[old_key]
was_complete = False
# apply defaults
was_complete = True
if apply_defaults:
for field_info in dataclasses.fields(cls):
key = field_info.name
@@ -432,13 +548,6 @@ class ProjectConfig(SharedConfig, ModeSelectionDefinitionWithAddedModes):
default_value = get_dataclass_default(cls, key)
data.setdefault(key, default_value)
# backward compatibility
# NOTE: This must also work for project.local.yml files, which may be highly incomplete
# * handle single "language" field
if "languages" not in data and "language" in data:
data["languages"] = [data["language"]]
del data["language"]
# Note: Checks for validity of fields must not happen here but in _from_dict.
# Here, the data may be incomplete, because this function is also used for
# loading project.local.yml files.
@@ -454,6 +563,7 @@ class ProjectConfig(SharedConfig, ModeSelectionDefinitionWithAddedModes):
the ProjectConfig dataclass
:param local_override_keys: the list of keys that have been overridden from project.local.yml
"""
# map languages to list of enum items, checking for errors
lang_name_mapping = {"javascript": "typescript"}
languages: list[Language] = []
for language_str in data["languages"]:
@@ -469,6 +579,15 @@ class ProjectConfig(SharedConfig, ModeSelectionDefinitionWithAddedModes):
f"Invalid language: {orig_language_str}.\nValid language_strings are: {[l.value for l in Language]}"
) from e
# Validate activation_command_timeout
activation_command_timeout_raw = data.get("activation_command_timeout", 180.0)
try:
activation_command_timeout = float(activation_command_timeout_raw)
except (TypeError, ValueError) as e:
raise ValueError(f"activation_command_timeout must be a number, got: {activation_command_timeout_raw}") from e
if activation_command_timeout <= 0:
raise ValueError(f"activation_command_timeout must be positive, got: {activation_command_timeout}")
# Validate symbol_info_budget
symbol_info_budget_raw = data["symbol_info_budget"]
symbol_info_budget = symbol_info_budget_raw
@@ -491,7 +610,7 @@ class ProjectConfig(SharedConfig, ModeSelectionDefinitionWithAddedModes):
fixed_tools = data["fixed_tools"] or []
excluded_tools = data["excluded_tools"] or []
included_optional_tools = data["included_optional_tools"] or []
additional_workspace_folders = data.get("additional_workspace_folders") or []
additional_workspace_folders = data.get("ls_additional_workspace_folders") or []
if "base_modes" in data and data["base_modes"] is not None:
log.warning("The base_modes setting in project.yml is deprecated and will be ignored.")
@@ -500,7 +619,8 @@ class ProjectConfig(SharedConfig, ModeSelectionDefinitionWithAddedModes):
project_name=data["project_name"],
languages=languages,
ignored_paths=ignored_paths,
additional_workspace_folders=additional_workspace_folders,
ls_workspace_folders=data["ls_workspace_folders"],
ls_additional_workspace_folders=additional_workspace_folders,
excluded_tools=excluded_tools,
fixed_tools=fixed_tools,
included_optional_tools=included_optional_tools,
@@ -516,6 +636,8 @@ class ProjectConfig(SharedConfig, ModeSelectionDefinitionWithAddedModes):
default_modes=data["default_modes"],
symbol_info_budget=symbol_info_budget,
ls_specific_settings=data.get("ls_specific_settings", {}),
activation_command=data.get("activation_command"),
activation_command_timeout=activation_command_timeout,
_local_override_keys=local_override_keys,
)
@@ -543,13 +665,18 @@ class ProjectConfig(SharedConfig, ModeSelectionDefinitionWithAddedModes):
return os.path.join(os.path.dirname(project_yml_path), cls.SERENA_LOCAL_PROJECT_FILE)
@classmethod
def load(cls, project_root: Path | str, serena_config: "SerenaConfig", autogenerate: bool = False) -> Self:
def load(
cls,
project_root: Path | str,
serena_config: "SerenaConfig",
autogen: ProjectConfigAutoGenerationMode = ProjectConfigAutoGenerationMode.NONE,
) -> Self:
"""
Load a ProjectConfig instance from the path to the project root.
:param project_root: the path to the project root
:param serena_config: the global Serena configuration
:param autogenerate: whether to auto-generate the configuration if it does not exist
:param autogen: the auto-generation mode to apply if the project configuration does not yet exist
"""
project_root = Path(project_root)
project_folder_name = project_root.name
@@ -558,8 +685,8 @@ class ProjectConfig(SharedConfig, ModeSelectionDefinitionWithAddedModes):
# auto-generate if necessary
if not os.path.exists(yaml_path):
if autogenerate:
return cls.autogenerate(project_root, serena_config)
if autogen.is_autogen_enabled():
return cls.autogenerate(project_root, serena_config, asynchronous=autogen == ProjectConfigAutoGenerationMode.ASYNCHRONOUS)
else:
raise FileNotFoundError(f"Project configuration file not found: {yaml_path}")
@@ -606,34 +733,35 @@ class ProjectConfig(SharedConfig, ModeSelectionDefinitionWithAddedModes):
config_path = project_yml_path
log.info("Saving updated project configuration to %s", config_path)
# get the current configuration as a dictionary
cur_dict = self._to_yaml_dict()
with self._save_lock:
# get the current configuration as a dictionary
cur_dict = self._to_yaml_dict()
# load commented map from the original file and update all non-overridden keys
config_with_comments, _ = self._load_yaml_dict(config_path, self.YAML_COMMENT_NORMALISATION)
for key in cur_dict:
if key not in self._local_override_keys:
config_with_comments[key] = cur_dict[key]
# load commented map from the original file and update all non-overridden keys
config_with_comments, _ = self._load_yaml_dict(config_path, self.YAML_COMMENT_NORMALISATION)
for key in cur_dict:
if key not in self._local_override_keys:
config_with_comments[key] = cur_dict[key]
# transfer missing comments from the template file
template_config, _ = self._load_yaml_dict(PROJECT_TEMPLATE_FILE, self.YAML_COMMENT_NORMALISATION)
transfer_yaml_comments(template_config, config_with_comments, self.YAML_COMMENT_NORMALISATION, force_update_all=True)
# transfer missing comments from the template file
template_config, _ = self._load_yaml_dict(PROJECT_TEMPLATE_FILE, self.YAML_COMMENT_NORMALISATION)
transfer_yaml_comments(template_config, config_with_comments, self.YAML_COMMENT_NORMALISATION, force_update_all=True)
# save project.yml
save_yaml(config_path, config_with_comments)
# save project.yml
save_yaml(config_path, config_with_comments)
# update project.local.yml to reflect overridden keys if necessary
if save_project_local_yml:
project_local_yml_path = self._project_local_yml_path(project_yml_path)
if self._local_override_keys and os.path.exists(project_local_yml_path):
log.info("Saving updated local project configuration to %s", project_local_yml_path)
local_config_with_comments, _ = self._load_yaml_dict(
project_local_yml_path, comment_normalisation=YamlCommentNormalisation.NONE, apply_defaults=False
)
for key in self._local_override_keys:
if key in cur_dict:
local_config_with_comments[key] = cur_dict[key]
save_yaml(project_local_yml_path, local_config_with_comments)
# update project.local.yml to reflect overridden keys if necessary
if save_project_local_yml:
project_local_yml_path = self._project_local_yml_path(project_yml_path)
if self._local_override_keys and os.path.exists(project_local_yml_path):
log.info("Saving updated local project configuration to %s", project_local_yml_path)
local_config_with_comments, _ = self._load_yaml_dict(
project_local_yml_path, comment_normalisation=YamlCommentNormalisation.NONE, apply_defaults=False
)
for key in self._local_override_keys:
if key in cur_dict:
local_config_with_comments[key] = cur_dict[key]
save_yaml(project_local_yml_path, local_config_with_comments)
class RegisteredProject(ToStringMixin):
@@ -670,8 +798,21 @@ class RegisteredProject(ToStringMixin):
)
@classmethod
def from_project_root(cls, project_root: str | Path, serena_config: "SerenaConfig") -> "RegisteredProject":
project_config = ProjectConfig.load(project_root, serena_config=serena_config)
def from_project_root(
cls,
project_root: str | Path,
serena_config: "SerenaConfig",
autogen: ProjectConfigAutoGenerationMode = ProjectConfigAutoGenerationMode.NONE,
) -> "RegisteredProject":
"""
Creates a RegisteredProject instance from a project root path, which must exist on disk.
:param project_root: path to an existing directory
:param serena_config: the Serena configuration
:param autogen: the auto-generation mode to use for the project configuration if it does not yet exist
:return: the RegisteredProject instance
"""
project_config = ProjectConfig.load(project_root, serena_config=serena_config, autogen=autogen)
return RegisteredProject(
project_root=str(project_root),
project_config=project_config,
@@ -682,9 +823,14 @@ class RegisteredProject(ToStringMixin):
Check if the given path matches the project root path.
:param path: the path to check
:return: True if the path matches the project root, False otherwise
:return: True if the path matches the project root, False otherwise (including the case
where this project's root directory no longer exists, e.g. a removed git worktree)
"""
return self.project_root.samefile(Path(path).resolve())
try:
return self.project_root.samefile(Path(path).resolve())
except OSError:
# typically raised if the path does not exist (e.g., a removed git worktree)
return False
def get_project_instance(self, serena_config: "SerenaConfig") -> "Project":
"""
@@ -727,6 +873,9 @@ class SerenaConfig(SharedConfig, ModeSelectionDefinitionWithBaseModes):
JetBrains IDE launch command, which can be used to auto-start an IDE instance on demand.
"""
tool_timeout: float = DEFAULT_TOOL_TIMEOUT
"""
timeout for tool calls in seconds; if a tool takes longer than this, it is aborted and an error is returned.
"""
token_count_estimator: str = RegisteredTokenCountEstimator.CHAR_COUNT.name
"""Only relevant if `record_tool_usage` is True; the name of the token count estimator to use for tool usage statistics.
@@ -757,7 +906,16 @@ class SerenaConfig(SharedConfig, ModeSelectionDefinitionWithBaseModes):
- "/projects-metadata/$projectFolderName/.serena" (stores data in a central location)
"""
trusted_project_path_patterns: list[str] = field(default_factory=lambda: ["**"])
"""
list of glob patterns for project root directories that are considered trusted.
The default "**" considers all project roots as trusted, which is necessary for backward compatibility.
The default will apply if a user does not yet have the setting, while new users will get the value
defined in the configuration template file.
"""
# settings with overridden defaults
language_backend: LanguageBackend = LanguageBackend.LSP
"""
the language backend to use for code understanding features
@@ -771,6 +929,7 @@ class SerenaConfig(SharedConfig, ModeSelectionDefinitionWithBaseModes):
If the budget is exceeded, Serena stops issuing further requests and returns partial info results.
0 disables the budget (no early stopping). Negative values are invalid.
"""
# *** fields that are NOT mapped to/from the configuration file ***
_loaded_commented_yaml: CommentedMap | None = None
@@ -957,7 +1116,7 @@ class SerenaConfig(SharedConfig, ModeSelectionDefinitionWithBaseModes):
# re-save the configuration file if any migrations were performed
if num_migrations > 0:
log.info("Legacy configuration was migrated; re-saving configuration file")
instance.save()
instance._save()
return instance
@@ -985,6 +1144,33 @@ class SerenaConfig(SharedConfig, ModeSelectionDefinitionWithBaseModes):
log.error(f"Error migrating configuration file: {e}")
return None
@classmethod
def init(cls, language_backend: LanguageBackend) -> "SerenaConfig":
"""
Supports the config initialisation CLI command, allowing the user to configure fundamental settings before
the first launch.
:param language_backend: the language backend to use
:return: the created SerenaConfig instance
"""
config = cls.from_config_file()
config.language_backend = language_backend
config._save()
return config
def with_headless_mode_overrides(self) -> "SerenaConfig":
"""
Modifies this instance to apply overrides for headless mode, where any GUI/user interaction-based features are disabled.
This is intended to be applied for cases where a `SerenaConfig` instance is needed to instantiate a `SerenaAgent` instance
while the user is not expected to interact with the system (e.g. a CLI command or a test).
:return: the instance with overrides applied for headless mode
"""
self.gui_log_window = False
self.web_dashboard = False
self.jetbrains_launch_command = None
return self
@cached_property
def project_paths(self) -> list[str]:
return sorted(str(project.project_root) for project in self.projects)
@@ -996,7 +1182,8 @@ class SerenaConfig(SharedConfig, ModeSelectionDefinitionWithBaseModes):
def get_registered_project(self, project_root_or_name: str, autoregister: bool = False) -> Optional[RegisteredProject]:
"""
:param project_root_or_name: path to the project root or the name of the project
:param autoregister: whether to register the project if it exists but is not registered yet
:param autoregister: whether to auto-register projects that are not yet registered in Serena's global configuration
but have an existing project configuration file. Project configuration files are never auto-generated.
:return: the registered project, or None if not found
"""
# look for project by name
@@ -1016,7 +1203,7 @@ class SerenaConfig(SharedConfig, ModeSelectionDefinitionWithBaseModes):
for project in self.projects:
if project.matches_root_path(project_root_or_name):
return project
# no registered project found; auto-register if project configuration exists
# no registered project found; optionally auto-register if a project configuration already exists
if autoregister:
config_path = self.get_project_yml_location(project_root_or_name)
if os.path.isfile(config_path):
@@ -1035,18 +1222,19 @@ class SerenaConfig(SharedConfig, ModeSelectionDefinitionWithBaseModes):
def add_registered_project(self, registered_project: RegisteredProject) -> None:
"""
Adds a registered project, saving the configuration file.
Adds a registered project, persisting the updated project list
"""
self.projects.append(registered_project)
self.save()
self._persist_projects()
def add_project_from_path(self, project_root: Path | str) -> "Project":
def add_project_from_path(self, project_root: Path | str, asynchronous_autogen: bool = False) -> "Project":
"""
Add a new project to the Serena configuration from a given path, auto-generating the project
Adds a new project to the Serena configuration from a given path, auto-generating the project
with defaults if it does not exist.
Will raise a FileExistsError if a project already exists at the path.
:param project_root: the path to the project to add
:param asynchronous_autogen: whether to use asynchronous auto-generation for the project configuration
:return: the project that was added
"""
from ..project import Project
@@ -1063,7 +1251,8 @@ class SerenaConfig(SharedConfig, ModeSelectionDefinitionWithBaseModes):
f"Project with path {project_root} was already added with name '{already_registered_project.project_name}'."
)
project_config = ProjectConfig.load(project_root, serena_config=self, autogenerate=True)
autogen = ProjectConfigAutoGenerationMode.ASYNCHRONOUS if asynchronous_autogen else ProjectConfigAutoGenerationMode.SYNCHRONOUS
project_config = ProjectConfig.load(project_root, serena_config=self, autogen=autogen)
new_project = Project(
project_root=str(project_root),
@@ -1083,11 +1272,31 @@ class SerenaConfig(SharedConfig, ModeSelectionDefinitionWithBaseModes):
break
else:
raise ValueError(f"Project '{project_name}' not found in Serena configuration; valid project names: {self.project_names}")
self.save()
self._persist_projects()
def save(self) -> None:
def _persist_projects(self) -> None:
"""
Saves the configuration to the file from which it was loaded (if any)
Persists ONLY the registered-projects list, leaving every other setting at its on-disk value.
Project (de)registration can happen while a session is running with transient runtime overrides applied
to this in-memory instance (e.g. ``start-mcp-server --language-backend`` / ``--log-level``). A full
:meth:`save` would write those overrides back to the global config, silently clobbering the user's
settings. Instead we re-load the persisted config, copy in the current project list, and save that — so
only ``projects`` is ever mutated on disk.
"""
if self.config_file_path is None:
return
persisted = SerenaConfig.from_config_file()
persisted.projects = list(self.projects)
persisted._save()
def _save(self) -> None:
"""
Saves the full configuration to the file from which it was loaded (if any)
NOTE: This method is private, because it is not usually safe to save a configuration instance used
at runtime, because it often contains transient overrides (e.g. specified through the CLI)
that should never be persisted back to the configuration file.
"""
if self.config_file_path is None:
return
@@ -1198,3 +1407,27 @@ class SerenaConfig(SharedConfig, ModeSelectionDefinitionWithBaseModes):
from serena.tools import JetBrainsPluginClient
JetBrainsPluginClient.set_server_address(self.jetbrains_plugin_server_address)
def is_trusted_project_path(self, project_root: str | Path) -> bool:
"""
Checks if the given project root path matches any of the trusted project root patterns.
:param project_root: the path to the project root directory
:return: True if the project root is trusted, False otherwise
"""
project_root_str = str(project_root)
for pattern in self.trusted_project_path_patterns:
if glob_match(pattern, project_root_str):
return True
return False
def determine_language_backend(self, project_config: ProjectConfig | None = None, log_choice: bool = False):
language_backend = self.language_backend
if project_config and project_config.language_backend is not None:
language_backend = project_config.language_backend
if log_choice:
log.info(f"Using language backend as configured in project: {language_backend.name}")
else:
if log_choice:
log.info(f"Using language backend from global configuration: {language_backend.name}")
return language_backend
src/serena/dashboard.py
+4 -2
View File
@@ -502,7 +502,7 @@ class SerenaDashboardAPI:
}
# Get active modes
modes = self._agent.get_active_modes()
modes = self._agent.get_active_modes().get_modes(include_background_base_modes=False)
modes_info = [
{"name": mode.name, "description": mode.description, "path": SerenaAgentMode.get_path(mode.name, instance=mode)}
for mode in modes
@@ -789,7 +789,9 @@ class SerenaDashboardAPI:
# patch flask.cli.show_server to avoid printing the server info
from flask import cli
cli.show_server_banner = lambda *args, **kwargs: None
# ty cannot model reassigning a third-party module's function attribute (it rejects any
# replacement, even one with an identical signature), so the monkeypatch is suppressed here
cli.show_server_banner = lambda *args, **kwargs: None # ty: ignore[invalid-assignment]
self._app.run(host=self._host, port=port, debug=False, use_reloader=False, threaded=True)
return port
src/serena/generated/generated_prompt_factory.py
+1
View File
@@ -42,6 +42,7 @@ class PromptFactory(PromptFactoryBase):
context_system_prompt: Any,
global_memories_list: Any,
mode_system_prompts: Any,
tool_names: Any,
) -> str:
return self._render_prompt("system_prompt", locals())
src/serena/gui_log_viewer.py
+2 -3
View File
@@ -1,4 +1,3 @@
# mypy: ignore-errors
import logging
import queue
import sys
@@ -109,7 +108,7 @@ class GuiLogViewer:
if self.menubar is not None:
dashboard_menu = tk.Menu(self.menubar, tearoff=0)
dashboard_menu.add_command(label="Copy URL", command=copy_url) # type: ignore
dashboard_menu.add_command(label="Copy URL", command=copy_url)
self.menubar.add_cascade(label="Dashboard", menu=dashboard_menu)
def add_log(self, message):
@@ -303,7 +302,7 @@ class GuiLogViewer:
if self.mode == "dashboard":
self.menubar = tk.Menu(self.root)
server_menu = tk.Menu(self.menubar, tearoff=0)
server_menu.add_command(label="Shutdown", command=self._shutdown_server) # type: ignore
server_menu.add_command(label="Shutdown", command=self._shutdown_server)
self.menubar.add_cascade(label="Server", menu=server_menu)
self.root.config(menu=self.menubar)
src/serena/hooks.py
+8 -3
View File
@@ -22,6 +22,7 @@ class HookClient(Enum):
"""The client application that triggered the hook."""
CLAUDE_CODE = "claude-code"
CODEBUDDY = "codebuddy"
VSCODE = "vscode"
CODEX = "codex"
@@ -70,7 +71,11 @@ class PreToolUseHook(Hook, ABC):
if not _tool_name:
raise ValueError("Tool name is required in the hook input data")
self._tool_name = _tool_name
self._tool_input: dict | None = self._input_data.get("tool_input") or self._input_data.get("toolInput")
raw_tool_input = self._input_data.get("tool_input") or self._input_data.get("toolInput")
# TODO: some agents, like copilot CLI, can send a string as value for raw_tool_input
# Example: "tool_input":"*** Begin Patch\n*** Add File: /Users/acbdef/.copilot/session-state/08a961db-02f0-4c7c-b783-1e9818290292/files/hook-tool-test-3.txt\n+third edit tool test\n*** End Patch\n"
# We currently don't parse such tool input and hence don't react to it in hooks
self._tool_input: dict | None = raw_tool_input if isinstance(raw_tool_input, dict) else None
# only relevant in claude code at the moment, (not all events include this field; default to empty string)
raw_permission_mode = self._input_data.get("permission_mode") or self._input_data.get("permissionMode") or ""
@@ -361,7 +366,7 @@ class PreToolUseRemindAboutSymbolicToolsHook(PreToolUseHook):
self._file_path = str(file_path).strip() or None
def is_grep_call(self) -> bool:
if self._client == HookClient.CLAUDE_CODE:
if self._client in (HookClient.CLAUDE_CODE, HookClient.CODEBUDDY):
return self._tool_name == "grep" or "search_for_pattern" in self._tool_name
if self._client == HookClient.CODEX and self._is_shell_command_call():
return self._command_name in self._GREP_SHELL_COMMANDS
@@ -369,7 +374,7 @@ class PreToolUseRemindAboutSymbolicToolsHook(PreToolUseHook):
return "grep" in self._tool_name
def is_read_call(self) -> bool:
if self._client == HookClient.CLAUDE_CODE:
if self._client in (HookClient.CLAUDE_CODE, HookClient.CODEBUDDY):
return self._tool_name == "read" or "read_file" in self._tool_name
if self._client == HookClient.CODEX and self._is_shell_command_call():
return self._command_name in self._READ_SHELL_COMMANDS
src/serena/jetbrains/jetbrains_plugin_client.py
+38 -9
View File
@@ -225,7 +225,7 @@ class JetBrainsPluginClient(ToStringMixin):
return ["_port", "project_root", "_plugin_version"]
@classmethod
def from_project(cls, project: Project, log_warning: bool = True) -> Self:
def from_project(cls, project: Project, log_warning: bool = True) -> "JetBrainsPluginClient":
resolved_path = Path(project.project_root).resolve()
if cls._last_port is not None:
@@ -303,7 +303,15 @@ class JetBrainsPluginClient(ToStringMixin):
f"{self._plugin_version}. Ask the user to update the plugin!"
)
def _make_request(self, method: str, endpoint: str, data: Optional[dict] = None) -> dict[str, Any]:
def _make_request(self, method: str, endpoint: str, data: Optional[dict] = None, pythonify: bool = True) -> dict[str, Any]:
"""
:param method: the HTTP method to use ("GET" or "POST")
:param endpoint: the endpoint to call (e.g., "/findSymbol")
:param data: the data to send in the request body (for POST requests)
:param pythonify: whether to recursively "pythonify" the response object, converting all keys to snake_case.
This must not be enabled if *any* key contains variable data rather (and therefore isn't a well-defined DTO structure).
:return: the response as a dictionary
"""
url = f"{self._base_url}{endpoint}"
response: Response | None = None
@@ -311,8 +319,8 @@ class JetBrainsPluginClient(ToStringMixin):
if method.upper() == "GET":
response = self._session.get(url, timeout=self._timeout)
elif method.upper() == "POST":
json_data = json.dumps(data) if data else None
response = self._session.post(url, data=json_data, timeout=self._timeout)
data_dict = data if data is not None else {}
response = self._session.post(url, data=json.dumps(data_dict), timeout=self._timeout)
else:
raise ValueError(f"Unsupported HTTP method: {method}")
@@ -320,7 +328,11 @@ class JetBrainsPluginClient(ToStringMixin):
# Try to parse JSON response
try:
return self._pythonify_response(response.json())
response_json = response.json()
if pythonify:
return self._pythonify_response(response_json)
else:
return response_json
except json.JSONDecodeError:
# If response is not JSON, return raw text
return {"response": response.text}
@@ -355,7 +367,7 @@ class JetBrainsPluginClient(ToStringMixin):
"""
to_snake_case = lambda s: "".join(["_" + c.lower() if c.isupper() else c for c in s])
def convert(x): # type: ignore
def convert(x):
if isinstance(x, dict):
return {to_snake_case(k): convert(v) for k, v in x.items()}
elif isinstance(x, list):
@@ -417,6 +429,8 @@ class JetBrainsPluginClient(ToStringMixin):
:param include_location: whether to include symbol location information
:param search_deps: whether to also search in dependencies
"""
if "*" in name_path:
self._require_version_at_least(2023, 3, 1)
request_data = {
"namePath": name_path,
"relativePath": relative_path,
@@ -674,7 +688,7 @@ class JetBrainsPluginClient(ToStringMixin):
inspection_names: list[str] | None = None,
start_line: int | None = None,
end_line: int | None = None,
) -> jb.RunInspectionsResponse:
) -> dict:
"""
Runs IDE inspections on the given file and returns the results.
@@ -683,7 +697,11 @@ class JetBrainsPluginClient(ToStringMixin):
:param inspection_names: optional list of specific inspection names to run
:param start_line: optional start line to restrict the inspection range
:param end_line: optional end line to restrict the inspection range
:return: the inspection results as a dictionary.
NOTE: The response is currently *not* a well-defined DTO, because it stores variable data in keys.
Consequently, the `pythonify` option is disabled for this request, and the response is returned as-is.
"""
self._require_version_at_least(2023, 2, 14)
request_data: dict[str, Any] = {
"relativePath": relative_path,
}
@@ -695,7 +713,7 @@ class JetBrainsPluginClient(ToStringMixin):
request_data["startLine"] = start_line
if end_line is not None:
request_data["endLine"] = end_line
return cast(jb.RunInspectionsResponse, self._make_request("POST", "/runInspectionsOnFile", request_data))
return self._make_request("POST", "/runInspectionsOnFile", request_data, pythonify=False)
def list_inspections(
self,
@@ -708,6 +726,7 @@ class JetBrainsPluginClient(ToStringMixin):
:param language: optional language to filter inspections by (e.g. "Java", "Python")
:param group_path_contains: optional substring to filter inspection group paths
"""
self._require_version_at_least(2023, 2, 14)
request_data: dict[str, Any] = {}
if language is not None:
request_data["language"] = language
@@ -715,11 +734,21 @@ class JetBrainsPluginClient(ToStringMixin):
request_data["groupPathContains"] = group_path_contains
return cast(jb.ListInspectionsResponse, self._make_request("POST", "/listInspections", request_data))
def read_file(self, relative_path: str) -> str:
self._require_version_at_least(2023, 3, 3)
request_data = {
"relativePath": relative_path,
}
response = self._make_request("POST", "/readFile", request_data)
if "content" not in response:
raise PluginServerError(f"Unexpected response from readFile: {response}")
return response["content"]
def close(self) -> None:
self._session.close()
def __enter__(self) -> Self:
return self
def __exit__(self, exc_type, exc_val, exc_tb): # type: ignore
def __exit__(self, exc_type, exc_val, exc_tb):
self.close()
src/serena/jetbrains/jetbrains_types.py
+8 -16
View File
@@ -6,6 +6,14 @@ Prefix used for in relative paths of symbols that are from external libraries (i
"""
def is_external_path(relative_path: str):
"""
:param relative_path: a relative path (e.g., from a symbol's `relative_path` field)
:return: whether the path is an external path (i.e., from a library, not the user's codebase)
"""
return relative_path.startswith(JB_EXTERNAL_FILE_PREFIX)
class PluginStatusDTO(TypedDict):
project_root: str
plugin_version: str
@@ -91,22 +99,6 @@ class TypeHierarchyResponse(TypedDict):
num_levels_not_included: NotRequired[int]
class InspectionResultDTO(TypedDict):
inspection_name: str
severity: str
message: str
relative_path: str
start_line: int
end_line: int
start_col: NotRequired[int]
end_col: NotRequired[int]
quick_fix_available: NotRequired[bool]
class RunInspectionsResponse(TypedDict):
inspections: list[InspectionResultDTO]
class InspectionInfoDTO(TypedDict):
name: str
group_path: str
src/serena/ls_manager.py
+123 -9
View File
@@ -2,14 +2,20 @@ import logging
import os.path
import threading
from collections.abc import Iterator
from pathlib import Path
from typing import TYPE_CHECKING
from sensai.util.logging import LogTime
from serena.config.serena_config import SerenaPaths
from serena.config.serena_config import ProjectConfig, SerenaPaths
from solidlsp import SolidLanguageServer
from solidlsp.ls_config import Language, LanguageServerConfig
from solidlsp.lsp_protocol_handler.lsp_types import DidChangeWatchedFilesParams, FileChangeType, FileEvent
from solidlsp.settings import SolidLSPSettings
if TYPE_CHECKING:
from .project import Project
log = logging.getLogger(__name__)
@@ -22,25 +28,27 @@ class LanguageServerFactory:
def __init__(
self,
project_root: str,
project_config: ProjectConfig,
project_data_path: str,
encoding: str,
ignored_patterns: list[str],
ls_timeout: float | None = None,
ls_specific_settings: dict | None = None,
additional_workspace_folders: list[str] | None = None,
trace_lsp_communication: bool = False,
):
self.project_root = project_root
self.project_config = project_config
self.project_data_path = project_data_path
self.encoding = encoding
self.ignored_patterns = ignored_patterns
self.ls_timeout = ls_timeout
self.ls_specific_settings = ls_specific_settings
self.additional_workspace_folders = additional_workspace_folders or []
self.trace_lsp_communication = trace_lsp_communication
def create_language_server(self, language: Language) -> SolidLanguageServer:
ls_config = LanguageServerConfig(
workspace_folders=self.project_config.ls_workspace_folders,
additional_workspace_folders=self.project_config.ls_additional_workspace_folders,
code_language=language,
ignored_paths=self.ignored_patterns,
trace_lsp_communication=self.trace_lsp_communication,
@@ -56,7 +64,6 @@ class LanguageServerFactory:
solidlsp_dir=SerenaPaths().serena_user_home_dir,
project_data_path=self.project_data_path,
ls_specific_settings=self.ls_specific_settings or {},
additional_workspace_folders=self.additional_workspace_folders,
),
)
@@ -67,9 +74,7 @@ class LanguageServerManager:
"""
def __init__(
self,
language_servers: dict[Language, SolidLanguageServer],
language_server_factory: LanguageServerFactory | None = None,
self, language_servers: dict[Language, SolidLanguageServer], language_server_factory: LanguageServerFactory, project: "Project"
) -> None:
"""
:param language_servers: a mapping from language to language server; the servers are assumed to be already started.
@@ -80,6 +85,7 @@ class LanguageServerManager:
"""
self._language_servers = language_servers
self._language_server_factory = language_server_factory
self._file_change_notifier = LanguageServerFileChangeNotifier(project, self)
@property
def _default_language_server(self) -> SolidLanguageServer:
@@ -88,13 +94,14 @@ class LanguageServerManager:
return next(iter(self._language_servers.values()))
@staticmethod
def from_languages(languages: list[Language], factory: LanguageServerFactory) -> "LanguageServerManager":
def from_languages(languages: list[Language], factory: LanguageServerFactory, project: "Project") -> "LanguageServerManager":
"""
Creates a manager with language servers for the given languages using the given factory.
The language servers are started in parallel threads.
:param languages: the languages for which to spawn language servers
:param factory: the factory for language server creation
:param project: the project for which the language servers are created
:return: the instance
"""
@@ -144,7 +151,7 @@ class LanguageServerManager:
failure_messages = "\n".join([f"{lang.value}: {e}" for lang, e in exceptions.items()])
raise LanguageServerManagerInitialisationError(f"Failed to start {len(exceptions)} language server(s):\n{failure_messages}")
return LanguageServerManager(language_servers, factory)
return LanguageServerManager(language_servers, factory, project)
def _ensure_functional_ls(self, ls: SolidLanguageServer) -> SolidLanguageServer:
if not ls.is_running():
@@ -253,3 +260,110 @@ class LanguageServerManager:
def has_suitable_ls_for_file(self, relative_file_path: str) -> bool:
return self._get_suitable_language_server(relative_file_path) is not None
def sync_file_system_changes(self) -> int:
"""
Polls the file system for changes to source files and notifies the language servers of any changes
(particularly changes that happened outside of Serena's own file tools, which are not covered by
the notifications sent by those tools/CodeEditor).
:return: the number of individual file change events detected (0 if nothing changed).
"""
log.info("Polling file system for changes to source files ...")
num_changes = self._file_change_notifier.poll_and_notify()
log.info(f"File system polling complete; {num_changes} change events sent to language servers.")
return num_changes
class LanguageServerFileChangeNotifier:
"""
Detects changes to source files on disk and notifies language servers of those changes.
"""
def __init__(self, project: "Project", language_server_manager: LanguageServerManager, initial_poll: bool = True) -> None:
self._project = project
self._language_server_manager = language_server_manager
self._freshness_last_seen_mtimes: dict[str, float] | None = None
self._freshness_lock = threading.Lock()
if initial_poll:
# Establish the baseline for the first poll; no notifications are sent on the first call.
with LogTime("Initialising file change notifier (polling for baseline)"):
self.poll_and_notify()
def poll_and_notify(self) -> int:
"""
Detects source files that were changed, created or deleted on disk since the last call
and notifies every language server managed for this project via the LSP
``workspace/didChangeWatchedFiles`` notification.
This exists because Serena's own file and symbol tools notify the language server inline
(via didOpen/didChange/didClose) when they edit a file, but edits made through any other
channel (another editor, a second agent, a git checkout, a build step) are otherwise
invisible to a warm language server, causing symbolic queries to answer from a stale index.
The set of files considered is exactly the set Serena itself tracks (see
:meth:`gather_source_files`), so no separate file-discovery logic has to be kept in sync.
The dominant cost is the directory walk plus one ``os.stat`` per tracked file; this is
intended to be called before symbolic tool invocations rather than on a timer.
:return: the number of change events sent (0 if nothing changed, if no language server is
running yet, or on the first call, which only establishes the baseline).
"""
current: dict[str, float] = {}
for rel_path in self._project.gather_source_files():
try:
current[rel_path] = os.stat(os.path.join(self._project.project_root, rel_path)).st_mtime
except OSError:
continue
# Read-diff-swap under the lock only; the filesystem walk above and the LSP notifications
# below stay outside it so concurrent callers do not serialize on I/O.
with self._freshness_lock:
previous = self._freshness_last_seen_mtimes
self._freshness_last_seen_mtimes = current
if previous is None:
return 0
# compute the set of individual events (created, changed, deleted)
events: list[tuple[str, FileChangeType]] = []
for rel_path, mtime in current.items():
prev_mtime = previous.get(rel_path)
if prev_mtime is None:
events.append((rel_path, FileChangeType.Created))
elif mtime > prev_mtime:
events.append((rel_path, FileChangeType.Changed))
events.extend((rel_path, FileChangeType.Deleted) for rel_path in previous if rel_path not in current)
if not events:
return 0
# create the change didChangeWatchedFiles notification
changes: list[FileEvent] = [
{"uri": Path(self._project.project_root, rel_path).resolve().as_uri(), "type": change_type} for rel_path, change_type in events
]
params: DidChangeWatchedFilesParams = {"changes": changes}
created_paths = [rel_path for rel_path, change_type in events if change_type == FileChangeType.Created]
for ls in self._language_server_manager.iter_language_servers():
# send the didChangeWatchedFiles notification to the language server
try:
ls.server.notify.did_change_watched_files(params)
except Exception as e:
log.error("Failed to notify language server of watched file changes", exc_info=e)
# A didChangeWatchedFiles(Created) notification alone is not enough for every backend
# (observed with pyright) to fold a brand-new file into its cross-file reference graph;
# an open/close cycle forces the parse+bind that Serena's own file tools trigger via
# SolidLanguageServer.open_file().
for rel_path in created_paths:
if ls.is_ignored_path(rel_path, ignore_unsupported_files=True):
continue
try:
with ls.open_file(rel_path):
pass
except Exception as e:
log.error(f"Failed to refresh newly created file {rel_path!r} in language server", exc_info=e)
return len(events)
src/serena/mcp.py
+117 -76
View File
@@ -11,8 +11,11 @@ from typing import Any, Literal, cast
import docstring_parser
from mcp.server.fastmcp import server
from mcp.server.fastmcp.server import FastMCP, Settings
from mcp.server.fastmcp.tools.base import Tool as MCPTool
from mcp.server.fastmcp.exceptions import ToolError
from mcp.server.fastmcp.server import Context, FastMCP, Settings
from mcp.server.fastmcp.tools.base import Tool as FastMCPTool
from mcp.server.session import ServerSessionT
from mcp.shared.context import LifespanContextT, RequestT
from mcp.types import ToolAnnotations
from pydantic_settings import SettingsConfigDict
from sensai.util import logging
@@ -24,14 +27,14 @@ from serena.agent import (
from serena.config.context_mode import SerenaAgentContext
from serena.config.serena_config import LanguageBackend, ModeSelectionDefinition
from serena.constants import DEFAULT_CONTEXT, SERENA_LOG_FORMAT
from serena.tools import Tool
from serena.tools import Tool, ToolCallError
from serena.util.exception import show_fatal_exception_safe
from serena.util.logging import MemoryLogHandler
log = logging.getLogger(__name__)
def configure_logging(*args, **kwargs) -> None: # type: ignore
def configure_logging(*args, **kwargs) -> None:
# We only do something here if logging has not yet been configured.
# Normally, logging is configured in the MCP server startup script.
if not logging.is_enabled():
@@ -47,6 +50,98 @@ class SerenaMCPRequestContext:
agent: SerenaAgent
class SerenaFastMCPTool(FastMCPTool):
def __init__(self, tool: Tool, openai_tool_compatible: bool, structured_output: bool | None):
"""
:param tool: the Serena tool
:param openai_tool_compatible: whether to process the tool schema to be compatible with OpenAI tools
(doesn't accept integer, needs number instead, etc.). This allows using Serena MCP within Codex.
:param structured_output: whether to use structured output for the tool (None = auto)
"""
func_name = tool.get_name()
func_doc = tool.get_apply_docstring() or ""
func_arg_metadata = tool.get_apply_fn_metadata(structured_output=structured_output)
is_async = False
parameters = func_arg_metadata.arg_model.model_json_schema()
if openai_tool_compatible:
parameters = SerenaMCPFactory._sanitize_for_openai_tools(parameters)
docstring = docstring_parser.parse(func_doc)
# Mount the tool description as a combination of the docstring description and
# the return value description, if it exists.
overridden_description = tool.agent.get_context().tool_description_overrides.get(func_name, None)
if overridden_description is not None:
func_doc = overridden_description
elif docstring.description:
func_doc = docstring.description
else:
func_doc = ""
func_doc = func_doc.strip().strip(".")
if func_doc:
func_doc += "."
if docstring.returns and (docstring_returns_descr := docstring.returns.description):
# Only add a space before "Returns" if func_doc is not empty
prefix = " " if func_doc else ""
func_doc = f"{func_doc}{prefix}Returns {docstring_returns_descr.strip().strip('.')}."
# Parse the parameter descriptions from the docstring and add pass its description
# to the parameter schema.
docstring_params = {param.arg_name: param for param in docstring.params}
parameters_properties: dict[str, dict[str, Any]] = parameters["properties"]
for parameter, properties in parameters_properties.items():
if (param_doc := docstring_params.get(parameter)) and param_doc.description:
param_desc = f"{param_doc.description.strip().strip('.') + '.'}"
properties["description"] = param_desc[0].upper() + param_desc[1:]
def execute_fn(**kwargs) -> str:
try:
return tool.apply_ex(log_call=True, catch_exceptions=False, **kwargs)
except ToolCallError as e:
raise ToolError(e.get_error_message()) from e
# Generate human-readable title from snake_case tool name
tool_title = " ".join(word.capitalize() for word in func_name.split("_"))
# Create annotations with appropriate hints based on tool capabilities
can_edit = tool.can_edit()
annotations = ToolAnnotations(
title=tool_title,
readOnlyHint=not can_edit,
destructiveHint=can_edit,
)
super().__init__(
fn=execute_fn,
name=func_name,
description=func_doc,
parameters=parameters,
fn_metadata=func_arg_metadata,
is_async=is_async,
# keep the value in sync with the kwarg name in Tool.apply_ex. The mcp sdk uses reflection to infer this
# when the tool is constructed via from_function (which is a bit crazy IMO, but well...)
context_kwarg="mcp_ctx",
annotations=annotations,
title=tool_title,
)
self._param_aliases = tool.get_param_aliases()
async def run(
self,
arguments: dict[str, Any],
context: Context[ServerSessionT, LifespanContextT, RequestT] | None = None,
convert_result: bool = False,
) -> Any:
# apply parameter aliases
for param_alias, param_name in self._param_aliases.items():
if param_alias in arguments and param_name not in arguments:
arguments[param_name] = arguments.pop(param_alias)
return await super().run(arguments, context, convert_result)
class SerenaMCPFactory:
"""
Factory for the creation of the Serena MCP server with an associated SerenaAgent.
@@ -87,7 +182,7 @@ class SerenaMCPFactory:
"""
s = deepcopy(schema)
def walk(node): # type: ignore
def walk(node):
if not isinstance(node, dict):
# lists get handled by parent calls
return node
@@ -181,90 +276,34 @@ class SerenaMCPFactory:
return walk(s)
@staticmethod
def make_mcp_tool(tool: Tool, openai_tool_compatible: bool = True) -> MCPTool:
def make_mcp_tool(tool: Tool, openai_tool_compatible: bool = True, structured_output: bool | None = None) -> SerenaFastMCPTool:
"""
Create an MCP tool from a Serena Tool instance.
Creates an MCP tool from a Serena Tool instance.
:param tool: The Serena Tool instance to convert.
:param tool: the Serena Tool instance to convert.
:param openai_tool_compatible: whether to process the tool schema to be compatible with OpenAI tools
(doesn't accept integer, needs number instead, etc.). This allows using Serena MCP within codex.
:param structured_output: whether to use structured output for the tool (None = auto)
"""
func_name = tool.get_name()
func_doc = tool.get_apply_docstring() or ""
func_arg_metadata = tool.get_apply_fn_metadata()
is_async = False
parameters = func_arg_metadata.arg_model.model_json_schema()
if openai_tool_compatible:
parameters = SerenaMCPFactory._sanitize_for_openai_tools(parameters)
docstring = docstring_parser.parse(func_doc)
# Mount the tool description as a combination of the docstring description and
# the return value description, if it exists.
overridden_description = tool.agent.get_context().tool_description_overrides.get(func_name, None)
if overridden_description is not None:
func_doc = overridden_description
elif docstring.description:
func_doc = docstring.description
else:
func_doc = ""
func_doc = func_doc.strip().strip(".")
if func_doc:
func_doc += "."
if docstring.returns and (docstring_returns_descr := docstring.returns.description):
# Only add a space before "Returns" if func_doc is not empty
prefix = " " if func_doc else ""
func_doc = f"{func_doc}{prefix}Returns {docstring_returns_descr.strip().strip('.')}."
# Parse the parameter descriptions from the docstring and add pass its description
# to the parameter schema.
docstring_params = {param.arg_name: param for param in docstring.params}
parameters_properties: dict[str, dict[str, Any]] = parameters["properties"]
for parameter, properties in parameters_properties.items():
if (param_doc := docstring_params.get(parameter)) and param_doc.description:
param_desc = f"{param_doc.description.strip().strip('.') + '.'}"
properties["description"] = param_desc[0].upper() + param_desc[1:]
def execute_fn(**kwargs) -> str: # type: ignore
return tool.apply_ex(log_call=True, catch_exceptions=True, **kwargs)
# Generate human-readable title from snake_case tool name
tool_title = " ".join(word.capitalize() for word in func_name.split("_"))
# Create annotations with appropriate hints based on tool capabilities
can_edit = tool.can_edit()
annotations = ToolAnnotations(
title=tool_title,
readOnlyHint=not can_edit,
destructiveHint=can_edit,
)
return MCPTool(
fn=execute_fn,
name=func_name,
description=func_doc,
parameters=parameters,
fn_metadata=func_arg_metadata,
is_async=is_async,
# keep the value in sync with the kwarg name in Tool.apply_ex. The mcp sdk uses reflection to infer this
# when the tool is constructed via from_function (which is a bit crazy IMO, but well...)
context_kwarg="mcp_ctx",
annotations=annotations,
title=tool_title,
)
return SerenaFastMCPTool(tool, openai_tool_compatible=openai_tool_compatible, structured_output=structured_output)
def _iter_tools(self) -> Iterator[Tool]:
assert self.agent is not None
yield from self.agent.get_exposed_tool_instances()
# noinspection PyProtectedMember
def _set_mcp_tools(self, mcp: FastMCP, openai_tool_compatible: bool = False) -> None:
"""Update the tools in the MCP server"""
def _set_mcp_tools(self, mcp: FastMCP, openai_tool_compatible: bool, structured_output: bool | None) -> None:
"""
Update the tools in the MCP server
:param mcp: The MCP server to update
:param openai_tool_compatible: whether to process the tool schema to be compatible with OpenAI tools
:param structured_output: whether to use structured output for the tools (None = auto)
"""
if mcp is not None:
mcp._tool_manager._tools = {}
for tool in self._iter_tools():
mcp_tool = self.make_mcp_tool(tool, openai_tool_compatible=openai_tool_compatible)
mcp_tool = self.make_mcp_tool(tool, openai_tool_compatible=openai_tool_compatible, structured_output=structured_output)
mcp._tool_manager._tools[tool.get_name()] = mcp_tool
log.info(f"Starting MCP server with {len(mcp._tool_manager._tools)} tools: {list(mcp._tool_manager._tools.keys())}")
@@ -358,7 +397,9 @@ class SerenaMCPFactory:
:param mcp_server: the MCP server instance to configure
"""
openai_tool_compatible = self.context.name in ["chatgpt", "codex", "oaicompat-agent"]
self._set_mcp_tools(mcp_server, openai_tool_compatible=openai_tool_compatible)
assert self.agent is not None
context = self.agent.get_context()
self._set_mcp_tools(mcp_server, openai_tool_compatible=openai_tool_compatible, structured_output=context.structured_tool_output)
log.info("MCP server lifetime setup complete")
try:
yield
src/serena/memories/memory_manager.py
+52 -21
View File
@@ -2,7 +2,7 @@ import logging
import os
import re
import shutil
from collections.abc import Sequence
from collections.abc import Iterator, Sequence
from pathlib import Path
from typing import Literal
@@ -147,11 +147,41 @@ class MemoryManager:
# use a callable replacement to avoid backreference interpretation of characters in ref_new
return re.subn(pattern, lambda _m: ref_new, content)
def _resolve_memory_path(self, base_dir: Path, parts: Sequence[str]) -> Path:
"""
Builds the ``*.md`` path for ``parts`` under ``base_dir``, creating any parent
subdirectories, and guarantees the result stays inside ``base_dir``.
The containment check is a defense-in-depth backstop for :meth:`get_memory_file_path`'s
up-front segment validation: even if a crafted name slipped through, the built path must
never escape the memories sandbox (which would let an agent read/write/delete arbitrary
files). The check runs *before* any directory is created, so a rejected name cannot leave
stray directories behind either. It is deliberately *lexical* (``normpath``, no symlink
resolution): directory symlinks placed inside the memories folder are a supported way to
share memories (e.g. a monorepo symlinking each submodule's memory dir), and those must
keep resolving to their targets at I/O time.
"""
filename = f"{parts[-1]}.md"
subdir = base_dir if len(parts) == 1 else base_dir.joinpath(*parts[:-1])
candidate = subdir / filename
base_norm = Path(os.path.normpath(base_dir))
if not Path(os.path.normpath(candidate)).is_relative_to(base_norm):
raise ValueError(f"Memory name resolves outside the memories directory. Got: {'/'.join(parts)}")
subdir.mkdir(parents=True, exist_ok=True)
return candidate
def get_memory_file_path(self, name: str) -> Path:
name = self._sanitize_name(name)
parts = name.split("/")
if ".." in parts:
raise ValueError(f"Memory name cannot contain '..' segments for security reasons. Got: {name}")
raise ValueError(f"Memory name cannot contain '..' segments. Got: {name}")
# Reject absolute names and empty path segments: pathlib discards the base directory when
# joined with an absolute path (e.g. "/etc/cron.d/backdoor" would reset to "/etc/cron.d"),
# letting a memory name escape the sandbox. A leading "/" produces an empty first segment.
if os.path.isabs(name) or "" in parts:
raise ValueError(f"Memory name cannot be absolute or contain empty path segments. Got: {name}")
if self._is_global(name):
if name == self.GLOBAL_TOPIC:
@@ -160,26 +190,11 @@ class MemoryManager:
)
# Strip "global/" prefix and resolve against global dir
sub_name = name[len(self.GLOBAL_TOPIC) + 1 :]
parts = sub_name.split("/")
filename = f"{parts[-1]}.md"
if len(parts) > 1:
subdir = self._global_memory_dir / "/".join(parts[:-1])
subdir.mkdir(parents=True, exist_ok=True)
return subdir / filename
return self._global_memory_dir / filename
return self._resolve_memory_path(self._global_memory_dir, sub_name.split("/"))
# Project-local memory
assert self._project_memory_dir is not None, "Project dir was not passed at initialization"
filename = f"{parts[-1]}.md"
if len(parts) > 1:
# Create subdirectory path
subdir = self._project_memory_dir / "/".join(parts[:-1])
subdir.mkdir(parents=True, exist_ok=True)
return subdir / filename
return self._project_memory_dir / filename
return self._resolve_memory_path(self._project_memory_dir, parts)
def _check_write_access(self, name: str, is_tool_context: bool) -> None:
# in tool context, memories can be read-only
@@ -191,7 +206,7 @@ class MemoryManager:
self._check_not_ignored(name)
memory_file_path = self.get_memory_file_path(name)
if not memory_file_path.exists():
return f"Memory file {name} not found, consider creating it with the `write_memory` tool if you need it."
raise FileNotFoundError(f"Memory named '{name}' not found")
with open(memory_file_path, encoding=self._encoding) as f:
return f.read()
@@ -233,11 +248,27 @@ class MemoryManager:
def get_full_list(self) -> list[str]:
return sorted(self.memories + self.read_only_memories)
@staticmethod
def _iter_memory_files(search_dir: Path) -> Iterator[Path]:
"""Yields every ``*.md`` file under ``search_dir``, descending into symlinked directories.
Directory symlinks must be followed so that memories shared via symlink are discovered
(e.g. a monorepo whose ``.serena/memories`` folder symlinks each submodule's memory
directory, making them addressable as ``<submodule>/<name>``). ``Path.rglob`` only
follows symlinked directories from Python 3.13 onwards, via ``recurse_symlinks``; on
3.11/3.12 it silently skips them, so we fall back to ``os.walk(followlinks=True)`` there.
``rglob`` is preferred when available as it is faster than ``os.walk``.
"""
for root, _dirs, files in os.walk(search_dir, followlinks=True):
for filename in files:
if filename.endswith(".md"):
yield Path(root) / filename
def _list_memories(self, search_dir: Path, base_dir: Path, prefix: str = "") -> MemoriesList:
result = self.MemoriesList()
if not search_dir.exists():
return result
for md_file in search_dir.rglob("*.md"):
for md_file in self._iter_memory_files(search_dir):
rel = str(md_file.relative_to(base_dir).with_suffix("")).replace(os.sep, "/")
memory_name = prefix + rel
if self._is_ignored_memory(memory_name):
src/serena/project.py
+104 -33
View File
@@ -9,16 +9,18 @@ from sensai.util.logging import LogTime
from sensai.util.string import TextBuilder, ToStringMixin
from serena.config.serena_config import (
LanguageBackend,
ProjectConfig,
ProjectConfigAutoGenerationMode,
SerenaConfig,
)
from serena.ls_manager import LanguageServerFactory, LanguageServerManager
from serena.memories.memory_manager import MemoryManager
from serena.util.file_system import GitignoreParser, match_path
from serena.util.file_proxy import FileCollection, FileProxy
from serena.util.file_system import GitignoreParser, match_path, scan_directory
from serena.util.text_utils import MatchedConsecutiveLines, search_files
from solidlsp import SolidLanguageServer
from solidlsp.ls_config import Language
from solidlsp.ls_utils import FileUtils
if TYPE_CHECKING:
from serena.agent import SerenaAgent
@@ -120,18 +122,28 @@ class Project(ToStringMixin):
def project_name(self) -> str:
return self.project_config.project_name
@property
def language_backend(self) -> LanguageBackend:
# The backend configuration is fundamentally owned by the agent, so it takes
# precedence. (Note: The agent does not necessary honour the project's choice,
# as it may be invalid.)
if self._agent is not None:
return self._agent.get_language_backend()
else:
return self.serena_config.determine_language_backend(self.project_config)
@classmethod
def load(
cls,
project_root: str | Path,
serena_config: "SerenaConfig",
autogenerate: bool = True,
autogen: ProjectConfigAutoGenerationMode = ProjectConfigAutoGenerationMode.SYNCHRONOUS,
) -> "Project":
assert serena_config is not None
project_root = Path(project_root).resolve()
if not project_root.exists():
raise FileNotFoundError(f"Project root not found: {project_root}")
project_config = ProjectConfig.load(project_root, serena_config=serena_config, autogenerate=autogenerate)
project_config = ProjectConfig.load(project_root, serena_config=serena_config, autogen=autogen)
return Project(project_root=str(project_root), project_config=project_config, serena_config=serena_config)
def save_config(self) -> None:
@@ -146,15 +158,23 @@ class Project(ToStringMixin):
def path_to_project_yml(self) -> str:
return self.serena_config.get_project_yml_location(self.project_root)
def is_trusted(self) -> bool:
"""
Checks whether the project is trusted, based on the global configuration.
:return: True if the project is trusted, False otherwise
"""
return self.serena_config.is_trusted_project_path(self.project_root)
def read_file(self, relative_path: str) -> str:
"""
Reads a file relative to the project root.
Reads a project file.
:param relative_path: the path to the file relative to the project root
:param relative_path: the path to the file relative to the project root or an external
path token like "<ext:FileUtil.class|472e0a13>"
:return: the content of the file
"""
abs_path = Path(self.project_root) / relative_path
return FileUtils.read_file(str(abs_path), self.project_config.encoding)
return FileProxy.from_project_relative_path(self, relative_path).get_contents()
@property
def _ignore_spec(self) -> pathspec.PathSpec:
@@ -211,17 +231,20 @@ class Project(ToStringMixin):
log.debug(f"Path {abs_path} does not exist, skipping ignore check")
return False
# Check file extension if it's a file
is_file = os.path.isfile(abs_path)
if is_file and ignore_non_source_files:
is_file_in_supported_language = False
for language in self.project_config.languages:
fn_matcher = language.get_source_fn_matcher()
if fn_matcher.is_relevant_filename(abs_path):
is_file_in_supported_language = True
break
if not is_file_in_supported_language:
return True
# check code file restriction (depending on backend)
if ignore_non_source_files:
# apply restriction only for LSP backend, which enumerates known languages
# and therefore can determine whether a file is a source file or not
if self.language_backend.is_lsp():
if os.path.isfile(abs_path):
is_file_in_supported_language = False
for language in self.project_config.languages:
fn_matcher = language.get_source_fn_matcher()
if fn_matcher.is_relevant_filename(abs_path):
is_file_in_supported_language = True
break
if not is_file_in_supported_language:
return True
# Create normalized path for consistent handling
rel_path = Path(relative_path)
@@ -285,17 +308,18 @@ class Project(ToStringMixin):
def validate_relative_path(self, relative_path: str, require_not_ignored: bool = False) -> None:
"""
Validates that the given relative path is safe to read or edit,
meaning it's inside the project directory and not ignored.
Non-existent paths are allowed (not considered ignored) to support
editing tools that create new files.
Validates that the given relative path is within the project directory
(and, optionally, not ignored according to the project's ignore settings),
raising a ValueError if the validation fails.
:param relative_path: the path to validate, relative to the project root
:param require_not_ignored: if True, the path must not be ignored according to the project's ignore settings
"""
if FileProxy.is_external_path(relative_path):
return
if not self.is_path_in_project(relative_path):
raise ValueError(f"{relative_path=} points to path outside of the repository root; cannot access for safety reasons")
raise ValueError(f"{relative_path=} points outside the project root ({self.project_root})")
if require_not_ignored:
if self.is_ignored_path(relative_path):
@@ -338,7 +362,35 @@ class Project(ToStringMixin):
)
return rel_file_paths
def search_source_files_for_pattern(
def _create_file_collection(self, relative_path: str, code_files_only: bool) -> FileCollection:
if FileProxy.is_external_path(relative_path):
# single external path: create appropriate proxy
file_collection = FileCollection([FileProxy.from_project_relative_path(self, relative_path)])
else:
# path is a local project path
abs_path = os.path.join(self.project_root, relative_path)
if not os.path.exists(abs_path):
raise FileNotFoundError(f"Relative path {relative_path} does not exist.")
if code_files_only:
relative_file_paths = self.gather_source_files(relative_path=relative_path)
file_collection = FileCollection.from_local_project_paths(relative_file_paths, self)
else:
abs_path = os.path.join(self.project_root, relative_path)
if os.path.isfile(abs_path):
rel_paths_to_search = [relative_path]
else:
_dirs, rel_paths_to_search = scan_directory(
path=abs_path,
recursive=True,
is_ignored_dir=self.is_ignored_path,
is_ignored_file=self.is_ignored_path,
relative_to=self.project_root,
)
file_collection = FileCollection.from_local_project_paths(rel_paths_to_search, self)
return file_collection
def search_project_files_for_pattern(
self,
pattern: str,
relative_path: str = "",
@@ -347,6 +399,7 @@ class Project(ToStringMixin):
paths_include_glob: str | None = None,
paths_exclude_glob: str | None = None,
multiline: bool = True,
code_files_only: bool = True,
) -> list[MatchedConsecutiveLines]:
"""
Search for a pattern across all (non-ignored) source files
@@ -360,12 +413,10 @@ class Project(ToStringMixin):
:param multiline: Whether to compile the regex with the DOTALL flag (``.`` matches newlines).
:return: List of matched consecutive lines with context
"""
relative_file_paths = self.gather_source_files(relative_path=relative_path)
file_collection = self._create_file_collection(relative_path, code_files_only)
return search_files(
relative_file_paths,
file_collection,
pattern,
root_path=self.project_root,
file_reader=self.read_file,
context_lines_before=context_lines_before,
context_lines_after=context_lines_after,
paths_include_glob=paths_include_glob,
@@ -402,6 +453,10 @@ class Project(ToStringMixin):
:return: the language server manager, which is also stored in the project instance
"""
try:
# ensure that the project configuration, particularly the list of languages is complete,
# despite asynchronous first-time project configuration generation (which may not have completed yet)
self.project_config.await_asynchronous_completion()
# determine timeout to use for LS calls
tool_timeout = self.serena_config.tool_timeout
if tool_timeout is None or tool_timeout < 0:
@@ -419,18 +474,26 @@ class Project(ToStringMixin):
log.info(f"Creating language server manager for {self.project_root}")
self._language_server_manager_init_error = None
ls_specific_settings = {**self.serena_config.ls_specific_settings, **self.project_config.ls_specific_settings}
ls_specific_settings = dict(self.serena_config.ls_specific_settings)
if self.project_config.ls_specific_settings:
if self.is_trusted():
ls_specific_settings.update(self.project_config.ls_specific_settings)
else:
log.warning(
f"Project path {self.project_root} is not trusted, ignoring LS-specific settings from project configuration. "
"To trust the project, modify the trusted path patterns in the global configuration."
)
factory = LanguageServerFactory(
project_root=self.project_root,
project_config=self.project_config,
project_data_path=self._serena_data_folder,
encoding=self.project_config.encoding,
ignored_patterns=self._ignored_patterns,
ls_timeout=ls_timeout,
ls_specific_settings=ls_specific_settings,
additional_workspace_folders=self.project_config.additional_workspace_folders,
trace_lsp_communication=self.serena_config.trace_lsp_communication,
)
self.language_server_manager = LanguageServerManager.from_languages(self.project_config.languages, factory)
self.language_server_manager = LanguageServerManager.from_languages(self.project_config.languages, factory, self)
return self.language_server_manager
except Exception as e:
self._language_server_manager_init_error = e
@@ -494,6 +557,14 @@ class Project(ToStringMixin):
log.info("Removing and stopping the language server for language %s ...", language.value)
self.language_server_manager.remove_language_server(language)
def ls_sync_file_system_changes(self) -> int:
"""
Synchronizes file system changes with the project's associated language server(s), if applicable
"""
if self.language_server_manager:
return self.language_server_manager.sync_file_system_changes()
return 0
def shutdown(self, timeout: float = 2.0) -> None:
if self.language_server_manager is not None:
self.language_server_manager.stop_all(save_cache=True, timeout=timeout)
src/serena/project_server.py
+3 -1
View File
@@ -107,7 +107,9 @@ class ProjectServer:
from flask import cli
# suppress the default Flask startup banner
cli.show_server_banner = lambda *args, **kwargs: None
# ty cannot model reassigning a third-party module's function attribute (it rejects any
# replacement, even one with an identical signature), so the monkeypatch is suppressed here
cli.show_server_banner = lambda *args, **kwargs: None # ty: ignore[invalid-assignment]
self._app.run(host=host, port=port, debug=False, use_reloader=False, threaded=True)
return port
src/serena/resources/config/contexts/claude-code.yml
+4
View File
@@ -50,3 +50,7 @@ tool_description_overrides: {}
# Tools explicitly disabled by the project will not be available at all.
# The `activate_project` tool is always disabled in this case, as project switching cannot be allowed.
single_project: true
# Claude Code has a bug (it does not unpack structured tool output, causing a lot of unnecessary escaping
# owing to the wrapped structure), so we disable it here.
structured_tool_output: false
src/serena/resources/config/contexts/codebuddy.yml
+52
View File
@@ -0,0 +1,52 @@
description: CodeBuddy (CLI agent where file operations, basic edits, etc. are already covered; single project mode)
prompt: |
You are running in a CLI coding agent context where file operations, basic (line-based) edits and reads
as well as shell commands are handled by your own, internal tools.
You have access to Serena's code intelligence tools that exploit the symbolic
structure of the code and are much more efficient than your own tools for most coding scenarios.
If you are working on any coding task and if Serena's tools are deferred, you
should load them all immediately, before performing any read, grep or bash commands.
If Serena's tools can be used to achieve your task, you should prioritize them.
**CRITICAL**: If you haven't seen Serena's tools yet, you should use the tool search tool to load all of them
**right now**.
For exploring and reading code in a token-efficient manner, use Serena's code-centric overview and symbolic search tools.
For edits, prefer Serena's editing tools since they are more token efficient.
Especially when editing code you have read through Serena, **ALWAYS** use Serena's tools for editing, because your built-in tools will deny such edits (they will assume you haven't read the content).
**CRITICAL**: Never use a tool before having read its schema via the tool-search tool!
**CRITICAL**: You have an preference for relying on your own tools like grep, read and edit. It is of paramount
importance that you overcome this preference and instead develop a preference for using Serena's tools since they
are superior for most coding tasks. Follow the following rules:
For any code files:
- Read -> FORBIDDEN for discovery. Use get_symbols_overview, then find_symbol with include_body.
You may use read only if you already have an overview of the file and if reading whole symbols
is inappropriate (e.g. a few lines of code).
- Glob (by name) -> Allowed for discovery only.
- Grep (content) -> Allowed for discovery only; follow up reads or reference searches must be Serena.
- Edit -> FORBIDDEN. Use replace_symbol_body / insert_*_symbol / replace_content.
Disallowed reasoning. Do NOT use any of the following to justify Read/Edit on a code file:
- "I already know the path"
- "one Read call is faster than three Serena calls"
- "the built-in tool description says to use Read for known paths"
If you catch yourself reaching for one of these, that is the signal to switch to Serena.
excluded_tools:
- create_text_file
- read_file
- execute_shell_command
- find_file
- list_dir
- search_for_pattern
tool_description_overrides: {}
# whether to assume that Serena shall only work on a single project in this context (provided that a project is given
# when Serena is started).
# If set to true and a project is provided at startup, the set of tools is limited to those required by the project's
# concrete configuration, and other tools are excluded completely, allowing the set of tools to be minimal.
# Tools explicitly disabled by the project will not be available at all.
# The `activate_project` tool is always disabled in this case, as project switching cannot be allowed.
single_project: true
src/serena/resources/config/contexts/context.template.yml
+4 -1
View File
@@ -15,4 +15,7 @@ tool_description_overrides: {}
# If set to true and a project is provided at startup, the set of tools is limited to those required by the project's
# concrete configuration, and other tools are excluded completely, allowing the set of tools to be minimal.
# The `activate_project` tool will, therefore, be disabled in this case, as project switching is not allowed.
single_project: false
single_project: false
# whether to make MCP tools return structured output (null = auto-detect, true = always, false = never)
structured_tool_output: null
src/serena/resources/config/internal_modes/jetbrains.yml
+1 -7
View File
@@ -1,11 +1,5 @@
description: JetBrains tools replace language server-based tools
prompt: |
You have access to the very powerful JetBrains tools for symbolic operations.
These partly replace regular tools you were informed about:
* `jet_brains_find_symbol` replaces `find_symbol`
* `jet_brains_find_referencing_symbols` replaces `find_referencing_symbols`
* `jet_brains_get_symbols_overview` replaces `get_symbols_overview`
* `jet_brains_rename` replaces `rename_symbol`
prompt:
excluded_tools:
- find_symbol
- find_referencing_symbols
src/serena/resources/config/modes/benchmark.yml
+20
View File
@@ -0,0 +1,20 @@
description: Focus on completely finishing a task without interaction
prompt: |
You are operating in one-shot mode. Your goal is to complete the entire task autonomously without further user interaction.
You should assume auto-approval for all tools and continue working until the task is completely finished.
If the task is planning, your final result should be a comprehensive plan. If the task is coding, your final result
should be working code with all requirements fulfilled. Try to understand what the user asks you to do
and to assume as little as possible.
Only abort the task if absolutely necessary, such as when critical information is missing that cannot be inferred
from the codebase.
The task is provided up front; begin work immediately and see it through to completion without pausing for user input.
excluded_tools:
- write_memory
- read_memory
- edit_memory
- delete_memory
- rename_memory
- onboarding
src/serena/resources/config/modes/editing.yml
+21 -18
View File
@@ -1,18 +1,16 @@
description: All tools, with detailed instructions for code editing
prompt: |
Use symbolic editing tools whenever possible for precise code modifications.
{% if 'jet_brains_move' in available_tools -%}
prompt: |
Use the symbolic editing tools for precise, structure-aware code modifications.
**Refactoring tools**
You also have access to Serena's refactoring tools (move, inline, safe_delete)
which should ALWAYS be preferred over naive and primitive approaches when applicable.
When you need to move, inline or delete a symbol, you should always try using the corresponding tool first.
{%- endif %}
{% if 'jet_brains_move' in available_tools -%}
Apart from refactoring tools
{%- endif %}
you have two main approaches for editing code: (a) editing at the symbol level and (b) file-based editing.
For operations on existing symbols, prefer the dedicated refactoring tools over hand-edits:
`{{ tool_names['rename_symbol'] }}` and `{{ tool_names['safe_delete_symbol'] }}`{% if 'jet_brains_move' in available_tools %}, plus `jet_brains_move` and `jet_brains_inline_symbol`{% endif %} —
they are reference-aware and update or check all usages atomically. When such a tool returns success, the
refactoring is already complete and consistent across all declarations, references, overrides and imports —
trust it: do not re-read the changed files or re-run the build / test suite just to confirm the refactor
itself propagated, this would be wasted effort.
Apart from those refactoring tools, you have two main approaches for editing code: (a) editing at the symbol level and (b) file-based editing.
The symbol-based approach is appropriate if you need to adjust an entire symbol, e.g. a method, a class, a function, etc.
It is not appropriate if you need to adjust just a few lines of code within a larger symbol.
@@ -21,18 +19,23 @@ prompt: |
If you need to replace the definition of a symbol, use the `replace_symbol_body` tool.
If you want to add some new code at the end of the file, use the `insert_after_symbol` tool with the last top-level symbol in the file.
Similarly, you can use `insert_before_symbol` with the first top-level symbol in the file to insert code at the beginning of a file.
You can understand relationships between symbols by using the `find_referencing_symbols` tool. If not explicitly requested otherwise by the user,
You can understand relationships between symbols by using the `{{ tool_names['find_referencing_symbols'] }}` tool. If not explicitly requested otherwise by the user,
you make sure that when you edit a symbol, the change is either backward-compatible or you find and update all references as needed.
The `find_referencing_symbols` tool will give you code snippets around the references as well as symbolic information.
You can assume that all symbol editing tools are reliable, so you never need to verify the results if the tools return without error.
The `{{ tool_names['find_referencing_symbols'] }}` tool will give you code snippets around the references as well as symbolic information.
These edits are reliable: once a tool returns without error the change is applied, so you do not need to re-read the file just to confirm it.
{% if 'replace_content' in available_tools %}
**File-based editing**
The `replace_content` tool allows you to perform regex-based replacements within files (as well as simple string replacements).
This is your primary tool for editing code whenever replacing or deleting a whole symbol would be a more expensive operation,
e.g. if you need to adjust just a few lines of code within a method.
You are extremely good at regex, so you never need to check whether the replacement produced the correct result.
In particular, you know how to use wildcards effectively in order to avoid specifying the full original text to be replaced!
In `regex` mode, wildcards like `start.*?end` let you match a span without quoting its full text; an ambiguous match
returns an error you can refine, so a tight wildcard pattern is both cheaper and safe.
For several small edits within one file, prefer a batch of targeted `replace_content` calls over rewriting the whole
file: rewriting has to re-emit the file's entire contents, whereas each targeted edit emits only the changed text.
{% if 'replace_in_files' in available_tools %}For the SAME edit across many files, use `replace_in_files`, which applies
it everywhere in one call — equally safe and transparent: its `dry_run` mode first previews every change as a diff with a
per-occurrence id, so you can then apply all of them or just a chosen subset.{% endif %}
{% endif %}
excluded_tools:
- replace_lines
src/serena/resources/config/modes/interactive.yml
+5 -6
View File
@@ -1,11 +1,10 @@
description: Interactive mode for clarification and step-by-step work
prompt: |
You are operating in interactive mode. You should engage with the user throughout the task, asking for clarification
whenever anything is unclear, insufficiently specified, or ambiguous.
Break down complex tasks into smaller steps and explain your thinking at each stage. When you're uncertain about
a decision, present options to the user and ask for guidance rather than making assumptions.
You should engage with the user throughout the task, asking for clarification whenever anything is unclear,
insufficiently specified, or ambiguous.
When you're uncertain about a decision, present options to the user and ask for guidance rather than making assumptions.
Focus on providing informative results for intermediate steps, such that the user can follow along with your progress and
provide feedback as needed.
This applies unless the user explicitly stated they are unavailable/instructs you to proceed without asking questions.
excluded_tools: []
src/serena/resources/config/prompt_templates/system_prompt.yml
+18 -18
View File
@@ -5,20 +5,14 @@ prompts:
connection_prompt: |
CRITICAL: Before starting to work on a coding task, call the `initial_instructions` tool to read the 'Serena Instructions Manual'.
system_prompt: |
You have access to semantic coding tools upon which you rely heavily for all your work.
You operate in a resource-efficient and intelligent manner, always keeping in mind to not read or generate
content that is not needed for the task at hand.
You have semantic coding tools that you rely on heavily. Work resource-efficiently: don't read or
generate content the task doesn't need.
Some tasks may require you to understand the architecture of large parts of the codebase, while for others,
it may be enough to read a small set of symbols or a single file.
You avoid reading entire files unless it is absolutely necessary, instead relying on intelligent step-by-step
acquisition of information. {% if 'ToolMarkerSymbolicRead' in available_markers %}Once you have read a full file, it does not make
sense to analyse it with the symbolic read tools; you already have the information.{% endif %}
You can achieve intelligent reading of code by using the symbolic tools for getting an overview of symbols and
the relations between them, and then only reading the bodies of symbols that are necessary to complete the task at hand.
You can use the standard tools like list_dir, find_file and search_for_pattern if you need to.
Where appropriate, you pass the `relative_path` parameter to restrict the search to a specific file or directory.
Some tasks require understanding a large part of the codebase; others need only a few symbols or a
single file. Avoid reading whole files unless necessary — acquire information step by step, using
the symbolic tools to get an overview of symbols and their relations, then reading only the bodies
you need. {% if 'ToolMarkerSymbolicRead' in available_markers %}Once you have read a full file, there is no point re-analysing it with the symbolic read
tools — you already have it.{% endif %}
{% if 'search_for_pattern' in available_tools %}
If you are unsure about a symbol's name or location{% if 'find_symbol' in available_tools %} (to the extent that substring_matching for the symbol name is not enough){% endif %}, you can use the `search_for_pattern` tool, which allows fast
and flexible search for patterns in the codebase.{% if 'ToolMarkerSymbolicRead' in available_markers %} In this way, you can first find candidates for symbols or files,
@@ -26,14 +20,14 @@ prompts:
{% endif %}
{% if 'ToolMarkerSymbolicRead' in available_markers %}
Symbols are identified by their `name_path` and `relative_path` (see the description of the `find_symbol` tool).
You can get information about the symbols in a file by using the `get_symbols_overview` tool or use the `find_symbol` to search.
Symbols are identified by their `name_path` and `relative_path`.
You can get an overview of the symbols in a file by using the `{{ tool_names['get_symbols_overview'] }}` tool, or search for a specific symbol with `{{ tool_names['find_symbol'] }}`.
You only read the bodies of symbols when you need to (e.g. if you want to fully understand or edit it).
For example, if you are working with Python code and already know that you need to read the body of the constructor of the class Foo, you can directly
use `find_symbol` with name path pattern `Foo/__init__` and `include_body=True`. If you don't know yet which methods in `Foo` you need to read or edit,
you can use `find_symbol` with name path pattern `Foo`, `include_body=False` and `depth=1` to get all (top-level) methods of `Foo` before proceeding
use `{{ tool_names['find_symbol'] }}` with name path pattern `Foo/__init__` and `include_body=True`. If you don't know yet which methods in `Foo` you need to read or edit,
you can use `{{ tool_names['find_symbol'] }}` with name path pattern `Foo`, `include_body=False` and `depth=1` to get all (top-level) methods of `Foo` before proceeding
to read the desired methods with `include_body=True`.
You can understand relationships between symbols by using the `find_referencing_symbols` tool.
You can understand relationships between symbols by using the `{{ tool_names['find_referencing_symbols'] }}` tool.
{% endif %}
{% if 'read_memory' in available_tools -%}
@@ -45,6 +39,12 @@ prompts:
{%- endif %}
Line numbers returned by Serena's tools are 0-based!
Serena executes tool calls one at a time internally, even when you request several at once. So whenever you have
independent operations to perform, you can safely issue them as parallel tool calls in a single turn: they are applied
in the order issued and cannot race with or corrupt one another, and batching them this way saves round-trips. Every
separate round-trip re-sends the whole growing context, so maximizing parallel tool calls is the single biggest lever
on cost — err on the side of batching. Only call tools sequentially when one genuinely depends on the result of another.
The context and modes of operation are described below. These determine how to interact with your user
and which kinds of interactions are expected of you.
src/serena/resources/dashboard/dashboard.js
+2
View File
@@ -913,6 +913,8 @@ class Dashboard {
if (response.status === 'success') {
if (response.last_execution !== null && response.last_execution.logged) {
self.displayLastExecution(response.last_execution);
} else {
self.displayLastExecution(null);
}
} else {
console.error('Error loading last execution:', response.message);
src/serena/resources/project.template.yml
+56 -23
View File
@@ -1,23 +1,25 @@
# the name by which the project can be referenced within Serena
# the name by which the project can be referenced within Serena/when chatting with the LLM.
project_name: "project_name"
# list of languages for which language servers are started; choose from:
# al angular ansible bash clojure
# cpp cpp_ccls crystal csharp csharp_omnisharp
# dart elixir elm erlang fortran
# fsharp go groovy haskell haxe
# hlsl html java json julia
# kotlin lean4 lua luau markdown
# list of languages for which language servers are started (LSP backend only); choose from:
# ada al angular ansible bash
# bsl clojure cpp cpp_ccls crystal
# csharp csharp_omnisharp cue dart elixir
# elm erlang fortran fsharp gdscript
# go groovy haskell haxe hlsl
# html java json julia kotlin
# latex lean4 lua luau markdown
# matlab msl nix ocaml pascal
# perl php php_phpactor powershell python
# python_jedi python_ty r rego ruby
# ruby_solargraph rust scala scss solidity
# svelte swift systemverilog terraform toml
# typescript typescript_vts vue yaml zig
# (This list may be outdated. For the current list, see values of Language enum here:
# https://github.com/oraios/serena/blob/main/src/solidlsp/ls_config.py
# For some languages, there are alternative language servers, e.g. csharp_omnisharp, ruby_solargraph.)
# perl php php_phpactor php_phpantom powershell
# python python_jedi python_pyrefly python_ty r
# rego ruby ruby_solargraph rust scala
# scss solidity svelte swift systemverilog
# terraform toml typescript typescript_vts vue
# yaml zig
# (This list may be outdated; generated with scripts/print_language_list.py;
# For the current list, see values of Language enum here:
# https://github.com/oraios/serena/blob/main/src/solidlsp/ls_config.py)
# For some languages, there are alternative language servers, e.g. csharp_omnisharp, ruby_solargraph.)
# Note:
# - For C, use cpp
# - For JavaScript, use typescript
@@ -37,6 +39,19 @@ languages: ["python"]
# For a list of possible encodings, see https://docs.python.org/3.11/library/codecs.html#standard-encodings
encoding: "utf-8"
# optional shell command to run before the language backend (LSP or JetBrains) is initialised.
# the command runs in the project root directory and is only executed if the project is trusted
# (see trusted_project_path_patterns in the global configuration).
# serena waits for the command to exit: a non-zero exit code is logged as an error but does not
# abort activation. a per-project timeout (activation_command_timeout, default 180s) is the safety
# backstop for non-terminating commands; on expiry the process is killed and activation continues.
# example: activation_command: "npx nx run-many -t build"
activation_command:
# maximum time in seconds to wait for activation_command to complete before killing it (default 180s).
# must be a positive number.
activation_command_timeout: 180
# line ending convention to use when writing source files.
# Possible values: unset (use global setting), "lf", "crlf", or "native" (platform default)
# This does not affect Serena's own files (e.g. memories and configuration files), which always use native line endings.
@@ -54,23 +69,41 @@ ignore_all_files_in_gitignore: true
# advanced configuration option allowing to configure language server-specific options.
# Maps the language key to the options.
# Have a look at the docstring of the constructors of the LS implementations within solidlsp (e.g., for C# or PHP) to see which options are available.
# No documentation on options means no options are available.
# The settings are considered only if the project is trusted (see global configuration to define trusted projects).
# See https://oraios.github.io/serena/02-usage/050_configuration.html#language-server-specific-settings
ls_specific_settings: {}
# list of additional workspace folder paths for cross-package reference support (e.g. in monorepos).
# list of workspace folder paths (LSP backend only).
# These folders will be used to build up Serena's symbol index.
# Paths must be within the project root and should thus be relative to the project root.
# Furthermore, the paths should not be filtered by ignore settings.
# Default setting: The entire project root folder (".") is considered.
# In (large) monorepos, this can be used to index only subfolders of the project root, e.g.
# ls_workspace_folders:
# - "./subproject1"
# - "./subproject2"
ls_workspace_folders: ["."]
# list of additional workspace folder paths for cross-package reference support.
# Paths can be absolute or relative to the project root.
# Each folder is registered as an LSP workspace folder, enabling language servers to discover
# symbols and references across package boundaries.
# Currently supported for: TypeScript.
# symbols and references across package boundaries, but these folders are not indexed by Serena,
# i.e. the respective symbols will not be found using Serena's symbol search tools.
# Example:
# additional_workspace_folders:
# - ../sibling-package
# - ../shared-lib
additional_workspace_folders: []
ls_additional_workspace_folders: []
# list of additional paths to ignore in this project.
# Same syntax as gitignore, so you can use * and **.
# Important: quote patterns that start with `*`, otherwise YAML treats them as aliases.
# Example:
# ignored_paths:
# - "examples/**"
# - ".worktrees/**"
# - "**/bin/**"
# - "**/obj/**"
# Note: global ignored_paths from serena_config.yml are also applied additively.
ignored_paths: []
src/serena/resources/serena_config.template.yml
+16
View File
@@ -103,6 +103,7 @@ ls_specific_settings: {}
# list of paths to ignore across all projects.
# Same syntax as gitignore, so you can use * and **.
# These patterns are merged additively with each project's own ignored_paths.
# Quote patterns that start with `*`, e.g. `"**/bin/**"`.
ignored_paths: []
# list of regex patterns which, when matched, mark a memory entry as read‑only.
@@ -183,5 +184,20 @@ symbol_info_budget: 10
# Example for a central location: "/projects-metadata/$projectFolderName/.serena"
project_serena_folder_location: "$projectDir/.serena"
# list of glob patterns for project root directories that are considered trusted.
# Some project settings will only be applied if the project is trusted.
# A glob pattern can contain the following:
# * matches any sequence of characters except path separators
# ** matches any sequence of characters including path separators
# ? matches any single character except path separators
# [abc] matches any single character in the set (here: a, b, or c)
# Path separators are normalised internally, so on Windows, both / and \ can be used in the patterns.
# Example:
# trusted_project_path_patterns:
# - /home/user/projects/**
# - C:\Users\Anna\Dev\work\**
# The pattern "**" matches any project path, so it can be used to trust all projects.
trusted_project_path_patterns: []
# the list of registered project paths (updated automatically).
projects: []
src/serena/symbol.py
+19 -1
View File
@@ -5,6 +5,7 @@ import os
from abc import ABC, abstractmethod
from collections import OrderedDict
from collections.abc import Callable, Iterable, Iterator, Sequence
from contextlib import contextmanager
from dataclasses import asdict, dataclass
from time import perf_counter
from typing import Any, Generic, Literal, NotRequired, Self, TypedDict, TypeVar
@@ -1261,6 +1262,7 @@ class SymbolDictGrouper(Generic[TSymbolDict], ABC):
self._group_keys = group_keys
self._group_children_keys = group_children_keys
self._collapse_singleton = collapse_singleton
self._is_enabled = True
def _group_by(self, l: list[dict], keys: list[str], children_keys: list[str], is_children: bool) -> dict[str, Any] | list[Any]:
"""
@@ -1321,7 +1323,23 @@ class SymbolDictGrouper(Generic[TSymbolDict], ABC):
"""
# avoid side effects by working on a deep-copy
symbols_copy = copy.deepcopy(symbols)
return self._group_by(symbols_copy, self._group_keys, self._group_children_keys, is_children=False) # type: ignore
if not self._is_enabled:
return symbols_copy
return self._group_by(symbols_copy, self._group_keys, self._group_children_keys, is_children=False)
@contextmanager
def disabled_context(self) -> Iterator[None]:
"""
A context manager that temporarily disables grouping.
"""
original_state = self._is_enabled
self._is_enabled = False
try:
yield
finally:
self._is_enabled = original_state
class LanguageServerSymbolDictGrouper(SymbolDictGrouper[LanguageServerSymbol.OutputDict]):
src/serena/task_executor.py
+1 -1
View File
@@ -200,7 +200,7 @@ class TaskExecutor:
self._task_executor_task_index += 1
else:
task_prefix_name = "BackgroundTask"
task_name = f"{task_prefix_name}:{name or task.__name__}"
task_name = f"{task_prefix_name}:{name or getattr(task, '__name__', 'task')}"
if logged:
log.info(f"Scheduling {task_name}")
task_obj = self.Task(function=task, name=task_name, logged=logged, timeout=timeout)
src/serena/tools/cmd_tools.py
+1 -1
View File
@@ -48,5 +48,5 @@ class ExecuteShellCommandTool(Tool, ToolMarkerCanEdit):
)
result = execute_shell_command(command, cwd=_cwd, capture_stderr=capture_stderr)
result = result.json()
result = result.model_dump_json()
return self._limit_length(result, max_answer_chars)
src/serena/tools/file_tools.py
+309 -58
View File
@@ -14,7 +14,14 @@ from typing import Literal
from serena.tools import SUCCESS_RESULT, EditedFileContext, EditingToolWithDiagnostics, Tool, ToolMarkerOptional
from serena.util.file_system import scan_directory
from serena.util.text_utils import ContentReplacer, search_files
from serena.util.text_utils import (
ContentReplacer,
MultiFileContentReplacer,
ReplacementOccurrence,
expand_braces,
glob_match,
)
from solidlsp.ls_utils import TextUtils
class ReadFileTool(Tool):
@@ -27,7 +34,7 @@ class ReadFileTool(Tool):
Reads the given file or a chunk of it.
:param relative_path: the relative path to the file to read
:param start_line: the 0-based index of the first line to be retrieved.
:param start_line: the 0-based index of the first line to be retrieved, negative values count from the end of the file.
:param end_line: the 0-based index of the last line to be retrieved (inclusive). If None, read until the end of the file.
:param max_answer_chars: if the file (chunk) is longer than this number of characters,
no content will be returned. Don't adjust unless there is really no other way to get the content
@@ -36,8 +43,10 @@ class ReadFileTool(Tool):
"""
self.project.validate_relative_path(relative_path, require_not_ignored=True)
# read lines, using the same (LSP-compliant) notion of line breaks as the line-based editing tools
result = self.project.read_file(relative_path)
result_lines = result.splitlines()
result_lines = TextUtils.split_lines(result)
if end_line is None:
result_lines = result_lines[start_line:]
else:
@@ -54,7 +63,7 @@ class CreateTextFileTool(EditingToolWithDiagnostics):
def apply(self, relative_path: str, content: str) -> str:
"""
Write a new file or overwrite an existing file.
Write a new file or overwrite an existing file with the given content.
:param relative_path: the relative path to the file to create
:param content: the (appropriately encoded) content to write to the file
@@ -175,16 +184,12 @@ class ReplaceContentTool(EditingToolWithDiagnostics):
r"""
Replaces one or more occurrences of a given pattern in a file with new content.
This is the preferred way to replace content in a file whenever the symbol-level
tools are not appropriate.
VERY IMPORTANT: The "regex" mode allows very large sections of code to be replaced without fully quoting them!
Use a regex of the form "beginning.*?end-of-text-to-be-replaced" to be faster and more economical!
ALWAYS try to use wildcards to avoid specifying the exact content to be replaced,
especially if it spans several lines. Note that you cannot make mistakes, because if the regex should match
multiple occurrences while you disabled `allow_multiple_occurrences`, an error will be returned, and you can retry
with a revised regex.
Therefore, using regex mode with suitable wildcards is usually the best choice!
VERY IMPORTANT: The "regex" mode allows very large sections of code to be replaced WITHOUT
quoting them fully: use a needle of the form "beginning.*?end-of-text-to-be-replaced" with
wildcards instead of pasting the exact original text — shorter, cheaper, and you cannot make
mistakes, because an ambiguous match returns an error you can refine, so wildcards are safe.
Prefer regex mode with suitable wildcards for long multi-line replacements; use the
symbol-level editors when replacing a whole method/class.
:param relative_path: the relative path to the file
:param needle: the string or regex pattern to search for.
@@ -225,6 +230,239 @@ class ReplaceContentTool(EditingToolWithDiagnostics):
return diagnostics_context.format_result(SUCCESS_RESULT)
class ReplaceInFilesTool(EditingToolWithDiagnostics):
"""
Replaces occurrences of a pattern across multiple files, with dry-run preview and per-occurrence selection.
"""
def apply(
self,
needle: str,
repl: str,
mode: Literal["literal", "regex"],
relative_path: str = "",
paths_include_glob: str = "",
paths_exclude_glob: str = "",
dry_run: bool = False,
occurrence_ids: list[str] | None = None,
expected_count: int = -1,
max_answer_chars: int = -1,
) -> str:
r"""
Replaces occurrences of a pattern across multiple files in ONE call.
This is the preferred tool for repeated small edits (renames, import swaps, annotation changes,
path prefixes) spanning several files or many places in one file: one call with a SHORT pattern
replaces many single-file replacements with long disambiguating needles.
Recommended protocol whenever there is ANY risk of unintended replacements:
1. Call with dry_run=True: every prospective change is returned as a minimal line diff with an
occurrence id; nothing is modified.
2. Call again with dry_run=False, passing the ids you want in occurrence_ids (omit it to apply
all). You pick the desired replacements from the list - no counting, no needle-crafting.
For clearly unambiguous bulk replacements you may skip the dry run; pass expected_count as a
guard. If the actual number of matches differs, NOTHING is changed and the diff list is
returned, so a failed guard costs one call and gives you the dry-run output to select from.
:param needle: the string (mode "literal") or regular expression (mode "regex"; Python `re`
syntax with DOTALL and MULTILINE) to search for
:param repl: the replacement string. In regex mode, backreferences to matched groups can be
specified as $!1, $!2, etc.
:param mode: either "literal" or "regex", specifying how `needle` is to be interpreted
:param relative_path: only consider this file or directory (default: the whole project)
:param paths_include_glob: optional glob (relative to the project root, e.g. "src/**/*.java")
restricting which files are considered
:param paths_exclude_glob: optional glob of files to exclude; takes precedence over the include glob
:param dry_run: if True, do not modify anything; return the prospective changes as a list of
diffs with occurrence ids
:param occurrence_ids: optional list of occurrence ids (obtained from a dry run) to which the
replacement is restricted; if any id is unknown or stale, NOTHING is changed. If omitted,
all occurrences are replaced.
:param expected_count: optional guard for calls without occurrence_ids: the number of
occurrences you expect to be replaced. If the actual count differs, nothing is changed and
the list of prospective changes is returned. -1 disables the guard.
:param max_answer_chars: if the output exceeds this many characters, a shortened version is
returned. -1 uses the configured default.
:return: in a dry run, the prospective changes; otherwise a summary of the applied replacements
"""
replacer = MultiFileContentReplacer(mode=mode)
files = self._collect_files(relative_path, paths_include_glob, paths_exclude_glob)
occurrences = replacer.find_occurrences(files, needle, repl)
contents = dict(files)
if dry_run:
return self._render_listing(replacer, occurrences, contents, max_answer_chars, dry_run=True)
if occurrence_ids is not None:
selected, problems = self._resolve_occurrence_ids(occurrence_ids, occurrences)
if problems:
problem_lines = "\n".join(f" {p}" for p in problems)
raise ValueError(
f"{len(problems)} of the given occurrence_ids could not be resolved - NO changes were applied:\n"
f"{problem_lines}\n"
"Re-run with dry_run=True to obtain current occurrence ids."
)
if not selected:
raise ValueError("occurrence_ids is empty - pass at least one id from a dry run, or omit the parameter to replace all.")
return self._apply_occurrences(replacer, selected, contents, needle, repl)
# blind apply (no ids)
if not occurrences:
raise ValueError(
"No occurrences of the pattern were found - NO changes were applied. "
"Check the mode (a literal needle containing regex metacharacters must use mode 'literal'; "
"wildcards require mode 'regex') and the path/glob restrictions, "
"or locate the content with search_for_pattern first."
)
if expected_count >= 0 and len(occurrences) != expected_count:
listing = self._render_listing(replacer, occurrences, contents, max_answer_chars, dry_run=False)
raise ValueError(
f"expected_count={expected_count}, but the pattern matches {len(occurrences)} occurrence(s) - "
f"NO changes were applied. Review the prospective changes below; re-issue with the corrected "
f"expectation, a refined pattern, or occurrence_ids selecting the intended subset.\n{listing}"
)
ambiguous = [o for o in occurrences if o.is_ambiguous]
if ambiguous:
listing = self._render_listing(replacer, occurrences, contents, max_answer_chars, dry_run=False)
raise ValueError(
f"{len(ambiguous)} occurrence(s) are ambiguous (the pattern matches again inside the matched text, "
f"indicating possible over-matching) - NO changes were applied. Review the prospective changes below "
f"and either refine the pattern or explicitly select occurrences via occurrence_ids.\n{listing}"
)
return self._apply_occurrences(replacer, occurrences, contents, needle, repl)
def _collect_files(self, relative_path: str, paths_include_glob: str, paths_exclude_glob: str) -> list[tuple[str, str]]:
"""Collects (relative_path, content) pairs of the non-ignored files in scope, in sorted path order."""
relative_path = relative_path.strip()
if relative_path:
self.project.validate_relative_path(relative_path, require_not_ignored=True)
abs_path = os.path.join(self.get_project_root(), relative_path)
if not os.path.exists(abs_path):
raise FileNotFoundError(f"Relative path {relative_path} does not exist.")
if os.path.isfile(abs_path):
rel_paths = [relative_path]
else:
_dirs, rel_paths = scan_directory(
path=abs_path,
recursive=True,
is_ignored_dir=self.project.is_ignored_path,
is_ignored_file=self.project.is_ignored_path,
relative_to=self.get_project_root(),
)
include_patterns = expand_braces(paths_include_glob.strip()) if paths_include_glob.strip() else None
exclude_patterns = expand_braces(paths_exclude_glob.strip()) if paths_exclude_glob.strip() else None
files: list[tuple[str, str]] = []
for path in sorted(rel_paths):
if include_patterns and not any(glob_match(p, path) for p in include_patterns):
continue
if exclude_patterns and any(glob_match(p, path) for p in exclude_patterns):
continue
try:
files.append((path, self.project.read_file(path)))
except Exception:
continue # skip unreadable (e.g. binary) files
return files
def _render_listing(
self,
replacer: MultiFileContentReplacer,
occurrences: list[ReplacementOccurrence],
contents: dict[str, str],
max_answer_chars: int,
dry_run: bool,
) -> str:
affected_files = sorted({o.relative_path for o in occurrences})
header = f"Found {len(occurrences)} occurrence(s) in {len(affected_files)} file(s)."
if dry_run:
header += (
" DRY RUN - no changes were applied.\n"
"Re-issue with dry_run=False to replace all of them, or additionally pass occurrence_ids "
"with the ids of the occurrences to replace."
)
parts = [header]
for path in affected_files:
file_occurrences = [o for o in occurrences if o.relative_path == path]
parts.append(f"\n{path} ({len(file_occurrences)} occurrence(s)):")
for occ in file_occurrences:
parts.append(replacer.render_occurrence_diff(occ, contents[path]))
result = "\n".join(parts)
def make_locations_only() -> str:
lines = [header] + [f" [{o.occurrence_id}] line {o.start_line}" for o in occurrences]
return "\n".join(lines)
def make_per_file_counts() -> str:
counts = {path: sum(1 for o in occurrences if o.relative_path == path) for path in affected_files}
return f"{header}\nOccurrence counts per file:\n{self._to_json(counts)}"
def make_summary() -> str:
return header
return self._limit_length(
result, max_answer_chars, shortened_result_factories=[make_locations_only, make_per_file_counts, make_summary]
)
@staticmethod
def _resolve_occurrence_ids(
occurrence_ids: list[str], occurrences: list[ReplacementOccurrence]
) -> tuple[list[ReplacementOccurrence], list[str]]:
"""Resolves the requested ids against the current occurrences, diagnosing each failure."""
occurrences_by_id = {o.occurrence_id: o for o in occurrences}
indices_by_path: dict[str, set[int]] = {}
for o in occurrences:
indices_by_path.setdefault(o.relative_path, set()).add(o.index_in_file)
selected: dict[str, ReplacementOccurrence] = {}
problems: list[str] = []
for oid in occurrence_ids:
occurrence = occurrences_by_id.get(oid)
if occurrence is not None:
selected[oid] = occurrence
continue
id_match = MultiFileContentReplacer.OCCURRENCE_ID_REGEX.match(oid)
if id_match is None:
problems.append(f"{oid}: malformed id (expected '<path>:<index>@<digest>' as returned by a dry run)")
elif id_match.group("path") not in indices_by_path:
problems.append(f"{oid}: the pattern currently has no matches in this file")
elif int(id_match.group("index")) not in indices_by_path[id_match.group("path")]:
problems.append(f"{oid}: the file now has fewer matches than at dry-run time (content changed)")
else:
problems.append(f"{oid}: the matched text changed since the dry run (content changed)")
return list(selected.values()), problems
def _apply_occurrences(
self,
replacer: MultiFileContentReplacer,
occurrences: list[ReplacementOccurrence],
contents: dict[str, str],
needle: str,
repl: str,
) -> str:
occurrences_by_file: dict[str, list[ReplacementOccurrence]] = {}
for occ in occurrences:
occurrences_by_file.setdefault(occ.relative_path, []).append(occ)
with self.DiagnosticsContext(self, *occurrences_by_file.keys()) as diagnostics_context:
code_editor = self.create_code_editor()
for path, file_occurrences in occurrences_by_file.items():
with EditedFileContext(path, code_editor) as context:
original_content = context.get_original_content()
if original_content != contents[path]:
# the editor's view differs from what was scanned (e.g. line-ending normalization);
# re-derive the occurrences from the authoritative content and re-validate by id
fresh_by_id = {o.occurrence_id: o for o in replacer.find_occurrences([(path, original_content)], needle, repl)}
try:
file_occurrences = [fresh_by_id[o.occurrence_id] for o in file_occurrences]
except KeyError as e:
raise ValueError(
f"The content of {path} changed while replacing (occurrence {e} no longer resolves); "
f"the file was NOT modified. Re-run with dry_run=True for current ids."
) from e
context.set_updated_content(replacer.apply_to_content(original_content, file_occurrences))
per_file = "\n".join(f" {path}: {len(occs)}" for path, occs in occurrences_by_file.items())
summary = f"Replaced {len(occurrences)} occurrence(s) in {len(occurrences_by_file)} file(s):\n{per_file}"
return diagnostics_context.format_result(summary)
class DeleteLinesTool(EditingToolWithDiagnostics, ToolMarkerOptional):
"""
Deletes a range of lines within a file.
@@ -355,44 +593,16 @@ class SearchForPatternTool(Tool):
if relative_path:
self.project.validate_relative_path(relative_path, require_not_ignored=True)
abs_path = os.path.join(self.get_project_root(), relative_path)
if not os.path.exists(abs_path):
raise FileNotFoundError(f"Relative path {relative_path} does not exist.")
if restrict_search_to_code_files:
matches = self.project.search_source_files_for_pattern(
pattern=substring_pattern,
relative_path=relative_path,
context_lines_before=context_lines_before,
context_lines_after=context_lines_after,
paths_include_glob=paths_include_glob.strip(),
paths_exclude_glob=paths_exclude_glob.strip(),
multiline=multiline,
)
else:
if os.path.isfile(abs_path):
rel_paths_to_search = [relative_path]
else:
_dirs, rel_paths_to_search = scan_directory(
path=abs_path,
recursive=True,
is_ignored_dir=self.project.is_ignored_path,
is_ignored_file=self.project.is_ignored_path,
relative_to=self.get_project_root(),
)
# TODO (maybe): not super efficient to walk through the files again and filter if glob patterns are provided
# but it probably never matters and this version required no further refactoring
matches = search_files(
rel_paths_to_search,
substring_pattern,
context_lines_before=context_lines_before,
context_lines_after=context_lines_after,
file_reader=self.project.read_file,
root_path=self.get_project_root(),
paths_include_glob=paths_include_glob,
paths_exclude_glob=paths_exclude_glob,
multiline=multiline,
)
matches = self.project.search_project_files_for_pattern(
pattern=substring_pattern,
relative_path=relative_path,
context_lines_before=context_lines_before,
context_lines_after=context_lines_after,
paths_include_glob=paths_include_glob.strip(),
paths_exclude_glob=paths_exclude_glob.strip(),
multiline=multiline,
code_files_only=restrict_search_to_code_files,
)
# group matches by file
file_to_matches: dict[str, list[str]] = defaultdict(list)
@@ -401,15 +611,48 @@ class SearchForPatternTool(Tool):
file_to_matches[match.source_file_path].append(match.to_display_string())
# capture lightweight match data for shortening before serialization
match_lines_by_file: dict[str, list[int]] = defaultdict(list)
match_lines_by_file: dict[str, list[dict[str, int | str]]] = defaultdict(list)
for match in matches:
assert match.source_file_path is not None
match_lines_by_file[match.source_file_path].append(match.matched_lines[0].line_number)
first = match.matched_lines[0]
match_lines_by_file[match.source_file_path].append({"line": first.line_number, "text": first.line_content.strip()})
# shortened result closures, from least to most aggressive shortening
def make_lines_only() -> str:
"""Match locations without surrounding context"""
return f"Match lines per file:\n{self._to_json(match_lines_by_file)}"
_TEXT_TRUNCATE = 60
def render_first_lines(truncate: bool) -> str:
"""Render each match's first line, either in full or truncated to a fixed length."""
def entry_text(text: str) -> str:
if truncate and len(text) > _TEXT_TRUNCATE:
return text[:_TEXT_TRUNCATE] + "..."
return text
compact = {
path: [{"line": m["line"], "text": entry_text(str(m["text"]))} for m in lines]
for path, lines in match_lines_by_file.items()
}
if truncate:
header = (
f"Matched lines (text over {_TEXT_TRUNCATE} chars is truncated, marked with a trailing '...'); "
"use read_file with the line numbers for full content:"
)
else:
header = "Matched lines per file; use read_file with the line numbers for surrounding context:"
return f"{header}\n{self._to_json(compact)}"
def make_first_lines_full() -> str:
"""Match locations with each match's full first line."""
return render_first_lines(truncate=False)
def make_first_lines_truncated() -> str:
"""Match locations with each match's first line truncated to a fixed length."""
return render_first_lines(truncate=True)
def make_line_numbers_only() -> str:
"""Match locations as bare line numbers (no text)."""
numbers = {path: [m["line"] for m in lines] for path, lines in match_lines_by_file.items()}
return f"Match lines per file:\n{self._to_json(numbers)}"
def make_per_file_counts() -> str:
counts = {path: len(lines) for path, lines in match_lines_by_file.items()}
@@ -420,5 +663,13 @@ class SearchForPatternTool(Tool):
result = self._to_json(file_to_matches)
return self._limit_length(
result, max_answer_chars, shortened_result_factories=[make_lines_only, make_per_file_counts, make_summary]
result,
max_answer_chars,
shortened_result_factories=[
make_first_lines_full,
make_first_lines_truncated,
make_line_numbers_only,
make_per_file_counts,
make_summary,
],
)
src/serena/tools/jetbrains_tools.py
+26 -13
View File
@@ -40,6 +40,7 @@ class JetBrainsFindSymbolTool(Tool, ToolMarkerSymbolicRead, ToolMarkerOptional):
Specify `depth > 0` to retrieve children (e.g., methods of a class).
Important: through `search_deps=True` dependencies can be searched, which
should be preferred to web search or other less sophisticated approaches to analyzing dependencies.
You will always receive at least quick info on the symbol (even if `include_info=False`).
A name path is a path in the symbol tree *within a source file*.
For example, the method `my_method` defined in class `MyClass` would have the name path `MyClass/my_method`.
@@ -52,6 +53,7 @@ class JetBrainsFindSymbolTool(Tool, ToolMarkerSymbolicRead, ToolMarkerOptional):
* a relative path like "class/method", which will match any symbol with that name path suffix
* an absolute name path "/class/method" (absolute name path), which requires an exact match of the full name path within the source file.
Append an index `[i]` to match a specific overload only, e.g. "MyClass/my_method[1]".
In any path component, using `*` will match any sequence of characters (excluding /), e.g. "Class/*substring*" matches a member substring.
:param name_path_pattern: the name path matching pattern (see above)
:param depth: depth up to which descendants shall be retrieved (e.g. use 1 to also retrieve immediate children;
@@ -60,7 +62,7 @@ class JetBrainsFindSymbolTool(Tool, ToolMarkerSymbolicRead, ToolMarkerOptional):
:param relative_path: Optional. Restrict search to this file or directory. If not specified, searches entire codebase.
Note: for external dependencies, this must be an identifier starting with `<ext` that you have received
earlier (don't try to guess!).
:param include_body: If True, include the symbol's source code. Use judiciously.
:param include_body: If True, include the symbol's full source code.
:param include_info: whether to include additional info (hover-like, typically including docstring and signature),
about the symbol.
Default False; info is never included for child symbols or if include_body is True.
@@ -122,6 +124,10 @@ class JetBrainsFindSymbolTool(Tool, ToolMarkerSymbolicRead, ToolMarkerOptional):
result = self._to_json(grouped_symbols)
return self._limit_length(result, max_answer_chars, shortened_result_factories=[create_shortened_result])
@classmethod
def get_param_aliases(cls) -> dict[str, str]:
return {"name_path": "name_path_pattern"}
class JetBrainsMoveTool(Tool, ToolMarkerSymbolicEdit, ToolMarkerOptional, ToolMarkerBeta):
"""
@@ -253,7 +259,6 @@ class JetBrainsFindReferencingSymbolsTool(Tool, ToolMarkerSymbolicRead, ToolMark
symbol_dict_grouper = JetBrainsSymbolDictGrouper(["relative_path", "type"], ["type"], collapse_singleton=True)
# TODO: (maybe) - add content snippets showing the references like in LS based version?
def apply(
self,
name_path: str,
@@ -261,11 +266,14 @@ class JetBrainsFindReferencingSymbolsTool(Tool, ToolMarkerSymbolicRead, ToolMark
max_answer_chars: int = -1,
) -> str:
"""
Finds symbols that reference the symbol at the specified symbol, i.e. returns symbols whose definitions (e.g. a function body) contain a reference to the given symbol.
The result will contain metadata about the referencing symbols.
Finds all symbols that reference the given symbol — its callers / usages / dependents, i.e. the
symbols whose own definition (e.g. a method body) contains a reference to it. For each, returns its
name path, file, and the surrounding line of code.
:param name_path: name path of the symbol for which to find references
:param relative_path: the relative path to the file containing the symbol (must be a file, not a directory)
Note: for external dependencies, this must be an identifier starting with `<ext` that you have received
earlier (don't try to guess!).
:param max_answer_chars: max characters for the result (-1 for default). If exceeded, no content/a shortened result is returned.
"""
relative_path = self._sanitize_input_param(relative_path)
@@ -319,22 +327,27 @@ class JetBrainsGetSymbolsOverviewTool(Tool, ToolMarkerSymbolicRead, ToolMarkerOp
def apply(
self,
relative_path: str,
depth: int = 0,
depth: int = -1,
max_answer_chars: int = -1,
include_file_documentation: bool = False,
) -> str:
"""
Gets an overview of the top-level symbols in the given file.
Calling this is often a good idea before more targeted reading, searching or editing operations on the code symbols.
Before requesting a symbol overview, it is usually a good idea to narrow down the scope of the overview
by first understanding the basic directory structure of the repository that you can get from memories
or by using the `list_dir` and `find_file` tools (or similar).
Gets an overview of the top-level symbols defined in the given file (classes, methods, fields) — its
STRUCTURE, without their bodies. This is the cheap, structure-first way to learn what a file
contains: it costs far less context than reading the whole file.
:param relative_path: the relative path to the file to get the overview of
:param depth: depth up to which descendants shall be retrieved (e.g., use 1 to also retrieve immediate children).
:param depth: depth up to which descendants shall be retrieved.
Default (-1) results in a language specific choice: 1 for java and kotlin and 0 for other languages
:param max_answer_chars: max characters for the result (-1 for default). If exceeded, no content/a shortened result is returned.
:param include_file_documentation: whether to include the file's docstring. Default False.
"""
if depth == -1:
if relative_path.endswith((".java", ".kt")):
depth = 1
else:
depth = 0
relative_path = self._sanitize_input_param(relative_path)
with JetBrainsPluginClient.from_project(self.project) as client:
symbol_overview = client.get_symbols_overview(
@@ -579,8 +592,8 @@ class JetBrainsDebugTool(Tool, ToolMarkerOptional, ToolMarkerBeta):
repl_key: str = "default",
) -> str:
"""
Debug code interactively by evaluating Groovy expressions in a persistent REPL.
Important: Debugging should only be applied if the user has requested it!
Debug code by evaluating Groovy/Java expressions in a persistent REPL attached to the IDE's
debugger (run configs, breakpoints, stepping, inspection of live state).
Use the `serena_info` tool with topic `jet_brains_debug_repl` for usage information.
src/serena/tools/query_project_tools.py
+1 -2
View File
@@ -53,7 +53,6 @@ class QueryProjectTool(Tool, ToolMarkerOptional, ToolMarkerDoesNotRequireActiveP
:param tool_params_json: the parameters to pass to the tool, encoded as a JSON string
"""
tool = self.agent.get_tool_by_name(tool_name)
assert tool.is_active(), f"Tool {tool_name} is not active."
assert tool.is_readonly(), f"Tool {tool_name} is not read-only and cannot be executed in another project."
if self._is_project_server_required(tool):
client = ProjectServerClient()
@@ -63,7 +62,7 @@ class QueryProjectTool(Tool, ToolMarkerOptional, ToolMarkerDoesNotRequireActiveP
assert registered_project is not None, f"Project {project_name} is not registered and cannot be queried."
project = registered_project.get_project_instance(self.agent.serena_config)
with tool.agent.active_project_context(project):
return tool.apply(**json.loads(tool_params_json)) # type: ignore
return tool.apply(**json.loads(tool_params_json))
def _is_project_server_required(self, tool: Tool) -> bool:
match self.agent.get_language_backend():
src/serena/tools/symbol_tools.py
+36 -9
View File
@@ -40,20 +40,28 @@ class GetSymbolsOverviewTool(Tool, ToolMarkerSymbolicRead):
symbol_dict_grouper = LanguageServerSymbolDictGrouper(["kind"], ["kind"], collapse_singleton=True)
def apply(self, relative_path: str, depth: int = 0, max_answer_chars: int = -1) -> str:
def apply(self, relative_path: str, depth: int = -1, max_answer_chars: int = -1) -> str:
"""
Use this tool to get a high-level understanding of the code symbols in a file.
This should be the first tool to call when you want to understand a new file, unless you already know
what you are looking for.
:param relative_path: the relative path to the file to get the overview of
:param depth: depth up to which descendants of top-level symbols shall be retrieved
(e.g. 1 retrieves immediate children). Default 0.
:param depth: depth up to which descendants shall be retrieved.
Default (-1) results in a language specific choice: 1 for java and kotlin and 0 for other languages
:param max_answer_chars: if the overview is longer than this number of characters,
no content will be returned. -1 means the default value from the config will be used.
Don't adjust unless there is really no other way to get the content required for the task.
:return: a JSON object containing symbols grouped by kind in a compact format.
"""
# Note: file system sync not required (relevant file is opened in the language server explicitly)
if depth == -1:
if relative_path.endswith((".java", ".kt")):
depth = 1
else:
depth = 0
result = self.get_symbol_overview(relative_path, depth=depth)
# capture kind names and depth-0 snapshots before grouping, which mutates the dicts
@@ -183,6 +191,8 @@ class FindSymbolTool(Tool, ToolMarkerSymbolicRead):
:param max_answer_chars: max result length; -1 for default
:return: symbols (with locations) matching the name.
"""
# Note: file system sync not required; the symbol finder opens all relevant source files explicitly in the case of changes
if include_body:
depth = 0 # ignore user-specified depth if include_body is True
assert max_matches != 0, "max_matches must be > 0 or equal to -1."
@@ -228,12 +238,16 @@ class FindSymbolTool(Tool, ToolMarkerSymbolicRead):
# In python 3.15 we could specify extra_items=True in the TypedDict definition,
# https://peps.python.org/pep-0728/
# If we ever upgrade to 3.15, we can remove the type: ignore[typeddict-unknown-key]
s_dict["info"] = symbol_info # type: ignore[typeddict-unknown-key]
s_dict["info"] = symbol_info
grouped_symbol_dicts = self.symbol_dict_grouper.group(symbol_dicts)
result = self._to_json(grouped_symbol_dicts)
return self._limit_length(result, max_answer_chars, shortened_result_factories=[create_short_result_relative_path_to_name_paths])
@classmethod
def get_param_aliases(cls) -> dict[str, str]:
return {"name_path": "name_path_pattern"}
class FindReferencingSymbolsTool(Tool, ToolMarkerSymbolicRead):
"""
@@ -257,18 +271,20 @@ class FindReferencingSymbolsTool(Tool, ToolMarkerSymbolicRead):
:param name_path: name path of the symbol
:param relative_path: the relative path to the file containing the symbol for which to find references.
Note: for external dependencies, this must be an identifier starting with `<ext` that you have received
earlier (don't try to guess!).
:param include_kinds: (optional) limits results to the given LSP symbol kinds (integers)
:param exclude_kinds: optional list of LSP symbol kinds (integers) to exclude.
:param max_answer_chars: max result length; -1 for default
:return: a list of JSON objects with the symbols referencing the requested symbol
"""
# file system sync needed for case where symbol finder does not perform a global search, updating everything
if relative_path:
self.project.ls_sync_file_system_changes()
include_body = False # It is probably never a good idea to include the body of the referencing symbols
parsed_include_kinds: Sequence[SymbolKind] | None = [SymbolKind(k) for k in include_kinds] if include_kinds else None
parsed_exclude_kinds: Sequence[SymbolKind] | None = [SymbolKind(k) for k in exclude_kinds] if exclude_kinds else None
symbol_retriever = self.create_language_server_symbol_retriever()
symbol_retriever = self.create_language_server_symbol_retriever()
references_in_symbols = symbol_retriever.find_referencing_symbols(
name_path,
relative_file_path=relative_path,
@@ -302,12 +318,12 @@ class FindReferencingSymbolsTool(Tool, ToolMarkerSymbolicRead):
}
)
result = self.symbol_dict_grouper.group(reference_dicts) # type: ignore
result = self.symbol_dict_grouper.group(reference_dicts)
# shortened result closures, from least to most aggressive shortening
def make_refs_without_context() -> str:
"""References with name_path and reference line, without surrounding code lines"""
grouped = self.symbol_dict_grouper.group(copy.deepcopy(ref_summaries)) # type: ignore
grouped = self.symbol_dict_grouper.group(copy.deepcopy(ref_summaries))
return f"References without surrounding lines:\n{self._to_json(grouped)}"
def make_per_file_counts() -> str:
@@ -351,6 +367,8 @@ class FindImplementationsTool(Tool, ToolMarkerSymbolicRead):
:param max_answer_chars: max result length; -1 for default
:return: a list of JSON objects with the symbols implementing the requested symbol
"""
self.project.ls_sync_file_system_changes()
include_body = False
parsed_include_kinds: Sequence[SymbolKind] | None = [SymbolKind(k) for k in include_kinds] if include_kinds else None
parsed_exclude_kinds: Sequence[SymbolKind] | None = [SymbolKind(k) for k in exclude_kinds] if exclude_kinds else None
@@ -404,6 +422,8 @@ class FindDeclarationTool(Tool, ToolMarkerSymbolicRead):
:param include_body: whether to include the symbol's body in the result. Default False.
:param include_info: whether to include additional info (hover-like). Default False.
"""
self.project.ls_sync_file_system_changes()
symbol_retriever = self.create_language_server_symbol_retriever()
relative_path = self._sanitize_input_param(relative_path)
regex = self._sanitize_input_param(regex)
@@ -486,6 +506,8 @@ class GetDiagnosticsForFileTool(Tool, ToolMarkerSymbolicRead):
:param max_answer_chars: max result length; -1 for default
:return: grouped diagnostics for the requested file.
"""
self.project.ls_sync_file_system_changes()
symbol_retriever = self.create_language_server_symbol_retriever()
diagnostics = symbol_retriever.get_file_diagnostics(
relative_file_path=relative_path,
@@ -537,6 +559,8 @@ class GetDiagnosticsForSymbolTool(Tool, ToolMarkerSymbolicRead, ToolMarkerOption
:param max_answer_chars: max result length; -1 for default
:return: grouped diagnostics for the requested symbol and, optionally, its referencing symbols.
"""
self.project.ls_sync_file_system_changes()
symbol_retriever = self.create_language_server_symbol_retriever()
diagnostics_by_symbol = symbol_retriever.get_symbol_diagnostics(
name_path=name_path,
@@ -665,6 +689,7 @@ class RenameSymbolTool(Tool, ToolMarkerSymbolicEdit):
:param new_name: the new name for the symbol
:return: result summary indicating success or failure
"""
self.project.ls_sync_file_system_changes()
code_editor = self.create_ls_code_editor()
status_message = code_editor.rename_symbol(name_path, relative_path=relative_path, new_name=new_name)
return status_message
@@ -683,6 +708,8 @@ class SafeDeleteSymbol(Tool, ToolMarkerSymbolicEdit):
:param name_path_pattern: name path of the symbol to delete
:param relative_path: the relative path to the file containing the symbol to delete
"""
self.project.ls_sync_file_system_changes()
ls_symbol_retriever = self.create_language_server_symbol_retriever()
symbol = ls_symbol_retriever.find_unique(name_path_pattern, substring_matching=False, within_relative_path=relative_path)
symbol_rel_path = symbol.relative_path
src/serena/tools/tools_base.py
+57 -23
View File
@@ -126,6 +126,19 @@ class ApplyMethodProtocol(Protocol):
pass
class ToolCallError(Exception):
"""
Represents an error raised during a tool call execution
"""
def __init__(self, error_message: str):
super().__init__(error_message)
self._error_message = error_message
def get_error_message(self) -> str:
return self._error_message
class Tool(Component):
# NOTE: each tool should implement the apply method, which is then used in
# the central method of the Tool class `apply_ex`.
@@ -233,12 +246,12 @@ class Tool(Component):
"""Gets the docstring for the tool application, used by the MCP server."""
return self.get_apply_docstring_from_cls()
def get_apply_fn_metadata(self) -> FuncMetadata:
def get_apply_fn_metadata(self, structured_output: bool | None = None) -> FuncMetadata:
"""Gets the metadata for the tool application function, used by the MCP server."""
return self.get_apply_fn_metadata_from_cls()
return self.get_apply_fn_metadata_from_cls(structured_output=structured_output)
@classmethod
def get_apply_fn_metadata_from_cls(cls) -> FuncMetadata:
def get_apply_fn_metadata_from_cls(cls, structured_output: bool | None = None) -> FuncMetadata:
"""Get the metadata for the apply method from the class (static metadata).
Needed for creating MCP tools in a separate process without running into serialization issues.
"""
@@ -251,7 +264,7 @@ class Tool(Component):
if apply_fn is None:
raise AttributeError(f"apply method not defined in {cls}. Did you forget to implement it?")
return func_metadata(apply_fn, skip_names=["self", "cls", cls.SESSION_ID_PARAM_NAME])
return func_metadata(apply_fn, skip_names=["self", "cls", cls.SESSION_ID_PARAM_NAME], structured_output=structured_output)
def _log_tool_application(self, frame: Any, session_id: str) -> None:
params = {}
@@ -306,10 +319,24 @@ class Tool(Component):
def is_symbolic(self) -> bool:
return issubclass(self.__class__, ToolMarkerSymbolicRead) or issubclass(self.__class__, ToolMarkerSymbolicEdit)
def apply_ex(self, log_call: bool = True, catch_exceptions: bool = True, mcp_ctx: Context | None = None, **kwargs) -> str: # type: ignore
@classmethod
def get_param_aliases(cls) -> dict[str, str]:
"""
Applies the tool with logging and exception handling, using the given keyword arguments
:return: a mapping of parameter aliases for the apply method, where the key is the alias and the value is the actual parameter name.
This can be used to define alternative parameter names for the same parameter.
"""
return {}
def apply_ex(self, log_call: bool = True, catch_exceptions: bool = True, mcp_ctx: Context | None = None, **kwargs) -> str:
"""
Applies the tool with logging and exception handling, using the given keyword arguments.
This method either returns a string result or raises a ToolCallError in case of an error during tool application
(but if `catch_exception is enabled, it will return the error message as a string instead of raising the exception).
:param log_call: whether to log the tool call and its result
:param catch_exceptions: whether to catch exceptions and return their messages as strings, instead of raising a ToolCallError
"""
# obtain session ID and client info
session_id = "global"
if mcp_ctx is not None:
try:
@@ -329,18 +356,18 @@ class Tool(Component):
try:
if not self.is_active():
return f"Error: Tool '{self.get_name_from_cls()}' is not active. Active tools: {self.agent.get_active_tool_names()}"
except Exception as e:
return f"RuntimeError while checking if tool {self.get_name_from_cls()} is active: {e}"
raise ToolCallError(
f"Tool '{self.get_name_from_cls()}' is not active. Active tools: {self.agent.get_active_tool_names()}"
)
if log_call:
self._log_tool_application(inspect.currentframe(), session_id)
if log_call:
self._log_tool_application(inspect.currentframe(), session_id)
try:
# check whether the tool requires an active project and language server
if not isinstance(self, ToolMarkerDoesNotRequireActiveProject):
if self.agent.get_active_project() is None:
return (
"Error: No active project. Ask the user to provide the project path or to select a project from this list of known projects: "
raise ToolCallError(
"No active project. Ask the user to provide the project path or to select a project from this list of known projects: "
+ f"{self.agent.serena_config.project_names}"
)
@@ -372,12 +399,12 @@ class Tool(Component):
# record tool usage
self.agent.record_tool_usage(apply_kwargs, result, self)
except ToolCallError:
raise
except Exception as e:
if not catch_exceptions:
raise
msg = f"Error executing tool: {e.__class__.__name__} - {e}"
log.error(f"Error executing tool: {e}", exc_info=e)
result = msg
msg = f"{e.__class__.__name__}: {e}"
log.error(msg, exc_info=e)
raise ToolCallError(msg)
if log_call:
log.info(f"Result: {result}")
@@ -392,13 +419,20 @@ class Tool(Component):
return result
# execute the tool in the agent's task executor, with timeout
tool_call_error: ToolCallError
try:
task_exec = self.agent.issue_task(task, name=self.__class__.__name__)
return task_exec.result(timeout=self.agent.serena_config.tool_timeout)
except Exception as e: # typically TimeoutError (other exceptions caught in task)
msg = f"Error: {e.__class__.__name__} - {e}"
except ToolCallError as e:
tool_call_error = e
except Exception as e: # typically TimeoutError (other exceptions caught and forwarded as ToolCallError in the task)
msg = f"{e.__class__.__name__}: {e}"
log.error(msg)
return msg
tool_call_error = ToolCallError(msg)
if catch_exceptions:
return tool_call_error.get_error_message()
else:
raise tool_call_error
@staticmethod
def _to_json(x: Any) -> str:
@@ -434,7 +468,7 @@ class EditingToolWithDiagnostics(Tool, ToolMarkerCanEdit):
def __enter__(self) -> Self:
return self
def __exit__(self, exc_type, exc_val, exc_tb): # type: ignore
def __exit__(self, exc_type, exc_val, exc_tb):
pass
def format_result(
src/serena/util/exception.py
+1 -1
View File
@@ -19,7 +19,7 @@ def is_headless_environment() -> bool:
return False
# Check for DISPLAY variable (required for X11)
if not os.environ.get("DISPLAY"): # type: ignore
if not os.environ.get("DISPLAY"):
return True
# Check for SSH session
src/serena/util/file_proxy.py
+131
View File
@@ -0,0 +1,131 @@
import logging
import os
from abc import ABC, abstractmethod
from collections.abc import Iterator
from typing import TYPE_CHECKING, Self
from serena.jetbrains import jetbrains_types as jb
if TYPE_CHECKING:
from serena.project import Project
log = logging.getLogger(__name__)
class FileProxy(ABC):
@abstractmethod
def get_contents(self) -> str:
""":return: the contents of the file as a string."""
@abstractmethod
def get_relative_path(self) -> str:
""":return: the relative path reported by Serena (actual relative path or encoded external path)"""
@abstractmethod
def is_glob_supported(self):
"""
:return: whether the proxy supports glob filtering based on its relative path
"""
@staticmethod
def is_external_path(relative_path: str) -> bool:
"""
:return: whether the given relative path is an encoded external path (not a local project file)
"""
# This is intended to be extended once we also support external paths in other backends
return jb.is_external_path(relative_path)
@classmethod
def from_project_relative_path(cls, project: "Project", relative_path: str) -> "FileProxy":
if cls.is_external_path(relative_path):
if project.language_backend.is_jetbrains():
return JetBrainsFileProxy(relative_path, project)
return LocalProjectFileProxy(relative_path, project)
class LocalProjectFileProxy(FileProxy):
def __init__(self, relative_path: str, project: "Project"):
self._relative_path = relative_path
self._project = project
def get_contents(self) -> str:
abs_path = os.path.join(self._project.project_root, self._relative_path)
with open(abs_path, encoding=self._project.project_config.encoding) as f:
return f.read()
def get_relative_path(self) -> str:
return self._relative_path
def is_glob_supported(self):
return True
class JetBrainsFileProxy(FileProxy):
"""
Retrieves the contents of a file from the JetBrains plugin via the plugin client, given its relative path,
which may be an external path (e.g., "<ext:FileUtil.class|472e0a13>")
"""
def __init__(self, relative_path: str, project: "Project"):
self._relative_path = relative_path
self._project = project
def get_contents(self) -> str:
from serena.jetbrains.jetbrains_plugin_client import JetBrainsPluginClient
client = JetBrainsPluginClient.from_project(self._project)
return client.read_file(self._relative_path)
def get_relative_path(self) -> str:
return self._relative_path
def is_glob_supported(self):
return False
class FileCollection:
def __init__(self, file_proxies: list[FileProxy]):
self._file_proxies = file_proxies
def __len__(self) -> int:
return len(self._file_proxies)
def __iter__(self) -> Iterator[FileProxy]:
return iter(self._file_proxies)
@classmethod
def from_local_project_paths(cls, relative_paths: list[str], project: "Project") -> Self:
return cls([LocalProjectFileProxy(path, project) for path in relative_paths])
def filter_glob(self, paths_include_glob: str | None = None, paths_exclude_glob: str | None = None) -> "FileCollection":
"""
Filters the collection based on the given patterns.
Note: Filtering is applied only to local project files. Other files are always retained.
:param paths_include_glob: optional glob pattern to include files from the list
:param paths_exclude_glob: optional glob pattern to exclude files from the list
:return: the filtered collection
"""
from serena.util.text_utils import GlobMatcher
if paths_include_glob is None and paths_exclude_glob is None:
return self
include_glob_matcher = GlobMatcher(paths_include_glob) if paths_include_glob else None
exclude_glob_matcher = GlobMatcher(paths_exclude_glob) if paths_exclude_glob else None
filtered_files = []
for f in self._file_proxies:
if f.is_glob_supported():
path = f.get_relative_path()
if include_glob_matcher:
if not include_glob_matcher.matches(path):
log.debug(f"Skipping {path}: does not match include pattern {paths_include_glob}")
continue
if exclude_glob_matcher:
if exclude_glob_matcher.matches(path):
log.debug(f"Skipping {path}: matches exclude pattern {paths_exclude_glob}")
continue
filtered_files.append(f)
return FileCollection(filtered_files)
src/serena/util/file_system.py
+6 -1
View File
@@ -164,7 +164,12 @@ class GitignoreParser:
queue: list[str] = [self.repo_root]
def scan(abs_path: str | None) -> Iterator[str]:
for entry in os.scandir(abs_path):
try:
entries = os.scandir(abs_path)
except PermissionError as ex:
log.debug(f"Skipping unreadable directory {abs_path}: {ex}")
return
for entry in entries:
try:
if entry.is_dir(follow_symlinks=follow_symlinks):
queue.append(entry.path)
src/serena/util/inspection.py
+26 -22
View File
@@ -31,36 +31,40 @@ def determine_programming_language_composition(repo_path: str) -> dict[Language,
"""
Determine the programming language composition of a repository.
:param repo_path: Path to the repository to analyze
Percentages are computed relative to the number of files that match at least
one supported language, not the total file count. This prevents files that
belong to no supported language (images, plain text, licenses, lock files, etc.)
from diluting language percentages in repositories where such files dominate.
:return: Dictionary mapping languages to percentages of files matching each language
:param repo_path: Path to the repository to analyze
:return: Dictionary mapping languages to percentages of recognised source files
matching each language (denominator = files matched by at least one language)
"""
all_files = find_all_non_ignored_files(repo_path)
if not all_files:
return {}
# Count files for each language
# collect all language matchers once
all_languages = list(Language.iter_all(include_experimental=False))
matchers = {lang: lang.get_source_fn_matcher() for lang in all_languages}
# count files per language in a single pass over the files
language_counts: dict[Language, int] = {}
total_files = len(all_files)
for language in Language.iter_all(include_experimental=False):
matcher = language.get_source_fn_matcher()
count = 0
for file_path in all_files:
# Use just the filename for matching, not the full path
filename = os.path.basename(file_path)
recognised_files = 0
for file_path in all_files:
# Use just the filename for matching, not the full path
filename = os.path.basename(file_path)
matched_any = False
for lang, matcher in matchers.items():
if matcher.is_relevant_filename(filename):
count += 1
language_counts[lang] = language_counts.get(lang, 0) + 1
matched_any = True
if matched_any:
recognised_files += 1
if count > 0:
language_counts[language] = count
if recognised_files == 0:
return {}
# Convert counts to percentages
language_percentages: dict[Language, float] = {}
for language, count in language_counts.items():
percentage = (count / total_files) * 100
language_percentages[language] = round(percentage, 2)
return language_percentages
# convert to percentages relative to recognised source files only
return {lang: round(count / recognised_files * 100, 2) for lang, count in language_counts.items()}
src/serena/util/logging.py
+1 -1
View File
@@ -145,7 +145,7 @@ class SuspendedLoggersContext:
root_logger.handlers.clear()
return self
def __exit__(self, exc_type, exc_val, exc_tb) -> None: # type: ignore
def __exit__(self, exc_type, exc_val, exc_tb) -> None:
root_logger = lg.getLogger()
root_logger.handlers = self.saved_root_handlers
if self.saved_root_level is not None:
src/serena/util/shell.py
+1 -1
View File
@@ -47,7 +47,7 @@ def subprocess_check_output(
) -> str:
output = subprocess.check_output(
args, stdin=subprocess.DEVNULL, stderr=subprocess.PIPE, timeout=timeout, env=os.environ.copy(), cwd=cwd, **subprocess_kwargs()
).decode(encoding) # type: ignore
).decode(encoding)
if strip:
output = output.strip()
return output
src/serena/util/text_utils.py
+220 -104
View File
@@ -1,6 +1,6 @@
import fnmatch
import hashlib
import logging
import os
import re
from collections.abc import Callable
from dataclasses import dataclass, field
@@ -10,7 +10,7 @@ from typing import Any, Literal, Self
from bs4 import BeautifulSoup
from joblib import Parallel, delayed
from serena.constants import DEFAULT_SOURCE_FILE_ENCODING
from serena.util.file_proxy import FileCollection, FileProxy
from solidlsp.ls_utils import TextUtils
log = logging.getLogger(__name__)
@@ -100,7 +100,7 @@ class MatchedConsecutiveLines:
def from_file_contents(
cls, file_contents: str, line: int, context_lines_before: int = 0, context_lines_after: int = 0, source_file_path: str | None = None
) -> Self:
line_contents = file_contents.split("\n")
line_contents = TextUtils.split_lines(file_contents)
start_lineno = max(0, line - context_lines_before)
end_lineno = min(len(line_contents) - 1, line + context_lines_after)
text_lines: list[TextLine] = []
@@ -124,7 +124,7 @@ def glob_to_regex(glob_pat: str) -> str:
if ch == "*":
regex_parts.append(".*")
elif ch == "?":
regex_parts.append("..")
regex_parts.append(".")
elif ch == "\\":
i += 1
if i < len(glob_pat):
@@ -141,7 +141,6 @@ def search_text(
pattern: str,
content: str | None = None,
source_file_path: str | None = None,
allow_multiline_match: bool = False,
context_lines_before: int = 0,
context_lines_after: int = 0,
is_glob: bool = False,
@@ -154,8 +153,6 @@ def search_text(
:param content: The text content to search. May be None if source_file_path is provided.
:param source_file_path: Optional path to the source file. If content is None,
this has to be passed and the file will be read.
:param allow_multiline_match: Whether to search across multiple lines. Currently, the default
option (False) is very inefficient, so it is recommended to set this to True.
:param context_lines_before: Number of context lines to include before matches
:param context_lines_after: Number of context lines to include after matches
:param is_glob: If True, pattern is treated as a glob-like pattern (e.g., "*.py", "test_??.py")
@@ -172,76 +169,50 @@ def search_text(
raise ValueError("Pass either content or source_file_path")
matches = []
lines = content.splitlines()
lines = TextUtils.split_lines(content)
total_lines = len(lines)
# Convert pattern to a compiled regex if it's a string
if is_glob:
pattern = glob_to_regex(pattern)
if allow_multiline_match:
# For multiline matches, optionally use DOTALL so '.' matches newlines
flags = (re.MULTILINE | re.DOTALL) if multiline else 0
compiled_pattern = re.compile(pattern, flags)
# Search across the entire content as a single string
for match in compiled_pattern.finditer(content):
start_pos = match.start()
end_pos = match.end()
# Find the line numbers for the start and end positions
start_line_num = content[:start_pos].count("\n")
end_line_num = content[:end_pos].count("\n")
# For multiline matches, optionally use DOTALL so '.' matches newlines
flags = (re.MULTILINE | re.DOTALL) if multiline else 0
compiled_pattern = re.compile(pattern, flags)
# Search across the entire content as a single string
for match in compiled_pattern.finditer(content):
start_pos = match.start()
end_pos = match.end()
# Calculate the range of lines to include in the context
context_start = max(0, start_line_num - context_lines_before)
context_end = min(total_lines - 1, end_line_num + context_lines_after)
# Find the line numbers for the start and end positions
start_line_num = TextUtils.get_line_from_index(content, start_pos)
end_line_num = TextUtils.get_line_from_index(content, end_pos)
if end_line_num > start_line_num and TextUtils.get_line_col_from_index(content, end_pos)[1] == 0:
# `end_pos` is exclusive, so if it is at the start of a line, the match ends with the
# preceding line's newline and does not extend into the line that `end_pos` points to
end_line_num -= 1
# Create TextLine objects for the context
context_lines = []
for line_num in range(context_start, context_end + 1):
if context_start <= line_num < start_line_num:
match_type = LineType.BEFORE_MATCH
elif end_line_num < line_num <= context_end:
match_type = LineType.AFTER_MATCH
else:
match_type = LineType.MATCH
# Calculate the range of lines to include in the context
context_start = max(0, start_line_num - context_lines_before)
context_end = min(total_lines - 1, end_line_num + context_lines_after)
context_lines.append(TextLine(line_number=line_num, line_content=lines[line_num], match_type=match_type))
# Create TextLine objects for the context
context_lines = []
for line_num in range(context_start, context_end + 1):
if context_start <= line_num < start_line_num:
match_type = LineType.BEFORE_MATCH
elif end_line_num < line_num <= context_end:
match_type = LineType.AFTER_MATCH
else:
match_type = LineType.MATCH
matches.append(MatchedConsecutiveLines(lines=context_lines, source_file_path=source_file_path))
else:
# TODO: extremely inefficient! Since we currently don't use this option in SerenaAgent or LanguageServer,
# it is not urgent to fix, but should be either improved or the option should be removed.
# Search line by line, normal compile without DOTALL
compiled_pattern = re.compile(pattern)
for i, line in enumerate(lines):
if compiled_pattern.search(line):
# Calculate the range of lines to include in the context
context_start = max(0, i - context_lines_before)
context_end = min(total_lines - 1, i + context_lines_after)
context_lines.append(TextLine(line_number=line_num, line_content=lines[line_num], match_type=match_type))
# Create TextLine objects for the context
context_lines = []
for j in range(context_start, context_end + 1):
if j < i:
match_type = LineType.BEFORE_MATCH
elif j > i:
match_type = LineType.AFTER_MATCH
else:
match_type = LineType.MATCH
context_lines.append(TextLine(line_number=j, line_content=lines[j], match_type=match_type))
matches.append(MatchedConsecutiveLines(lines=context_lines, source_file_path=source_file_path))
matches.append(MatchedConsecutiveLines(lines=context_lines, source_file_path=source_file_path))
return matches
def default_file_reader(file_path: str) -> str:
"""Reads using the default encoding."""
with open(file_path, encoding=DEFAULT_SOURCE_FILE_ENCODING) as f:
return f.read()
def expand_braces(pattern: str) -> list[str]:
"""
Expands brace patterns in a glob string.
@@ -249,18 +220,22 @@ def expand_braces(pattern: str) -> list[str]:
Handles multiple brace sets as well.
"""
patterns = [pattern]
while any("{" in p for p in patterns):
while any("{" in p or "}" in p for p in patterns):
new_patterns = []
for p in patterns:
match = re.search(r"\{([^{}]+)\}", p)
match = re.search(r"\{([^{}]*)\}", p)
if match:
options_expr = match.group(1)
if not options_expr:
raise ValueError(f"Invalid glob brace expression in {pattern!r}: empty braces are not allowed")
prefix = p[: match.start()]
suffix = p[match.end() :]
options = match.group(1).split(",")
options = options_expr.split(",")
for option in options:
new_patterns.append(f"{prefix}{option}{suffix}")
else:
new_patterns.append(p)
raise ValueError(f"Invalid glob brace expression in {pattern!r}: unmatched brace")
patterns = new_patterns
return patterns
@@ -318,11 +293,17 @@ def glob_match(pattern: str, path: str) -> bool:
return fnmatch.fnmatch(path, pattern)
class GlobMatcher:
def __init__(self, expr: str):
self._patterns = expand_braces(expr)
def matches(self, path: str) -> bool:
return any(glob_match(p, path) for p in self._patterns)
def search_files(
relative_file_paths: list[str],
file_collection: FileCollection,
pattern: str,
root_path: str = "",
file_reader: Callable[[str], str] = default_file_reader,
context_lines_before: int = 0,
context_lines_after: int = 0,
paths_include_glob: str | None = None,
@@ -332,11 +313,8 @@ def search_files(
"""
Search for a pattern in a list of files.
:param relative_file_paths: list of relative file paths in which to search
:param file_collection: the collection of files to search (will be optionally filtered by glob patterns)
:param pattern: pattern to search for
:param root_path: root path to resolve relative paths against (by default, current working directory).
:param file_reader: function to read a file, by default will just use os.open.
All files that can't be read by it will be skipped.
:param context_lines_before: number of context lines to include before matches
:param context_lines_after: number of context lines to include after matches
:param paths_include_glob: optional glob pattern to include files from the list
@@ -344,53 +322,35 @@ def search_files(
:param multiline: whether to apply multi-line matching, enabling the flags re.DOTALL and re.MULTILINE (default: True)
:return: list of MatchedConsecutiveLines objects
"""
# Pre-filter paths (done sequentially to avoid overhead)
# Use proper glob matching instead of gitignore patterns
include_patterns = expand_braces(paths_include_glob) if paths_include_glob else None
exclude_patterns = expand_braces(paths_exclude_glob) if paths_exclude_glob else None
# apply glob filter
file_collection = file_collection.filter_glob(paths_include_glob=paths_include_glob, paths_exclude_glob=paths_exclude_glob)
log.info(f"Processing {len(file_collection)} files.")
filtered_paths = []
for path in relative_file_paths:
if include_patterns:
if not any(glob_match(p, path) for p in include_patterns):
log.debug(f"Skipping {path}: does not match include pattern {paths_include_glob}")
continue
if exclude_patterns:
if any(glob_match(p, path) for p in exclude_patterns):
log.debug(f"Skipping {path}: matches exclude pattern {paths_exclude_glob}")
continue
filtered_paths.append(path)
log.info(f"Processing {len(filtered_paths)} files.")
def process_single_file(path: str) -> dict[str, Any]:
def process_single_file(file_proxy: FileProxy) -> dict[str, Any]:
"""Process a single file - this function will be parallelized."""
relative_path = file_proxy.get_relative_path()
try:
abs_path = os.path.join(root_path, path)
file_content = file_reader(abs_path)
file_content = file_proxy.get_contents()
search_results = search_text(
pattern,
content=file_content,
source_file_path=path,
allow_multiline_match=True,
source_file_path=relative_path,
context_lines_before=context_lines_before,
context_lines_after=context_lines_after,
multiline=multiline,
)
if len(search_results) > 0:
log.debug(f"Found {len(search_results)} matches in {path}")
return {"path": path, "results": search_results, "error": None}
log.debug(f"Found {len(search_results)} matches in {relative_path}")
return {"path": relative_path, "results": search_results, "error": None}
except Exception as e:
log.debug(f"Error processing {path}: {e}")
return {"path": path, "results": [], "error": str(e)}
log.debug(f"Error processing {relative_path}: {e}")
return {"path": relative_path, "results": [], "error": str(e)}
# Execute in parallel using joblib
results = Parallel(
n_jobs=-1,
backend="threading",
)(delayed(process_single_file)(path) for path in filtered_paths)
)(delayed(process_single_file)(file_proxy) for file_proxy in file_collection)
# Collect results and errors
matches = []
@@ -405,7 +365,7 @@ def search_files(
if skipped_file_error_tuples:
log.debug(f"Failed to read {len(skipped_file_error_tuples)} files: {skipped_file_error_tuples}")
log.info(f"Found {len(matches)} total matches across {len(filtered_paths)} files")
log.info(f"Found {len(matches)} total matches across {len(file_collection)} files")
return matches
@@ -528,6 +488,162 @@ class ContentReplacer:
return updated_content
@dataclass
class ReplacementOccurrence:
"""A single prospective replacement of a pattern match within one file."""
occurrence_id: str
"""stable, content-anchored identifier: '<relative_path>:<index_in_file>@<digest>'"""
relative_path: str
index_in_file: int
"""0-based index of this match among the matches within its file (in position order)"""
start: int
"""character offset of the match start within the file content"""
end: int
"""character offset of the match end within the file content"""
matched_text: str
replacement: str
"""the fully expanded replacement text (backreferences already resolved)"""
start_line: int
"""0-based line number of the match start"""
end_line: int
"""0-based line number of the match end"""
is_ambiguous: bool = False
"""whether the pattern matches again within the matched text (possible over-match)"""
class MultiFileContentReplacer:
"""
Occurrence-level counterpart of :class:`ContentReplacer` operating on multiple files:
finds every match of a pattern across a set of file contents, assigns each occurrence a
stable content-anchored id, renders minimal line diffs for previewing, and computes the
updated content of a file for a selected subset of occurrences.
"""
OCCURRENCE_ID_REGEX = re.compile(r"^(?P<path>.+):(?P<index>\d+)@(?P<digest>[0-9a-f]{6})$")
_DIGEST_LEN = 6
def __init__(self, mode: Literal["literal", "regex"], regex_multiline: bool = True):
"""
:param mode: whether the needle is a literal string ("literal") or a regular expression ("regex")
:param regex_multiline: whether to apply multi-line regex matching, enabling the flags re.DOTALL and re.MULTILINE
"""
if mode not in ("literal", "regex"):
raise ValueError(f"Invalid mode: '{mode}', expected 'literal' or 'regex'.")
self.mode = mode
self._flags = (re.MULTILINE | re.DOTALL) if regex_multiline else 0
def _compile(self, needle: str) -> re.Pattern:
return re.compile(re.escape(needle) if self.mode == "literal" else needle, flags=self._flags)
@classmethod
def _digest(cls, matched_text: str) -> str:
return hashlib.sha1(matched_text.encode("utf-8")).hexdigest()[: cls._DIGEST_LEN]
@classmethod
def make_occurrence_id(cls, relative_path: str, index_in_file: int, matched_text: str) -> str:
return f"{relative_path}:{index_in_file}@{cls._digest(matched_text)}"
@staticmethod
def _expand_backreferences(match: re.Match, repl_template: str) -> str:
"""Expands $!1, $!2, ... in the replacement template (same syntax as :class:`ContentReplacer`)."""
def expand(m: re.Match) -> str:
group_value = match.group(int(m.group(1)))
return group_value if group_value is not None else m.group(0)
return re.sub(r"\$!(\d+)", expand, repl_template)
def find_occurrences(self, files: list[tuple[str, str]], needle: str, repl: str) -> list[ReplacementOccurrence]:
"""
Finds all matches of the needle in the given files.
:param files: (relative_path, content) pairs; processed in the given order
:param needle: the search expression (literal string or regex, depending on the mode)
:param repl: the replacement template (may contain $!N backreferences in regex mode)
:return: occurrences in deterministic order (file order, then position within the file)
"""
pattern = self._compile(needle)
occurrences: list[ReplacementOccurrence] = []
for relative_path, content in files:
for index_in_file, match in enumerate(pattern.finditer(content)):
matched_text = match.group(0)
replacement = self._expand_backreferences(match, repl) if self.mode == "regex" else repl
# same over-match heuristic as ContentReplacer: for a multi-line match, the pattern
# matching again within the matched text indicates the match may have swallowed
# more than intended
is_ambiguous = "\n" in matched_text and pattern.search(matched_text[1:]) is not None
occurrences.append(
ReplacementOccurrence(
occurrence_id=self.make_occurrence_id(relative_path, index_in_file, matched_text),
relative_path=relative_path,
index_in_file=index_in_file,
start=match.start(),
end=match.end(),
matched_text=matched_text,
replacement=replacement,
start_line=content.count("\n", 0, match.start()),
end_line=content.count("\n", 0, match.end()),
is_ambiguous=is_ambiguous,
)
)
return occurrences
@staticmethod
def apply_to_content(content: str, occurrences: list[ReplacementOccurrence]) -> str:
"""
Applies the given occurrences (which must have been derived from exactly this content)
and returns the updated content.
"""
for occ in sorted(occurrences, key=lambda o: o.start, reverse=True):
assert content[occ.start : occ.end] == occ.matched_text, (
f"Occurrence {occ.occurrence_id} does not match the content it is being applied to"
)
content = content[: occ.start] + occ.replacement + content[occ.end :]
return content
@staticmethod
def _format_block(block: str, prefix: str, max_lines: int, max_line_chars: int) -> list[str]:
lines = block.split("\n")
shown = lines[:max_lines]
result = []
for line in shown:
if len(line) > max_line_chars:
line = line[:max_line_chars] + f"… (+{len(line) - max_line_chars} chars)"
result.append(f" {prefix} {line}")
if len(lines) > max_lines:
result.append(f" {prefix} … ({len(lines) - max_lines} more lines)")
return result
def render_occurrence_diff(
self, occ: ReplacementOccurrence, content: str, max_lines_per_side: int = 6, max_line_chars: int = 200
) -> str:
"""
Renders a minimal line diff for the occurrence: the full lines spanned by the match,
before and after the replacement.
:param occ: the occurrence (must have been derived from exactly this content)
:param content: the file content the occurrence was found in
:param max_lines_per_side: cap on the number of displayed lines per diff side
:param max_line_chars: cap on the number of displayed characters per line
:return: the rendered diff
"""
line_start = content.rfind("\n", 0, occ.start) + 1
line_end = content.find("\n", occ.end)
if line_end == -1:
line_end = len(content)
old_block = content[line_start:line_end]
new_block = content[line_start : occ.start] + occ.replacement + content[occ.end : line_end]
location = f"line {occ.start_line}" if occ.start_line == occ.end_line else f"lines {occ.start_line}-{occ.end_line}"
header = f" [{occ.occurrence_id}] {location}"
if occ.is_ambiguous:
header += " (WARNING: the pattern matches again inside this match — possible over-match, verify the diff)"
diff_lines = [header]
diff_lines += self._format_block(old_block, "-", max_lines_per_side, max_line_chars)
diff_lines += self._format_block(new_block, "+", max_lines_per_side, max_line_chars)
return "\n".join(diff_lines)
@dataclass
class TextCoords:
line: int
src/serena/util/yaml.py
+50 -4
View File
@@ -1,5 +1,7 @@
import logging
import os
import tempfile
import time
from collections.abc import Sequence
from enum import Enum
from typing import Any
@@ -68,7 +70,17 @@ def load_yaml(path: str, comment_normalisation: YamlCommentNormalisation = YamlC
"""
with open(path, encoding=SERENA_FILE_ENCODING) as f:
yaml = _create_yaml(preserve_comments=True)
commented_map: CommentedMap | None = yaml.load(f)
try:
commented_map: CommentedMap | None = yaml.load(f)
except Exception as e:
msg = str(e)
if "undefined alias" in msg:
raise ValueError(
f"Invalid YAML in {path}: values that start with `*` must be quoted. "
"This often happens in `ignored_paths` when using gitignore-style globs like "
'`"**/bin/**"` or `"**/obj/**"`.'
) from e
raise
if commented_map is None: # ruamel returns None for empty documents, but we want an empty CommentedMap
commented_map = CommentedMap()
normalise_yaml_comments(commented_map, comment_normalisation)
@@ -199,9 +211,43 @@ def normalise_yaml_comments(commented_map: CommentedMap, comment_normalisation:
def save_yaml(path: str, data: dict | CommentedMap, preserve_comments: bool = True) -> None:
yaml = _create_yaml(preserve_comments)
os.makedirs(os.path.dirname(path), exist_ok=True)
with open(path, "w", encoding=SERENA_FILE_ENCODING) as f:
yaml.dump(data, f)
target_dir = os.path.dirname(path)
os.makedirs(target_dir, exist_ok=True)
# Atomic write: dump to a temp file in the SAME directory, then os.replace onto the target.
# A plain truncate-and-write (open(path, "w")) is NOT atomic: a concurrent writer (e.g. a second
# Serena process updating the auto-managed registered-projects list) or an interrupted write can
# leave the file half-overwritten — writing a shorter value over a longer one leaves a stale tail,
# which corrupts the YAML so it no longer parses and every later load fails. temp + os.replace makes
# each write all-or-nothing (last-writer-wins, never a corrupt interleave).
fd, tmp = tempfile.mkstemp(dir=target_dir, prefix=os.path.basename(path) + ".", suffix=".tmp")
try:
with os.fdopen(fd, "w", encoding=SERENA_FILE_ENCODING) as f:
yaml.dump(data, f)
_replace_with_retry(tmp, path)
except BaseException:
try:
os.unlink(tmp)
except OSError:
pass
raise
def _replace_with_retry(src: str, dst: str, *, attempts: int = 10, delay_s: float = 0.05) -> None:
"""``os.replace(src, dst)`` with a short retry on a Windows sharing violation.
On Windows the atomic rename fails with ``PermissionError`` (WinError 5/32) if another process
momentarily holds ``dst`` open — e.g. a second Serena process reading or replacing the same
config. A brief bounded retry rides out that contention; the temp file is still complete, so we
never fall back to a non-atomic write.
"""
for attempt in range(attempts):
try:
os.replace(src, dst)
return
except PermissionError:
if attempt == attempts - 1:
raise
time.sleep(delay_s)
def yaml_comment_entry_is_empty(comment_entry: Any) -> bool:
src/solidlsp/dependency_provider.py
+241
View File
@@ -0,0 +1,241 @@
import logging
import os
import shutil
from abc import ABC, abstractmethod
from collections.abc import Sequence
from solidlsp.settings import SolidLSPSettings
log = logging.getLogger(__name__)
class LanguageServerDependencyProvider(ABC):
"""
Prepares dependencies for a language server (if any), ultimately enabling the launch command to be constructed
and optionally providing environment variables that are necessary for the execution.
"""
def __init__(self, custom_settings: SolidLSPSettings.CustomLSSettings, ls_resources_dir: str):
"""
:param custom_settings: the (user-provided) language server-specific settings
:param ls_resources_dir: the directory in which data for the language-server shall be stored
(this is already specific to the concrete language server, i.e. no further subdirectory is needed)
"""
self._custom_settings = custom_settings
self._ls_resources_dir = ls_resources_dir
@abstractmethod
def create_launch_command(self) -> list[str]:
"""
Creates the launch command for this language server, potentially downloading and installing dependencies
beforehand.
:return: the launch command as a list containing the executable and its arguments
"""
def create_launch_command_env(self) -> dict[str, str]:
"""
Provides environment variables to be set when executing the launch command.
This method is intended to be overridden by subclasses that need to set variables.
:return: a mapping for variable names to values
"""
return {}
class LanguageServerDependencyProviderBaseCommand(LanguageServerDependencyProvider, ABC):
"""
Special case of a dependency provider, where the launch command is constructed from a base command.
The user can configure aspects of the launch command in LS-specific settings:
* ``ls_base_cmd`: overrides the base command
* ``ls_path``: overrides the path of the language server's core dependency (e.g. its executable or a JAR file),
from which the base command is formed, bypassing Serena's managed installation
* ``ls_args``: overrides the arguments that are added to the base command in order to form the launch command
* ``ls_extra_args``: additional arguments to append to the launch command
"""
@abstractmethod
def _create_default_base_command(self) -> list[str]:
"""
Obtains the default base command for this language server, potentially downloading and installing dependencies
beforehand.
Note: The user can override the base command, so this will only be run if no custom base command is provided.
:return: the base command as a list containing the executable and its arguments
"""
@abstractmethod
def _create_launch_command_from_base_command(self, base_command: list[str]) -> list[str]:
"""
Adds any additional arguments to the base command to create the final launch command.
:param base_command: the base command
:return: the extended command
"""
def create_launch_command(self) -> list[str]:
# obtain base command
base_command = self._custom_settings.get("ls_base_cmd", None)
if base_command is not None and not isinstance(base_command, list):
log.warning("The 'ls_base_cmd' setting should be a list of strings. Ignoring the provided value: %s", base_command)
base_command = None
if base_command is None:
ls_path = self._custom_settings.get("ls_path", None)
if ls_path is not None:
base_command = [ls_path]
else:
# default case: base command is constructed by the provider implementation
base_command = self._create_default_base_command()
# create launch command from base command
ls_args = self._custom_settings.get("ls_args")
if ls_args is not None:
if not isinstance(ls_args, list):
log.warning("The 'ls_args' setting should be a list of strings. Ignoring the provided value: %s", ls_args)
ls_args = None
if ls_args is not None:
cmd = list(base_command) + ls_args
else:
cmd = self._create_launch_command_from_base_command(list(base_command))
# add user-provided extra arguments (if any)
ls_extra_args = self._custom_settings.get("ls_extra_args", [])
if ls_extra_args:
if not isinstance(ls_extra_args, list):
log.warning("The 'ls_extra_args' setting should be a list of strings. Ignoring the provided value: %s", ls_extra_args)
else:
cmd = cmd + ls_extra_args
return cmd
class LanguageServerDependencyProviderSinglePath(LanguageServerDependencyProviderBaseCommand, ABC):
"""
Special case of a dependency provider, where there is a single core dependency which provides
the basis for the launch command.
The core dependency's path can be overridden by the user in LS-specific settings (SerenaConfig)
via the key "ls_path". If the user provides the key, the specified path is used directly.
Otherwise, the provider implementation is called to get or install the core dependency.
Note: The inheritance from the BaseCommand class serves to allow user overrides to be handled
centrally, but implementations of this class do not necessarily follow the principle that
the base command is strictly extended.
Yet incompatibility arises only if (a) the user overrides the base command with a command that
has arguments, and (b) the implementation of this class does not construct the launch command
by appending arguments to the core dependency's path.
"""
@abstractmethod
def _get_or_install_core_dependency(self) -> str:
"""
Gets the language server's core path, potentially installing dependencies beforehand.
:return: the core dependency's path (e.g. executable, jar, etc.)
"""
@abstractmethod
def _create_launch_command(self, core_path: str) -> list[str]:
"""
:param core_path: path to the core dependency
:return: the launch command as a list containing the executable and its arguments
"""
def _create_default_base_command(self) -> list[str]:
# We treat the core path as the only element of the base command,
# noting that the construction of the launch command from the base command
# is not necessarily to append arguments only.
core_path = self._get_or_install_core_dependency()
return [core_path]
def _create_launch_command_from_base_command(self, base_command: list[str]) -> list[str]:
core_path = base_command[0]
cmd = self._create_launch_command(core_path)
if len(base_command) == 1:
# This is the regular case, where the base command consists only of the core
# dependency's path, and the launch command is constructed from it.
return cmd
else:
# In this case, the user has overridden the base command via LS-specific settings
# with a command that has arguments and therefore does not map cleanly to
# the single path assumption. However, in special cases where the full command
# was constructed by appending arguments to the core dependency's path,
# we simply assume that the provided base command can be substituted.
if cmd[0] == core_path:
return base_command + cmd[1:]
else:
raise ValueError("Language server base launch command with arguments unsupported")
class LanguageServerDependencyProviderUvx(LanguageServerDependencyProviderBaseCommand):
"""
Dependency provider for language servers distributed as a PyPI package, run on demand via ``uvx`` / ``uv x``.
The pinned package version can be overridden by the user via the LS-specific setting given by
``version_setting_key``. Alternatively, the LS-specific setting "ls_path" can be set to the path of an
already-installed language server executable, in which case it is launched directly, bypassing uv entirely.
"""
DEFAULT_UVX_PYTHON_VERSION = "3.13"
def __init__(
self,
custom_settings: "SolidLSPSettings.CustomLSSettings",
ls_resources_dir: str,
*,
package: str,
entrypoint: str,
default_version: str,
version_setting_key: str,
extra_args: Sequence[str] = (),
):
"""
:param package: the PyPI package name (e.g. ``"pyright"``)
:param entrypoint: the console script provided by the package (e.g. ``"pyright-langserver"``)
:param default_version: the package version to pin unless overridden
:param version_setting_key: the LS-specific setting key through which the user can override the version
:param extra_args: arguments appended after the entrypoint (e.g. ``("--stdio",)``)
"""
super().__init__(custom_settings, ls_resources_dir)
self._package = package
self._entrypoint = entrypoint
self._default_version = default_version
self._version_setting_key = version_setting_key
self._extra_args = tuple(extra_args)
@staticmethod
def _build_uvx_base_command(
package: str,
version: str,
entrypoint: str,
python_version: str = DEFAULT_UVX_PYTHON_VERSION,
) -> list[str]:
"""Build a command that runs a pinned PyPI package's console script on demand via ``uvx`` / ``uv tool run``.
:param package: PyPI package name (e.g. ``"pyright"``).
:param version: Pinned package version.
:param entrypoint: Console script provided by the package (e.g. ``"pyright-langserver"``).
:param python_version: Python interpreter version passed via ``-p`` (uv will fetch it if missing).
"""
base_args = ["-p", python_version, "--from", f"{package}=={version}", entrypoint]
uvx_path = os.environ.get("UVX") or shutil.which("uvx")
if uvx_path is not None:
return [uvx_path, *base_args]
uv_path = shutil.which("uv")
if uv_path is not None:
return [uv_path, "tool", "run", *base_args] # `uv tool run` is the same as `uvx`
raise RuntimeError("Could not find 'uvx' or 'uv' in PATH. Install uv (https://docs.astral.sh/uv/).")
def _create_default_base_command(self):
version = self._custom_settings.get(self._version_setting_key, self._default_version)
return self._build_uvx_base_command(self._package, version, self._entrypoint)
def _create_launch_command_from_base_command(self, base_command: list[str]) -> list[str]:
return base_command + list(self._extra_args)
src/solidlsp/initialize_params.py
+82
View File
@@ -0,0 +1,82 @@
import logging
import os
import pathlib
from abc import ABC, abstractmethod
from typing import TYPE_CHECKING, Any, Self, cast
from solidlsp.lsp_protocol_handler.lsp_types import InitializeParams, WorkspaceFolder
if TYPE_CHECKING:
from solidlsp import SolidLanguageServer
log = logging.getLogger(__name__)
class InitializeParamsBuilder(ABC):
def __init__(self):
self._params = {}
def with_base_options(self, options: dict | InitializeParams) -> Self:
self._params.update(options)
return self
def _set(self, key: str, value: Any) -> None:
if key in self._params:
log.debug("Overriding existing option '%s' with new value: %s (old value: %s)", key, value, self._params[key])
self._params[key] = value
@abstractmethod
def _apply_updates(self) -> None:
"""
Applies implementation-specific updates to the options.
"""
def build(self) -> InitializeParams:
self._apply_updates()
return cast(InitializeParams, self._params)
class DefaultInitializeParamsBuilder(InitializeParamsBuilder):
def __init__(self, ls: "SolidLanguageServer", set_workspace_folders: bool = True):
super().__init__()
self._ls = ls
self._set_workspace_folders = set_workspace_folders
@staticmethod
def _create_workspace_folder_entry(path: str) -> WorkspaceFolder:
abs_path = os.path.abspath(path)
return {"uri": pathlib.Path(abs_path).as_uri(), "name": os.path.basename(abs_path)}
def _apply_updates(self):
root_abs_path = self._ls.repository_root_path
self._set("processId", os.getpid())
self._set("rootPath", root_abs_path)
self._set("rootUri", pathlib.Path(root_abs_path).as_uri())
self._set("clientInfo", {"name": "Serena"})
if self._set_workspace_folders:
abs_workspace_paths = self._ls.config.get_absolute_workspace_folders(root_abs_path)
log.info("Workspace folders: %s", abs_workspace_paths)
workspace_folders = [self._create_workspace_folder_entry(abs_path) for abs_path in abs_workspace_paths]
additional_abs_workspace_paths = [
p for p in self._ls.config.get_absolute_additional_workspace_folders(root_abs_path) if p not in abs_workspace_paths
]
if additional_abs_workspace_paths:
log.info("Additional workspace folders (not indexed by SolidLSP): %s", additional_abs_workspace_paths)
workspace_folders.extend([self._create_workspace_folder_entry(abs_path) for abs_path in additional_abs_workspace_paths])
self._set("workspaceFolders", workspace_folders)
init_options_key = "initializationOptions"
custom_init_options = self._ls.custom_settings.get(init_options_key, None)
if custom_init_options:
if not isinstance(custom_init_options, dict):
log.error("Custom initialization options should be a dictionary, but got %s. Ignoring.", type(custom_init_options).__name__)
else:
log.info("Applying custom initialization options: %s", custom_init_options)
if init_options_key in self._params:
self._params[init_options_key].update(custom_init_options)
else:
self._params[init_options_key] = custom_init_options
log.info("Final initialization options: %s", self._params[init_options_key])
src/solidlsp/language_servers/ada_language_server.py
+3 -13
View File
@@ -9,15 +9,12 @@ pragmas/aspects in source rather than file extension, so a single
import logging
import os
import pathlib
import threading
from typing import cast
from overrides import override
from solidlsp.ls import LanguageServerDependencyProvider, LanguageServerDependencyProviderSinglePath, SolidLanguageServer
from solidlsp.ls_config import LanguageServerConfig
from solidlsp.lsp_protocol_handler.lsp_types import InitializeParams
from solidlsp.settings import SolidLSPSettings
from .common import RuntimeDependency, RuntimeDependencyCollection
@@ -155,15 +152,9 @@ class AdaLanguageServer(SolidLanguageServer):
def _create_dependency_provider(self) -> LanguageServerDependencyProvider:
return self.DependencyProvider(self._custom_settings, self._ls_resources_dir)
def _get_initialize_params(self) -> InitializeParams:
def _create_base_initialize_params(self) -> dict:
"""Returns the initialization params for the Ada Language Server."""
repository_absolute_path = self.repository_root_path
root_uri = pathlib.Path(repository_absolute_path).as_uri()
result = {
"processId": os.getpid(),
"rootPath": repository_absolute_path,
"rootUri": root_uri,
"capabilities": {
"workspace": {
"applyEdit": True,
@@ -188,9 +179,8 @@ class AdaLanguageServer(SolidLanguageServer):
},
"initializationOptions": {},
"trace": "off",
"workspaceFolders": [{"uri": root_uri, "name": os.path.basename(repository_absolute_path)}],
}
return cast(InitializeParams, result)
return result
def _start_server(self) -> None:
"""Start the Ada Language Server process and complete the LSP handshake."""
@@ -213,7 +203,7 @@ class AdaLanguageServer(SolidLanguageServer):
log.info("Starting ada_language_server process")
self.server.start()
initialize_params = self._get_initialize_params()
initialize_params = self._create_initialize_params()
log.info("Sending initialize request from LSP client to LSP server and awaiting response")
init_response = self.server.send.initialize(initialize_params)
src/solidlsp/language_servers/al_language_server.py
+4 -14
View File
@@ -7,7 +7,6 @@ You can pass the following entries in ``ls_specific_settings["al"]``:
import logging
import os
import pathlib
import platform
import re
import stat
@@ -448,20 +447,12 @@ class ALLanguageServer(SolidLanguageServer):
return None
@staticmethod
def _get_initialize_params(repository_absolute_path: str) -> dict:
def _create_base_initialize_params(self) -> dict:
"""
Returns the initialize params for the AL Language Server.
"""
# Ensure we have an absolute path for URI generation
repository_path = pathlib.Path(repository_absolute_path).resolve()
root_uri = repository_path.as_uri()
# AL requires extensive capabilities based on VS Code trace
initialize_params = {
"processId": os.getpid(),
"rootPath": str(repository_path),
"rootUri": root_uri,
"capabilities": {
"workspace": {
"applyEdit": True,
@@ -512,7 +503,6 @@ class ALLanguageServer(SolidLanguageServer):
},
},
"trace": "verbose",
"workspaceFolders": [{"uri": root_uri, "name": repository_path.name}],
}
return initialize_params
@@ -556,12 +546,12 @@ class ALLanguageServer(SolidLanguageServer):
self.server.start()
# Send initialize request
initialize_params = self._get_initialize_params(self.repository_root_path)
initialize_params = self._create_initialize_params()
log.info("Sending initialize request from LSP client to AL LSP server and awaiting response")
# Send initialize and wait for response
resp = self.server.send_request("initialize", initialize_params)
resp = self.server.send_request("initialize", dict(initialize_params))
if resp is None:
raise RuntimeError("AL Language Server initialization failed - no response")
@@ -1038,7 +1028,7 @@ class ALLanguageServer(SolidLanguageServer):
if original_name != normalized_name and self._AL_OBJECT_NAME_PATTERN.match(original_name):
sel_range = symbol.get("selectionRange")
if sel_range:
start = sel_range.get("start") # type: ignore
start = sel_range.get("start")
if start and "line" in start and "character" in start:
line = start["line"]
char = start["character"]
src/solidlsp/language_servers/angular_language_server.py
+6 -16
View File
@@ -70,7 +70,7 @@ from solidlsp.language_servers.typescript_language_server import (
from solidlsp.language_servers.vscode_html_language_server import VsCodeHtmlLanguageServer
from solidlsp.ls import LanguageServerDependencyProvider, LSPFileBuffer, SolidLanguageServer
from solidlsp.ls_config import FilenameMatcher, Language, LanguageServerConfig
from solidlsp.lsp_protocol_handler.lsp_types import DocumentSymbol, InitializeParams, SymbolInformation
from solidlsp.lsp_protocol_handler.lsp_types import DocumentSymbol, SymbolInformation
from solidlsp.lsp_protocol_handler.server import ProcessLaunchInfo
from solidlsp.settings import SolidLSPSettings
@@ -166,8 +166,8 @@ class AngularTypeScriptServer(TypeScriptLanguageServer):
)
@override
def _get_initialize_params(self, repository_absolute_path: str) -> InitializeParams:
params = super()._get_initialize_params(repository_absolute_path)
def _create_base_initialize_params(self) -> dict:
params = super()._create_base_initialize_params()
# Load @angular/language-service as a tsserver plugin via typescript-language-server's
# initializationOptions.plugins API (the same API Vue uses for @vue/typescript-plugin).
params["initializationOptions"] = {
@@ -502,8 +502,7 @@ class AngularLanguageServer(SolidLanguageServer):
else:
log.debug("Found @angular/core at %s", found)
def _get_initialize_params(self, repository_absolute_path: str) -> InitializeParams:
root_uri = pathlib.Path(repository_absolute_path).as_uri()
def _create_base_initialize_params(self) -> dict:
params: dict = {
"locale": "en",
"capabilities": {
@@ -531,17 +530,8 @@ class AngularLanguageServer(SolidLanguageServer):
"tsProbeLocations": [os.path.join(self._install_dir, "node_modules")],
"forceStrictTemplates": False,
},
"processId": os.getpid(),
"rootPath": repository_absolute_path,
"rootUri": root_uri,
"workspaceFolders": [
{
"uri": root_uri,
"name": os.path.basename(repository_absolute_path),
}
],
}
return params # type: ignore[return-value]
return params
@override
def _start_server(self) -> None:
@@ -580,7 +570,7 @@ class AngularLanguageServer(SolidLanguageServer):
try:
log.info("Starting Angular language server (ngserver)")
self.server.start()
init_params = self._get_initialize_params(self.repository_root_path)
init_params = self._create_initialize_params()
init_response = self.server.send.initialize(init_params)
log.debug("Angular LS initialize response: %s", init_response)
self.server.notify.initialized({})
src/solidlsp/language_servers/ansible_language_server.py
+3 -16
View File
@@ -6,7 +6,6 @@ Contains various configurations and settings specific to Ansible YAML files (pla
import fnmatch
import logging
import os
import pathlib
import shutil
from typing import Any, ClassVar
@@ -15,7 +14,6 @@ from overrides import override
from solidlsp.language_servers.common import RuntimeDependency, RuntimeDependencyCollection, build_npm_install_command
from solidlsp.ls import LanguageServerDependencyProvider, LanguageServerDependencyProviderSinglePath, SolidLanguageServer
from solidlsp.ls_config import LanguageServerConfig
from solidlsp.lsp_protocol_handler.lsp_types import InitializeParams
from solidlsp.settings import SolidLSPSettings
log = logging.getLogger(__name__)
@@ -217,14 +215,12 @@ class AnsibleLanguageServer(SolidLanguageServer):
def _create_launch_command(self, core_path: str) -> list[str]:
return [core_path, "--stdio"]
def _get_initialize_params(self, repository_absolute_path: str) -> InitializeParams:
def _create_base_initialize_params(self) -> dict:
"""Returns the initialize params for the Ansible Language Server.
Reads shortcut keys and the ``ansible_settings`` dict from ``_custom_settings``
to build ``initializationOptions``.
"""
root_uri = pathlib.Path(repository_absolute_path).as_uri()
# default ansible settings, populated from shortcut keys
ansible_settings: dict[str, Any] = {
"ansible": {
@@ -282,18 +278,9 @@ class AnsibleLanguageServer(SolidLanguageServer):
"symbol": {"dynamicRegistration": True},
},
},
"processId": os.getpid(),
"rootPath": repository_absolute_path,
"rootUri": root_uri,
"workspaceFolders": [
{
"uri": root_uri,
"name": os.path.basename(repository_absolute_path),
}
],
"initializationOptions": {"ansible": ansible_settings},
}
return initialize_params # type: ignore
return initialize_params
def _start_server(self) -> None:
"""Starts the Ansible Language Server, waits for the server to be ready."""
@@ -325,7 +312,7 @@ class AnsibleLanguageServer(SolidLanguageServer):
log.info("Starting Ansible server process")
self.server.start()
initialize_params = self._get_initialize_params(self.repository_root_path)
initialize_params = self._create_initialize_params()
log.info("Sending initialize request from LSP client to LSP server and awaiting response")
init_response = self.server.send.initialize(initialize_params)
src/solidlsp/language_servers/bash_language_server.py
+3 -16
View File
@@ -5,7 +5,6 @@ Contains various configurations and settings specific to Bash scripting.
import logging
import os
import pathlib
import shutil
import threading
@@ -19,7 +18,6 @@ from solidlsp.ls import (
)
from solidlsp.ls_config import LanguageServerConfig
from solidlsp.ls_utils import FileUtils, PlatformId, PlatformUtils
from solidlsp.lsp_protocol_handler.lsp_types import InitializeParams
from solidlsp.settings import SolidLSPSettings
log = logging.getLogger(__name__)
@@ -208,12 +206,10 @@ class BashLanguageServer(SolidLanguageServer):
def _create_launch_command(self, core_path: str) -> list[str]:
return [core_path, "start"]
@staticmethod
def _get_initialize_params(repository_absolute_path: str) -> InitializeParams:
def _create_base_initialize_params(self) -> dict:
"""
Returns the initialize params for the Bash Language Server.
"""
root_uri = pathlib.Path(repository_absolute_path).as_uri()
initialize_params = {
"locale": "en",
"capabilities": {
@@ -237,17 +233,8 @@ class BashLanguageServer(SolidLanguageServer):
"symbol": {"dynamicRegistration": True},
},
},
"processId": os.getpid(),
"rootPath": repository_absolute_path,
"rootUri": root_uri,
"workspaceFolders": [
{
"uri": root_uri,
"name": os.path.basename(repository_absolute_path),
}
],
}
return initialize_params # type: ignore
return initialize_params
def _start_server(self) -> None:
"""
@@ -283,7 +270,7 @@ class BashLanguageServer(SolidLanguageServer):
log.info("Starting Bash server process")
self.server.start()
initialize_params = self._get_initialize_params(self.repository_root_path)
initialize_params = self._create_initialize_params()
log.info("Sending initialize request from LSP client to LSP server and awaiting response")
init_response = self.server.send.initialize(initialize_params)
src/solidlsp/language_servers/bsl_language_server.py
+5 -15
View File
@@ -14,7 +14,6 @@ You can configure the following options in ls_specific_settings (in serena_confi
import logging
import os
import pathlib
import re
import shutil
import subprocess
@@ -27,7 +26,6 @@ from solidlsp.ls import (
SolidLanguageServer,
)
from solidlsp.ls_config import LanguageServerConfig
from solidlsp.lsp_protocol_handler.lsp_types import InitializeParams
from solidlsp.settings import SolidLSPSettings
log = logging.getLogger(__name__)
@@ -166,10 +164,8 @@ class BSLLanguageServer(SolidLanguageServer):
_verify_java_available()
return ["java", "-jar", core_path]
@staticmethod
def _get_initialize_params(repository_absolute_path: str) -> InitializeParams:
root_uri = pathlib.Path(repository_absolute_path).as_uri()
return { # type: ignore[return-value]
def _create_base_initialize_params(self) -> dict:
return {
"locale": "en",
"capabilities": {
"textDocument": {
@@ -179,9 +175,9 @@ class BSLLanguageServer(SolidLanguageServer):
"documentSymbol": {
"dynamicRegistration": True,
"hierarchicalDocumentSymbolSupport": True,
"symbolKind": {"valueSet": list(range(1, 27))}, # type: ignore
"symbolKind": {"valueSet": list(range(1, 27))},
},
"hover": {"dynamicRegistration": True, "contentFormat": ["markdown", "plaintext"]}, # type: ignore
"hover": {"dynamicRegistration": True, "contentFormat": ["markdown", "plaintext"]},
"rename": {"dynamicRegistration": True, "prepareSupport": True},
},
"workspace": {
@@ -190,12 +186,6 @@ class BSLLanguageServer(SolidLanguageServer):
"symbol": {"dynamicRegistration": True},
},
},
"processId": os.getpid(),
"rootPath": repository_absolute_path,
"rootUri": root_uri,
"workspaceFolders": [
{"uri": root_uri, "name": os.path.basename(repository_absolute_path)},
],
}
def _start_server(self) -> None:
@@ -216,7 +206,7 @@ class BSLLanguageServer(SolidLanguageServer):
log.info("Starting BSL language server process")
self.server.start()
init_params = self._get_initialize_params(self.repository_root_path)
init_params = self._create_initialize_params()
init_response = self.server.send.initialize(init_params)
log.debug("BSL LSP initialize response: %s", init_response)
src/solidlsp/language_servers/ccls_language_server.py
+10 -22
View File
@@ -29,20 +29,18 @@ https://github.com/MaskRay/ccls
"""
import logging
import os
import pathlib
import threading
from typing import Any, cast
from typing import Any
from overrides import override
from solidlsp.language_servers.common import UE_IGNORED_DIRNAMES, is_unreal_engine_project
from solidlsp.ls import (
LanguageServerDependencyProvider,
LanguageServerDependencyProviderSinglePath,
SolidLanguageServer,
)
from solidlsp.ls_config import LanguageServerConfig
from solidlsp.lsp_protocol_handler.lsp_types import InitializeParams
from solidlsp.settings import SolidLSPSettings
log = logging.getLogger(__name__)
@@ -71,10 +69,11 @@ class CCLS(SolidLanguageServer):
@override
def is_ignored_dirname(self, dirname: str) -> bool:
ignored_dirs = [
".ccls-cache",
]
return super().is_ignored_dirname(dirname) or dirname in ignored_dirs
return (
super().is_ignored_dirname(dirname)
or dirname == ".ccls-cache"
or (is_unreal_engine_project(self.repository_root_path) and dirname in UE_IGNORED_DIRNAMES)
)
class DependencyProvider(LanguageServerDependencyProviderSinglePath):
def _get_or_install_core_dependency(self) -> str:
@@ -103,12 +102,10 @@ class CCLS(SolidLanguageServer):
def _create_launch_command(self, core_path: str) -> list[str]:
return [core_path]
@staticmethod
def _get_initialize_params(repository_absolute_path: str) -> InitializeParams:
def _create_base_initialize_params(self) -> dict:
"""
Returns the initialize params for the ccls Language Server.
"""
root_uri = pathlib.Path(repository_absolute_path).as_uri()
initialize_params = {
"locale": "en",
"capabilities": {
@@ -121,18 +118,9 @@ class CCLS(SolidLanguageServer):
},
"workspace": {"workspaceFolders": True, "didChangeConfiguration": {"dynamicRegistration": True}},
},
"processId": os.getpid(),
"rootPath": repository_absolute_path,
"rootUri": root_uri,
"workspaceFolders": [
{
"uri": root_uri,
"name": os.path.basename(repository_absolute_path),
}
],
# ccls supports initializationOptions but none are required for basic functionality
}
return cast(InitializeParams, initialize_params)
return initialize_params
def _start_server(self) -> None:
"""
@@ -152,7 +140,7 @@ class CCLS(SolidLanguageServer):
log.info("Starting ccls server process")
self.server.start()
initialize_params = self._get_initialize_params(self.repository_root_path)
initialize_params = self._create_initialize_params()
log.info("Sending initialize request from LSP client to ccls and awaiting response")
self.server.send.initialize(initialize_params)
src/solidlsp/language_servers/clangd_language_server.py
+42 -28
View File
@@ -2,19 +2,19 @@ import hashlib
import json
import logging
import os
import pathlib
import threading
from collections.abc import Hashable
from typing import Any, cast
from typing import Any
from overrides import override
from solidlsp.language_servers.common import UE_IGNORED_DIRNAMES
from solidlsp.ls import LanguageServerDependencyProvider, LanguageServerDependencyProviderSinglePath, ProcessLaunchInfo, SolidLanguageServer
from solidlsp.ls_config import LanguageServerConfig
from solidlsp.lsp_protocol_handler.lsp_types import InitializeParams
from solidlsp.ls_exceptions import SolidLSPException
from solidlsp.settings import SolidLSPSettings
from .common import RuntimeDependency, RuntimeDependencyCollection
from .common import RuntimeDependency, RuntimeDependencyCollection, is_unreal_engine_project
log = logging.getLogger(__name__)
@@ -78,9 +78,9 @@ class ClangdLanguageServer(SolidLanguageServer):
self.resolve_main_method_available = threading.Event()
@override
def _document_symbols_cache_fingerprint(self) -> Hashable:
def _raw_document_symbols_cache_fingerprint(self) -> Hashable:
cache_format_version = 1
cpp_settings: dict[str, Any] = self._custom_settings or {}
cpp_settings = self._custom_settings or {}
return (
cache_format_version,
cpp_settings.get("clangd_version"),
@@ -103,10 +103,32 @@ class ClangdLanguageServer(SolidLanguageServer):
@override
def is_ignored_dirname(self, dirname: str) -> bool:
ignored_dirs = [
".ccls-cache",
]
return super().is_ignored_dirname(dirname) or dirname in ignored_dirs
return (
super().is_ignored_dirname(dirname)
or dirname == ".ccls-cache"
or (is_unreal_engine_project(self.repository_root_path) and dirname in UE_IGNORED_DIRNAMES)
)
def _raise_if_unreal_without_compile_db(self) -> None:
"""Raise if this is an Unreal Engine project without a usable compilation database.
clangd cannot resolve engine headers or macros without one. Non-UE projects return
without raising, since clangd works without a database there.
"""
if not is_unreal_engine_project(self.repository_root_path):
return
raise SolidLSPException(
f"No usable compile_commands.json at {self.repository_root_path}, but this looks like an "
"Unreal Engine project (.uproject present). clangd needs a non-empty compilation database "
"to resolve engine headers and macros.\n\n"
"Generate one from the engine's <Engine>\\Binaries\\DotNET\\UnrealBuildTool\\ directory:\n"
' UnrealBuildTool.exe -mode=GenerateClangDatabase -project="<Proj>.uproject" '
'<Proj>Editor Win64 Development -OutputDir="<projectDir>"\n'
"Without a clang toolchain, append -Compiler=VisualStudio2022 to build an MSVC database instead.\n\n"
"Build the editor target once first so the generated headers exist. After creating the "
"database, reconnect or restart Serena's MCP so the language server re-checks for it.\n"
"See docs/03-special-guides/unreal_engine_setup_guide_for_serena.md for details."
)
def _prepare_compile_commands(self) -> str | None:
"""
@@ -128,7 +150,8 @@ class ClangdLanguageServer(SolidLanguageServer):
compile_db_path = os.path.join(self.repository_root_path, "compile_commands.json")
if not os.path.exists(compile_db_path):
# No compile_commands.json, nothing to do
# clangd can't resolve UE engine headers without the database; fail with guidance.
self._raise_if_unreal_without_compile_db()
return None
try:
@@ -136,6 +159,8 @@ class ClangdLanguageServer(SolidLanguageServer):
compile_commands = json.load(f)
if not compile_commands:
# An empty [] database is treated like a missing one.
self._raise_if_unreal_without_compile_db()
return None
# Check if any entries have relative directory paths
@@ -152,7 +177,7 @@ class ClangdLanguageServer(SolidLanguageServer):
return None
# Get the target directory from ls_specific_settings, default to .serena
cpp_settings: dict[str, Any] = self._custom_settings or {}
cpp_settings = self._custom_settings or {}
compile_commands_rel_dir = cpp_settings.get("compile_commands_dir", ".serena")
compile_commands_dir = os.path.join(self.repository_root_path, compile_commands_rel_dir)
os.makedirs(compile_commands_dir, exist_ok=True)
@@ -288,12 +313,10 @@ class ClangdLanguageServer(SolidLanguageServer):
# which is required for finding cross-file references
return [core_path, "--background-index"]
@staticmethod
def _get_initialize_params(repository_absolute_path: str) -> InitializeParams:
def _create_base_initialize_params(self) -> dict:
"""
Returns the initialize params for the clangd Language Server.
"""
root_uri = pathlib.Path(repository_absolute_path).as_uri()
initialize_params = {
"locale": "en",
"capabilities": {
@@ -309,18 +332,9 @@ class ClangdLanguageServer(SolidLanguageServer):
},
"workspace": {"workspaceFolders": True, "didChangeConfiguration": {"dynamicRegistration": True}},
},
"processId": os.getpid(),
"rootPath": repository_absolute_path,
"rootUri": root_uri,
"workspaceFolders": [
{
"uri": root_uri,
"name": os.path.basename(repository_absolute_path),
}
],
}
return cast(InitializeParams, initialize_params)
return initialize_params
def _start_server(self) -> None:
"""
@@ -376,7 +390,7 @@ class ClangdLanguageServer(SolidLanguageServer):
log.info("Starting Clangd server process")
self.server.start()
initialize_params = self._get_initialize_params(self.repository_root_path)
initialize_params = self._create_initialize_params()
log.info("Sending initialize request from LSP client to LSP server and awaiting response")
init_response = self.server.send.initialize(initialize_params)
@@ -384,9 +398,9 @@ class ClangdLanguageServer(SolidLanguageServer):
text_document_sync = capabilities["textDocumentSync"]
if isinstance(text_document_sync, int):
assert text_document_sync == 2 # type: ignore
assert text_document_sync == 2
else:
assert text_document_sync["change"] == 2 # type: ignore
assert text_document_sync["change"] == 2
assert "completionProvider" in capabilities
completion_provider = capabilities["completionProvider"]
src/solidlsp/language_servers/clojure_lsp.py
+4 -12
View File
@@ -9,13 +9,11 @@ import re
import shutil
import subprocess
import threading
from typing import cast
from overrides import override
from solidlsp.ls import LanguageServerDependencyProvider, LanguageServerDependencyProviderSinglePath, SolidLanguageServer
from solidlsp.ls_config import LanguageServerConfig
from solidlsp.lsp_protocol_handler.lsp_types import InitializeParams
from solidlsp.settings import SolidLSPSettings
from .common import RuntimeDependency, RuntimeDependencyCollection
@@ -308,20 +306,15 @@ class ClojureLSP(SolidLanguageServer):
log.info(f"clojure-lsp source-paths scanned from project descriptors: {scanned}")
return scanned
def _get_initialize_params(self) -> InitializeParams:
def _create_base_initialize_params(self) -> dict:
"""Returns the init params for clojure-lsp."""
repository_absolute_path = self.repository_root_path
root_uri = pathlib.Path(repository_absolute_path).as_uri()
source_paths = self._resolve_source_paths()
initialization_options: dict = {"dependency-scheme": "jar", "text-document-sync-kind": "incremental"}
if source_paths is not None:
initialization_options["source-paths"] = source_paths
result = { # type: ignore
"processId": os.getpid(),
"rootPath": repository_absolute_path,
"rootUri": root_uri,
result = {
"capabilities": {
"workspace": {
"applyEdit": True,
@@ -344,9 +337,8 @@ class ClojureLSP(SolidLanguageServer):
},
"initializationOptions": initialization_options,
"trace": "off",
"workspaceFolders": [{"uri": root_uri, "name": os.path.basename(repository_absolute_path)}],
}
return cast(InitializeParams, result)
return result
def _start_server(self) -> None:
def register_capability_handler(params: dict) -> None:
@@ -389,7 +381,7 @@ class ClojureLSP(SolidLanguageServer):
log.info("Starting clojure-lsp server process")
self.server.start()
initialize_params = self._get_initialize_params()
initialize_params = self._create_initialize_params()
log.info("Sending initialize request from LSP client to LSP server and awaiting response")
init_response = self.server.send.initialize(initialize_params)
src/solidlsp/language_servers/common.py
+13 -39
View File
@@ -2,8 +2,8 @@ from __future__ import annotations
import logging
import os
import pathlib
import platform
import shutil
import subprocess
from collections.abc import Iterable, Mapping, Sequence
from dataclasses import dataclass, replace
@@ -105,7 +105,7 @@ class RuntimeDependencyCollection:
if not PlatformUtils.get_platform_id().is_windows():
import pwd
kwargs["user"] = pwd.getpwuid(os.getuid()).pw_name # type: ignore
kwargs["user"] = pwd.getpwuid(os.getuid()).pw_name
command = subprocess_util.convert_shell_cmd(command)
log.info("Running command %s in '%s'", f"'{command}'" if isinstance(command, str) else command, cwd)
@@ -119,7 +119,7 @@ class RuntimeDependencyCollection:
stdout=subprocess.PIPE,
stderr=subprocess.STDOUT,
**kwargs,
) # type: ignore
)
if completed_process.returncode != 0:
log.warning("Command '%s' failed with return code %d", command, completed_process.returncode)
log.warning("Command output:\n%s", completed_process.stdout)
@@ -150,42 +150,6 @@ class RuntimeDependencyCollection:
)
DEFAULT_UVX_PYTHON_VERSION = "3.13"
def build_uvx_launch_command(
package: str,
version: str,
entrypoint: str,
extra_args: Sequence[str] = (),
python_version: str = DEFAULT_UVX_PYTHON_VERSION,
) -> list[str]:
"""Build a command that runs a pinned PyPI package's console script on demand via ``uvx`` / ``uv x``.
Resolution order:
1. Prefer ``uvx`` (env var ``UVX`` or PATH lookup).
2. Fall back to ``uv x`` if only ``uv`` is on PATH.
3. Raise ``RuntimeError`` if neither is available.
:param package: PyPI package name (e.g. ``"pyright"``).
:param version: Pinned package version.
:param entrypoint: Console script provided by the package (e.g. ``"pyright-langserver"``).
:param extra_args: Arguments appended after the entrypoint (e.g. ``("--stdio",)``).
:param python_version: Python interpreter version passed via ``-p`` (uv will fetch it if missing).
"""
base_args = ["-p", python_version, "--from", f"{package}=={version}", entrypoint, *extra_args]
uvx_path = os.environ.get("UVX") or shutil.which("uvx")
if uvx_path is not None:
return [uvx_path, *base_args]
uv_path = shutil.which("uv")
if uv_path is not None:
return [uv_path, "x", *base_args]
raise RuntimeError("Could not find 'uvx' or 'uv' in PATH. Install uv (https://docs.astral.sh/uv/).")
def build_npm_install_command(package_name: str, version: str, registry: str | None = None) -> list[str]:
"""Build a pinned npm install command for a package in a Serena-managed install directory."""
command = ["npm", "install", "--prefix", "./"]
@@ -216,3 +180,13 @@ def quote_windows_path(path: str) -> str:
return path
return f'"{path}"'
return path
UE_IGNORED_DIRNAMES = frozenset({"Binaries", "DerivedDataCache", "Intermediate", "Saved"})
"""Unreal Engine build and cache directories. Matched per directory name, so per-plugin and
per-module copies (e.g. ``Plugins/Foo/Intermediate``) are covered too."""
def is_unreal_engine_project(repository_root_path: str) -> bool:
""":return: whether the repository root contains an Unreal Engine ``.uproject`` file."""
return any(pathlib.Path(repository_root_path).glob("*.uproject"))
src/solidlsp/language_servers/crystal_language_server.py
+3 -17
View File
@@ -3,14 +3,11 @@ Provides Crystal specific instantiation of the LanguageServer class using Crysta
"""
import logging
import os
import pathlib
import shutil
import time
from solidlsp.ls import SolidLanguageServer
from solidlsp.ls_config import LanguageServerConfig
from solidlsp.lsp_protocol_handler.lsp_types import InitializeParams
from solidlsp.lsp_protocol_handler.server import ProcessLaunchInfo
from solidlsp.settings import SolidLSPSettings
@@ -83,12 +80,10 @@ class CrystalLanguageServer(SolidLanguageServer):
log.info(f"Waiting {remaining_delay:.1f}s for Crystalline to compile the project")
time.sleep(remaining_delay)
@staticmethod
def _get_initialize_params(repository_absolute_path: str) -> InitializeParams:
def _create_base_initialize_params(self) -> dict:
"""
Return the initialize params for the Crystal language server.
"""
root_uri = pathlib.Path(repository_absolute_path).as_uri()
initialize_params = {
"locale": "en",
"capabilities": {
@@ -119,17 +114,8 @@ class CrystalLanguageServer(SolidLanguageServer):
"configuration": True,
},
},
"processId": os.getpid(),
"rootPath": repository_absolute_path,
"rootUri": root_uri,
"workspaceFolders": [
{
"uri": root_uri,
"name": os.path.basename(repository_absolute_path),
}
],
}
return initialize_params # type: ignore[return-value]
return initialize_params
def _start_server(self) -> None:
"""Start the Crystal language server process."""
@@ -150,7 +136,7 @@ class CrystalLanguageServer(SolidLanguageServer):
log.info("Starting Crystal language server (Crystalline) process")
self.server.start()
initialize_params = self._get_initialize_params(self.repository_root_path)
initialize_params = self._create_initialize_params()
log.info("Sending initialize request from LSP client to LSP server and awaiting response")
init_response = self.server.send.initialize(initialize_params)
src/solidlsp/language_servers/csharp_language_server.py
+48 -51
View File
@@ -6,6 +6,7 @@ import logging
import os
import platform
import shutil
import tempfile
import threading
from collections.abc import Hashable, Iterable
from dataclasses import replace
@@ -25,7 +26,7 @@ from solidlsp.ls_config import LanguageServerConfig
from solidlsp.ls_exceptions import SolidLSPException
from solidlsp.ls_types import Hover
from solidlsp.ls_utils import FileUtils, PathUtils
from solidlsp.lsp_protocol_handler.lsp_types import InitializeParams, InitializeResult
from solidlsp.lsp_protocol_handler.lsp_types import InitializeResult
from solidlsp.settings import SolidLSPSettings
from .common import RuntimeDependency, RuntimeDependencyCollection
@@ -34,6 +35,8 @@ log = logging.getLogger(__name__)
NUGET_ALLOWED_HOSTS = ("www.nuget.org", "nuget.org", "globalcdn.nuget.org")
DEFAULT_CSHARP_LANGUAGE_SERVER_VERSION = "5.5.0-2.26078.4"
WINDOWS_MAX_PATH = 260
ROSLYN_LONGEST_KNOWN_PACKAGE_MEMBER = "Microsoft.CodeAnalysis.ExternalAccess.CompilerDeveloperSDK.dll"
_RUNTIME_DEPENDENCIES = [
RuntimeDependency(
@@ -274,7 +277,7 @@ class CSharpLanguageServer(SolidLanguageServer):
if original_name != normalized_name:
sel_range = symbol.get("selectionRange")
if sel_range:
start = sel_range.get("start") # type: ignore
start = sel_range.get("start")
if start and "line" in start and "character" in start:
line = start["line"]
char = start["character"]
@@ -460,12 +463,15 @@ class CSharpLanguageServer(SolidLanguageServer):
if url is None:
raise SolidLSPException(f"No URL specified for package {package_name} version {package_version}")
temp_dir = Path(self._ls_resources_dir) / "temp_downloads"
temp_dir.mkdir(parents=True, exist_ok=True)
package_extract_dir = Path(self._ls_resources_dir) / "temp_downloads" / f"{package_name}.{package_version}"
projected_path = package_extract_dir / (dependency.extract_path or "") / ROSLYN_LONGEST_KNOWN_PACKAGE_MEMBER
if platform.system() == "Windows" and len(str(projected_path)) >= WINDOWS_MAX_PATH:
log.warning("Using a short temporary directory for Roslyn package extraction because the configured cache path is too deep")
package_extract_dir = Path(tempfile.mkdtemp(prefix="serena-roslyn-")) / package_extract_dir.name
package_extract_dir.parent.mkdir(parents=True, exist_ok=True)
try:
log.debug(f"Downloading package from: {url}")
package_extract_dir = temp_dir / f"{package_name}.{package_version}"
FileUtils.download_and_extract_archive_verified(
url,
str(package_extract_dir),
@@ -480,60 +486,51 @@ class CSharpLanguageServer(SolidLanguageServer):
except Exception as e:
raise SolidLSPException(f"Failed to download package {package_name} version {package_version} from NuGet.org: {e}") from e
def _get_initialize_params(self) -> InitializeParams:
def _create_base_initialize_params(self) -> dict:
"""
Returns the initialize params for the Microsoft.CodeAnalysis.LanguageServer.
"""
root_uri = PathUtils.path_to_uri(self.repository_root_path)
root_name = os.path.basename(self.repository_root_path)
return cast(
InitializeParams,
{
"workspaceFolders": [{"uri": root_uri, "name": root_name}],
"processId": os.getpid(),
"rootPath": self.repository_root_path,
"rootUri": root_uri,
"capabilities": {
"window": {
"workDoneProgress": True,
"showMessage": {"messageActionItem": {"additionalPropertiesSupport": True}},
"showDocument": {"support": True},
return {
"capabilities": {
"window": {
"workDoneProgress": True,
"showMessage": {"messageActionItem": {"additionalPropertiesSupport": True}},
"showDocument": {"support": True},
},
"workspace": {
"applyEdit": True,
"workspaceEdit": {"documentChanges": True},
"didChangeConfiguration": {"dynamicRegistration": True},
"didChangeWatchedFiles": {"dynamicRegistration": True},
"symbol": {
"dynamicRegistration": True,
"symbolKind": {"valueSet": list(range(1, 27))},
},
"workspace": {
"applyEdit": True,
"workspaceEdit": {"documentChanges": True},
"didChangeConfiguration": {"dynamicRegistration": True},
"didChangeWatchedFiles": {"dynamicRegistration": True},
"symbol": {
"dynamicRegistration": True,
"symbolKind": {"valueSet": list(range(1, 27))},
"executeCommand": {"dynamicRegistration": True},
"configuration": True,
"workspaceFolders": True,
"workDoneProgress": True,
},
"textDocument": {
"synchronization": {"dynamicRegistration": True, "willSave": True, "willSaveWaitUntil": True, "didSave": True},
"hover": {"dynamicRegistration": True, "contentFormat": ["markdown", "plaintext"]},
"signatureHelp": {
"dynamicRegistration": True,
"signatureInformation": {
"documentationFormat": ["markdown", "plaintext"],
"parameterInformation": {"labelOffsetSupport": True},
},
"executeCommand": {"dynamicRegistration": True},
"configuration": True,
"workspaceFolders": True,
"workDoneProgress": True,
},
"textDocument": {
"synchronization": {"dynamicRegistration": True, "willSave": True, "willSaveWaitUntil": True, "didSave": True},
"hover": {"dynamicRegistration": True, "contentFormat": ["markdown", "plaintext"]},
"signatureHelp": {
"dynamicRegistration": True,
"signatureInformation": {
"documentationFormat": ["markdown", "plaintext"],
"parameterInformation": {"labelOffsetSupport": True},
},
},
"definition": {"dynamicRegistration": True},
"references": {"dynamicRegistration": True},
"documentSymbol": {
"dynamicRegistration": True,
"symbolKind": {"valueSet": list(range(1, 27))},
"hierarchicalDocumentSymbolSupport": True,
},
"definition": {"dynamicRegistration": True},
"references": {"dynamicRegistration": True},
"documentSymbol": {
"dynamicRegistration": True,
"symbolKind": {"valueSet": list(range(1, 27))},
"hierarchicalDocumentSymbolSupport": True,
},
},
},
)
}
def _start_server(self) -> None:
indexing_complete = threading.Event()
@@ -666,7 +663,7 @@ class CSharpLanguageServer(SolidLanguageServer):
raise SolidLSPException(f"Failed to start C# language server: {e}")
# Send initialization
initialize_params = self._get_initialize_params()
initialize_params = self._create_initialize_params()
log.info("Sending initialize request to language server")
try:
src/solidlsp/language_servers/cue_language_server.py
+3 -13
View File
@@ -7,15 +7,12 @@ from __future__ import annotations
import logging
import os
import pathlib
import threading
from typing import cast
from overrides import override
from solidlsp.ls import LanguageServerDependencyProvider, LanguageServerDependencyProviderSinglePath, SolidLanguageServer
from solidlsp.ls_config import LanguageServerConfig
from solidlsp.lsp_protocol_handler.lsp_types import InitializeParams
from solidlsp.settings import SolidLSPSettings
from .common import RuntimeDependency, RuntimeDependencyCollection
@@ -184,15 +181,9 @@ class CueLanguageServer(SolidLanguageServer):
# cue's LSP mode is activated via the `lsp` subcommand; it speaks LSP over stdio.
return [core_path, "lsp"]
def _get_initialize_params(self) -> InitializeParams:
def _create_base_initialize_params(self) -> dict:
"""Returns the init params for ``cue lsp``."""
repository_absolute_path = self.repository_root_path
root_uri = pathlib.Path(repository_absolute_path).as_uri()
result = {
"processId": os.getpid(),
"rootPath": repository_absolute_path,
"rootUri": root_uri,
"capabilities": {
"workspace": {
"applyEdit": True,
@@ -221,9 +212,8 @@ class CueLanguageServer(SolidLanguageServer):
},
"initializationOptions": {},
"trace": "off",
"workspaceFolders": [{"uri": root_uri, "name": os.path.basename(repository_absolute_path)}],
}
return cast(InitializeParams, result)
return result
def _start_server(self) -> None:
def register_capability_handler(params: dict) -> None:
@@ -252,7 +242,7 @@ class CueLanguageServer(SolidLanguageServer):
log.info("Starting cue lsp server process")
self.server.start()
initialize_params = self._get_initialize_params()
initialize_params = self._create_initialize_params()
log.info("Sending initialize request from LSP client to LSP server and awaiting response")
init_response = self.server.send.initialize(initialize_params)
src/solidlsp/language_servers/dart_language_server.py
+3 -17
View File
@@ -1,8 +1,6 @@
import logging
import os
import pathlib
from collections.abc import Hashable
from typing import cast
from overrides import override
@@ -11,7 +9,6 @@ from solidlsp.lsp_protocol_handler.server import ProcessLaunchInfo
from solidlsp.settings import SolidLSPSettings
from ..ls_config import Language, LanguageServerConfig
from ..lsp_protocol_handler.lsp_types import InitializeParams
from .common import RuntimeDependency, RuntimeDependencyCollection
log = logging.getLogger(__name__)
@@ -149,12 +146,10 @@ class DartLanguageServer(SolidLanguageServer):
return f"{dart_executable_path} language-server --client-id multilspy.dart --client-version 1.2"
@staticmethod
def _get_initialize_params(repository_absolute_path: str) -> InitializeParams:
def _create_base_initialize_params(self) -> dict:
"""
Returns the initialize params for the Dart Language Server.
"""
root_uri = pathlib.Path(repository_absolute_path).as_uri()
initialize_params = {
"capabilities": {
"textDocument": {
@@ -171,18 +166,9 @@ class DartLanguageServer(SolidLanguageServer):
"allowOpenUri": False,
},
"trace": "verbose",
"processId": os.getpid(),
"rootPath": repository_absolute_path,
"rootUri": pathlib.Path(repository_absolute_path).as_uri(),
"workspaceFolders": [
{
"uri": root_uri,
"name": os.path.basename(repository_absolute_path),
}
],
}
return cast(InitializeParams, initialize_params)
return initialize_params
def _start_server(self) -> None:
"""
@@ -212,7 +198,7 @@ class DartLanguageServer(SolidLanguageServer):
log.info("Starting dart-language-server server process")
self.server.start()
initialize_params = self._get_initialize_params(self.repository_root_path)
initialize_params = self._create_initialize_params()
log.debug("Sending initialize request to dart-language-server")
init_response = self.server.send_request("initialize", initialize_params) # type: ignore
log.info(f"Received initialize response from dart-language-server: {init_response}")
src/solidlsp/language_servers/eclipse_jdtls.py
+152 -67
View File
@@ -4,6 +4,7 @@ Provides Java specific instantiation of the LanguageServer class. Contains vario
import dataclasses
import hashlib
import json
import logging
import os
import pathlib
@@ -14,7 +15,7 @@ import subprocess
import threading
from pathlib import Path, PurePath
from time import sleep
from typing import cast
from typing import Any, cast
from overrides import override
@@ -24,7 +25,7 @@ from solidlsp.ls_config import LanguageServerConfig
from solidlsp.ls_exceptions import SolidLSPException
from solidlsp.ls_types import UnifiedSymbolInformation
from solidlsp.ls_utils import FileUtils, PlatformUtils
from solidlsp.lsp_protocol_handler.lsp_types import DocumentSymbol, InitializeParams, SymbolInformation
from solidlsp.lsp_protocol_handler.lsp_types import DocumentSymbol, SymbolInformation
from solidlsp.settings import SolidLSPSettings
log = logging.getLogger(__name__)
@@ -179,8 +180,10 @@ class EclipseJDTLS(SolidLanguageServer):
- maven_user_settings: Path to Maven settings.xml file (default: ~/.m2/settings.xml)
- gradle_user_home: Path to Gradle user home directory (default: ~/.gradle)
- gradle_wrapper_enabled: Whether to use the project's Gradle wrapper (default: false)
- gradle_java_home: Path to JDK for Gradle (default: null, uses bundled JRE)
- use_system_java_home: Whether to use the system's JAVA_HOME for JDTLS itself (default: false)
- gradle_java_home: Path to JDK for Gradle (default: null, falls back to JAVA_HOME when
use_system_java_home is true, then the bundled JRE)
- use_system_java_home: Whether to use the system's JAVA_HOME for JDTLS itself and, when
gradle_java_home is unset, Gradle import (default: false)
- jdtls_xmx: Maximum heap size for the JDTLS server JVM (default: "3G")
- jdtls_xms: Initial heap size for the JDTLS server JVM (default: "100m")
- intellicode_xmx: Maximum heap size for the IntelliCode embedded JVM (default: "1G")
@@ -196,6 +199,17 @@ class EclipseJDTLS(SolidLanguageServer):
are not supported in default VSIX mode (the resource paths inside the archive change between
releases); use upstream-jdtls mode for arbitrary versions.
- intellicode_version: Override the pinned IntelliCode VSIX version downloaded by Serena
- runtimes: Additional JRE/JDK entries to register with JDT-LS's ``java.configuration.runtimes``,
for projects whose source/target level exceeds the JDK JDT-LS itself runs on (currently
JDK 21 in default vscode-java VSIX mode). Each entry is a mapping with:
- name (required): the JRE container name JDT-LS should register the entry under,
e.g. "JavaSE-25" (must match the ``JavaSE-NN`` the build tool requests).
- path (required): filesystem path to the JDK/JRE home directory; must exist.
- default (optional): whether this runtime is JDT-LS's default when no container name matches.
- sources / javadoc (optional): passed through unchanged to JDT-LS.
These entries extend rather than replace the bundled JRE, which is still registered as
"JavaSE-21" unless a configured entry reuses that same name (in which case it is overridden).
See serena #1478.
Example configuration for upstream JDTLS mode (no downloads, suitable for offline/corporate):
```yaml
@@ -216,7 +230,7 @@ class EclipseJDTLS(SolidLanguageServer):
# gradle_user_home: 'C:\\Users\\YourName\\.gradle' # Windows (use single quotes!)
gradle_wrapper_enabled: true # set to true for projects with custom plugins/repositories
gradle_java_home: "/path/to/jdk" # set to override Gradle's JDK
use_system_java_home: true # set to true to use system JAVA_HOME for JDTLS
use_system_java_home: true # set to true to use system JAVA_HOME for JDTLS and Gradle fallback
jdtls_xmx: "3G" # maximum heap size for the JDTLS server JVM
jdtls_xms: "100m" # initial heap size for the JDTLS server JVM
intellicode_xmx: "1G" # maximum heap size for the IntelliCode embedded JVM
@@ -225,6 +239,10 @@ class EclipseJDTLS(SolidLanguageServer):
gradle_version: "8.14.2"
vscode_java_version: "1.54.0-923" # also accepts pinned legacy "1.42.0-561"
intellicode_version: "1.2.30"
runtimes: # register additional JDKs for projects targeting a newer Java version
- name: "JavaSE-25"
path: "/home/user/Java/jdk25"
default: true
```
"""
@@ -251,24 +269,6 @@ class EclipseJDTLS(SolidLanguageServer):
ls_resources_dir = self.ls_resources_dir(self._solidlsp_settings)
return self.DependencyProvider(self._custom_settings, ls_resources_dir, self._solidlsp_settings, self.repository_root_path)
@override
def is_ignored_dirname(self, dirname: str) -> bool:
# Ignore common Java build directories from different build tools:
# - Maven: target
# - Gradle: build, .gradle
# - Eclipse: bin, .settings
# - IntelliJ IDEA: out, .idea
# - General: classes, dist, lib
return super().is_ignored_dirname(dirname) or dirname in [
"target", # Maven
"build", # Gradle
"bin", # Eclipse
"out", # IntelliJ IDEA
"classes", # General
"dist", # General
"lib", # General
]
class DependencyProvider(LanguageServerDependencyProvider):
def __init__(
self,
@@ -776,20 +776,38 @@ class EclipseJDTLS(SolidLanguageServer):
vscode-java VSIX bump or upstream install change) lands in a separate ws_dir
and avoids stale OSGi configs from the previous version blocking startup.
Workspace-affecting settings are also mixed in so changing Maven/Gradle/JDK
inputs lands in a fresh workspace instead of silently reusing stale import
state from a previous configuration.
Exception: legacy default-mode users on INITIAL_VSCODE_JAVA_VERSION keep the
original ``md5(repository_root_path)`` format. These are users who installed
under the legacy unversioned ``vscode-java/`` directory (see the version-pinning
convention at the top of this module) — preserving their hash means existing
JDTLS workspaces and project caches are reused without a one-time reindex.
original ``md5(repository_root_path)`` format *until* one of the tracked
workspace-affecting settings is set. This preserves existing caches for the
old default setup while still fixing stale-workspace bugs once users opt into
custom Java import settings.
"""
workspace_setting_keys = (
"maven_user_settings",
"gradle_user_home",
"gradle_wrapper_enabled",
"gradle_java_home",
"java_home",
"use_system_java_home",
"maven_offline",
"runtimes",
)
workspace_settings = {key: custom_settings.settings[key] for key in workspace_setting_keys if key in custom_settings.settings}
workspace_settings_json = json.dumps(workspace_settings, sort_keys=True, separators=(",", ":"))
is_legacy_initial = (
not custom_settings.get("jdtls_path")
and custom_settings.get("vscode_java_version", DEFAULT_VSCODE_JAVA_VERSION) == INITIAL_VSCODE_JAVA_VERSION
and not workspace_settings
)
if is_legacy_initial:
ws_hash_input = repository_root_path.encode()
else:
ws_hash_input = (repository_root_path + "|" + jdtls_launcher_jar_path).encode()
ws_hash_input = (repository_root_path + "|" + jdtls_launcher_jar_path + "|" + workspace_settings_json).encode()
return hashlib.md5(ws_hash_input).hexdigest()
def create_launch_command(self) -> list[str]:
@@ -886,15 +904,93 @@ class EclipseJDTLS(SolidLanguageServer):
log.info(f"Using bundled JRE for JDTLS: {java_home}")
return {"syntaxserver": "false", "JAVA_HOME": java_home}
def _get_initialize_params(self, repository_absolute_path: str) -> InitializeParams:
def _resolve_gradle_java_home(self) -> str:
"""
Resolve the Java home configured for JDTLS Gradle import settings.
:return: Java home selected from explicit Gradle override, system ``JAVA_HOME`` when enabled, or bundled runtime.
:raises FileNotFoundError: If an explicitly configured ``gradle_java_home`` path does not exist.
"""
# explicit Gradle override...
gradle_java_home = self._custom_settings.get("gradle_java_home")
if gradle_java_home is not None:
if not os.path.exists(gradle_java_home):
error_msg = (
f"Gradle Java home not found: {gradle_java_home}. "
f"Fix: update path in ~/.serena/serena_config.yml (ls_specific_settings -> java -> gradle_java_home), "
f"or remove the setting to use JAVA_HOME or the bundled JRE fallback"
)
log.error(error_msg)
raise FileNotFoundError(error_msg)
log.info(f"Using Gradle Java home from custom location: {gradle_java_home}")
return gradle_java_home
# system JAVA_HOME fallback...
use_system_java_home = self._custom_settings.get("use_system_java_home", False)
if use_system_java_home:
system_java_home = os.environ.get("JAVA_HOME")
if system_java_home:
log.info(f"Using system JAVA_HOME for Gradle: {system_java_home}")
return system_java_home
log.warning("use_system_java_home is set but JAVA_HOME is not set in environment, falling back to bundled JRE for Gradle")
# bundled runtime fallback...
log.info(f"Using bundled JRE for Gradle: {self.runtime_dependency_paths.jre_path}")
return self.runtime_dependency_paths.jre_path
def _resolve_configured_runtimes(self) -> list[dict[str, Any]]:
"""
Validate and normalize the optional extra JRE/JDK runtimes to register with JDT-LS, as configured
via ``ls_specific_settings.java.runtimes``. Each entry mirrors the shape VS Code's Java extension
sends via ``java.configuration.runtimes`` (``name``, ``path``, optional ``default``/``sources``/
``javadoc``); see serena #1478.
:return: the validated list of runtime dicts (empty if the setting is unset)
:raises ValueError: if the setting or one of its entries is malformed
:raises FileNotFoundError: if an entry's ``path`` does not exist
"""
configured_runtimes = self._custom_settings.get("runtimes", [])
if not isinstance(configured_runtimes, list):
raise ValueError(
f"ls_specific_settings.java.runtimes must be a list of {{name, path}} entries, "
f"got {type(configured_runtimes).__name__}: {configured_runtimes!r}"
)
# validate each entry and normalize it to the shape JDT-LS expects...
validated_runtimes: list[dict[str, Any]] = []
for entry in configured_runtimes:
if not isinstance(entry, dict) or "name" not in entry or "path" not in entry:
raise ValueError(
f"Invalid ls_specific_settings.java.runtimes entry {entry!r}: each entry requires at least a 'name' and a 'path' key."
)
path = entry["path"]
if not os.path.exists(path):
error_msg = (
f"ls_specific_settings.java.runtimes entry '{entry['name']}' points to a path that "
f"does not exist: {path}. Fix: update the path in ~/.serena/serena_config.yml "
f"(ls_specific_settings -> java -> runtimes), or remove the entry."
)
log.error(error_msg)
raise FileNotFoundError(error_msg)
runtime: dict[str, Any] = {"name": entry["name"], "path": path}
if "default" in entry:
runtime["default"] = bool(entry["default"])
for optional_key in ("sources", "javadoc"):
if optional_key in entry:
runtime[optional_key] = entry[optional_key]
validated_runtimes.append(runtime)
log.info(f"Registering additional JDT-LS runtime '{runtime['name']}' from custom settings: {path}")
return validated_runtimes
def _create_base_initialize_params(self) -> dict:
"""
Returns the initialize parameters for the EclipseJDTLS server.
"""
# Look into https://github.com/eclipse/eclipse.jdt.ls/blob/master/org.eclipse.jdt.ls.core/src/org/eclipse/jdt/ls/core/internal/preferences/Preferences.java to understand all the options available
if not os.path.isabs(repository_absolute_path):
repository_absolute_path = os.path.abspath(repository_absolute_path)
repo_uri = pathlib.Path(repository_absolute_path).as_uri()
repo_uri = pathlib.Path(self.repository_root_path).as_uri()
# Load user's Maven and Gradle configuration paths from ls_specific_settings["java"]
@@ -961,25 +1057,11 @@ class EclipseJDTLS(SolidLanguageServer):
f"Gradle wrapper {'enabled' if gradle_wrapper_enabled else 'disabled'} (configurable via ls_specific_settings -> java -> gradle_wrapper_enabled)"
)
# Gradle Java home: default to None, which means the bundled JRE is used
gradle_java_home = self._custom_settings.get("gradle_java_home")
if gradle_java_home is not None:
if not os.path.exists(gradle_java_home):
error_msg = (
f"Gradle Java home not found: {gradle_java_home}. "
f"Fix: update path in ~/.serena/serena_config.yml (ls_specific_settings -> java -> gradle_java_home), "
f"or remove the setting to use the bundled JRE"
)
log.error(error_msg)
raise FileNotFoundError(error_msg)
log.info(f"Using Gradle Java home from custom location: {gradle_java_home}")
else:
log.info(f"Using bundled JRE for Gradle: {self.runtime_dependency_paths.jre_path}")
# Gradle Java home: explicit setting, system JAVA_HOME when requested, then bundled runtime.
gradle_java_home = self._resolve_gradle_java_home()
initialize_params = {
"locale": "en",
"rootPath": repository_absolute_path,
"rootUri": pathlib.Path(repository_absolute_path).as_uri(),
"capabilities": {
"workspace": {
"applyEdit": True,
@@ -1272,41 +1354,44 @@ class EclipseJDTLS(SolidLanguageServer):
},
},
"trace": "verbose",
"processId": os.getpid(),
"workspaceFolders": [
{
"uri": repo_uri,
"name": os.path.basename(repository_absolute_path),
}
],
}
initialize_params["initializationOptions"]["workspaceFolders"] = [repo_uri] # type: ignore
initialize_params["initializationOptions"]["workspaceFolders"] = [repo_uri]
# IntelliCode bundle: only attached in default vscode-java VSIX mode.
# In upstream-jdtls mode (jdtls_path set) we don't ship IntelliCode — agentic Serena workflows
# don't use completion ranking, so the bundle would be inert dead weight.
if self.runtime_dependency_paths.intellicode_jar_path is not None:
initialize_params["initializationOptions"]["bundles"] = [self.runtime_dependency_paths.intellicode_jar_path] # type: ignore
initialize_params["initializationOptions"]["bundles"] = [self.runtime_dependency_paths.intellicode_jar_path]
else:
initialize_params["initializationOptions"]["bundles"] = [] # type: ignore
initialize_params["initializationOptions"]["bundles"] = []
initialize_params["initializationOptions"]["settings"]["java"]["configuration"]["runtimes"] = [ # type: ignore
{"name": "JavaSE-21", "path": self.runtime_dependency_paths.jre_home_path, "default": True}
]
# merge the bundled JRE with any additional runtimes configured via ls_specific_settings.java.runtimes
# (e.g. so projects targeting a newer Java version than the bundled JRE resolve their JRE container)...
default_runtime = {"name": "JavaSE-21", "path": self.runtime_dependency_paths.jre_home_path, "default": True}
configured_runtimes = self._resolve_configured_runtimes()
runtimes_by_name = {default_runtime["name"]: default_runtime}
for runtime in configured_runtimes:
runtimes_by_name[runtime["name"]] = runtime
if runtimes_by_name["JavaSE-21"] is default_runtime and any(runtime.get("default") for runtime in configured_runtimes):
# a configured runtime claims the JDT-LS default; the bundled runtime must not also claim it,
# since JDT-LS expects at most one default runtime
default_runtime["default"] = False
for runtime in initialize_params["initializationOptions"]["settings"]["java"]["configuration"]["runtimes"]: # type: ignore
initialize_params["initializationOptions"]["settings"]["java"]["configuration"]["runtimes"] = list(runtimes_by_name.values())
for runtime in initialize_params["initializationOptions"]["settings"]["java"]["configuration"]["runtimes"]:
assert "name" in runtime
assert "path" in runtime
assert os.path.exists(runtime["path"]), f"Runtime required for eclipse_jdtls at path {runtime['path']} does not exist"
gradle_settings = initialize_params["initializationOptions"]["settings"]["java"]["import"]["gradle"] # type: ignore
gradle_settings = initialize_params["initializationOptions"]["settings"]["java"]["import"]["gradle"]
# In upstream-jdtls mode we don't ship a Gradle distribution — Buildship will use the project's
# ./gradlew wrapper or a system-installed Gradle via its standard discovery rules.
if self.runtime_dependency_paths.gradle_path is not None:
gradle_settings["home"] = self.runtime_dependency_paths.gradle_path
gradle_settings["java"] = {"home": gradle_java_home if gradle_java_home is not None else self.runtime_dependency_paths.jre_path}
return cast(InitializeParams, initialize_params)
gradle_settings["java"] = {"home": gradle_java_home}
return initialize_params
def _start_server(self) -> None:
"""
@@ -1359,7 +1444,7 @@ class EclipseJDTLS(SolidLanguageServer):
log.info("Starting EclipseJDTLS server process")
self.server.start()
initialize_params = self._get_initialize_params(self.repository_root_path)
initialize_params = self._create_initialize_params()
log.info("Sending initialize request from LSP client to LSP server and awaiting response")
init_response = self.server.send.initialize(initialize_params)
@@ -1465,7 +1550,7 @@ class EclipseJDTLS(SolidLanguageServer):
symbol["name"] = symbol["name"][: symbol["name"].index("(")]
children = symbol.get("children")
if children:
for child in children: # type: ignore
for child in children:
fix_name(child)
for root_symbol in result:
src/solidlsp/language_servers/elixir_tools/elixir_tools.py
+5 -13
View File
@@ -5,14 +5,13 @@ import stat
import subprocess
import threading
from collections.abc import Hashable
from typing import Any, cast
from typing import Any
from overrides import override
from solidlsp.ls import RawDocumentSymbol, SolidLanguageServer
from solidlsp.ls_config import Language, LanguageServerConfig
from solidlsp.ls_utils import FileUtils, PlatformId, PlatformUtils
from solidlsp.lsp_protocol_handler.lsp_types import InitializeParams
from solidlsp.lsp_protocol_handler.server import ProcessLaunchInfo
from solidlsp.settings import SolidLSPSettings
@@ -212,6 +211,7 @@ class ElixirTools(SolidLanguageServer):
)
self.server_ready = threading.Event()
self.request_id = 0
self._building_project = False
# Set generous timeout for Expert which can be slow to initialize and respond
self.set_request_timeout(180.0)
@@ -255,19 +255,12 @@ class ElixirTools(SolidLanguageServer):
return None
@staticmethod
def _get_initialize_params(repository_absolute_path: str) -> InitializeParams:
def _create_base_initialize_params(self) -> dict:
"""
Returns the initialize params for the Expert Language Server.
"""
# Ensure the path is absolute
abs_path = os.path.abspath(repository_absolute_path)
root_uri = pathlib.Path(abs_path).as_uri()
initialize_params = {
"processId": os.getpid(),
"locale": "en",
"rootPath": abs_path,
"rootUri": root_uri,
"initializationOptions": {
"mix_env": "dev",
"mix_target": "host",
@@ -318,10 +311,9 @@ class ElixirTools(SolidLanguageServer):
"workDoneProgress": True,
},
},
"workspaceFolders": [{"uri": root_uri, "name": os.path.basename(repository_absolute_path)}],
}
return cast(InitializeParams, initialize_params)
return initialize_params
def _start_server(self) -> None:
"""Start Expert server process"""
@@ -380,7 +372,7 @@ class ElixirTools(SolidLanguageServer):
log.debug("Starting Expert server process")
self.server.start()
initialize_params = self._get_initialize_params(self.repository_root_path)
initialize_params = self._create_initialize_params()
log.debug("Sending initialize request to Expert")
init_response = self.server.send.initialize(initialize_params)
src/solidlsp/language_servers/elm_language_server.py
+3 -17
View File
@@ -4,7 +4,6 @@ Provides Elm specific instantiation of the LanguageServer class. Contains variou
import logging
import os
import pathlib
import shutil
import threading
@@ -14,7 +13,6 @@ from sensai.util.logging import LogTime
from solidlsp import ls_types
from solidlsp.ls import SolidLanguageServer
from solidlsp.ls_config import Language, LanguageServerConfig
from solidlsp.lsp_protocol_handler.lsp_types import InitializeParams
from solidlsp.lsp_protocol_handler.server import ProcessLaunchInfo
from solidlsp.settings import SolidLSPSettings
@@ -131,13 +129,10 @@ class ElmLanguageServer(SolidLanguageServer):
)
return [elm_ls_executable_path, "--stdio"]
@staticmethod
def _get_initialize_params(repository_absolute_path: str) -> InitializeParams:
def _create_base_initialize_params(self) -> dict:
"""
Returns the initialize params for the Elm Language Server.
"""
root_uri = pathlib.Path(repository_absolute_path).as_uri()
initialize_params = {
"locale": "en",
"capabilities": {
@@ -169,17 +164,8 @@ class ElmLanguageServer(SolidLanguageServer):
"skipInstallPackageConfirmation": True,
"onlyUpdateDiagnosticsOnSave": False,
},
"processId": os.getpid(),
"rootPath": repository_absolute_path,
"rootUri": root_uri,
"workspaceFolders": [
{
"uri": root_uri,
"name": os.path.basename(repository_absolute_path),
}
],
}
return initialize_params # type: ignore[return-value]
return initialize_params
def _start_server(self) -> None:
"""
@@ -204,7 +190,7 @@ class ElmLanguageServer(SolidLanguageServer):
log.info("Starting Elm server process")
self.server.start()
initialize_params = self._get_initialize_params(self.repository_root_path)
initialize_params = self._create_initialize_params()
log.info("Sending initialize request from LSP client to LSP server and awaiting response")
init_response = self.server.send.initialize(initialize_params)
src/solidlsp/language_servers/fortran_language_server.py
+5 -19
View File
@@ -3,8 +3,6 @@ Fortran Language Server implementation using fortls.
"""
import logging
import os
import pathlib
import re
from overrides import override
@@ -19,7 +17,6 @@ from solidlsp.ls import (
SolidLanguageServer,
)
from solidlsp.ls_config import LanguageServerConfig
from solidlsp.lsp_protocol_handler.lsp_types import InitializeParams
from solidlsp.settings import SolidLSPSettings
log = logging.getLogger(__name__)
@@ -190,10 +187,8 @@ class FortranLanguageServer(SolidLanguageServer):
version_setting_key="fortls_version",
)
@staticmethod
def _get_initialize_params(repository_absolute_path: str) -> InitializeParams:
def _create_base_initialize_params(self) -> dict:
"""Initialize params for Fortran Language Server."""
root_uri = pathlib.Path(repository_absolute_path).as_uri()
initialize_params = {
"locale": "en",
"capabilities": {
@@ -231,17 +226,8 @@ class FortranLanguageServer(SolidLanguageServer):
},
},
},
"processId": os.getpid(),
"rootPath": repository_absolute_path,
"rootUri": root_uri,
"workspaceFolders": [
{
"uri": root_uri,
"name": os.path.basename(repository_absolute_path),
}
],
}
return initialize_params # type: ignore[return-value]
return initialize_params
def _start_server(self) -> None:
"""Start Fortran Language Server process."""
@@ -264,7 +250,7 @@ class FortranLanguageServer(SolidLanguageServer):
log.info("Starting Fortran Language Server (fortls) process")
self.server.start()
initialize_params = self._get_initialize_params(self.repository_root_path)
initialize_params = self._create_initialize_params()
log.info("Sending initialize request to Fortran Language Server")
init_response = self.server.send.initialize(initialize_params)
@@ -299,8 +285,8 @@ class FortranLanguageServer(SolidLanguageServer):
with self.open_file(relative_file_path):
self.server.notify.did_save_text_document(
{
LSPConstants.TEXT_DOCUMENT: { # type: ignore
{ # ty: ignore[invalid-argument-type] # dict built from LSPConstants keys; shape matches the TypedDict
LSPConstants.TEXT_DOCUMENT: {
LSPConstants.URI: uri,
}
}
src/solidlsp/language_servers/fsharp_language_server.py
+3 -11
View File
@@ -4,7 +4,6 @@ Provides F# specific instantiation of the LanguageServer class.
import logging
import os
import pathlib
import shutil
import threading
from pathlib import Path
@@ -16,7 +15,6 @@ from solidlsp.language_servers.common import RuntimeDependency, RuntimeDependenc
from solidlsp.ls import SolidLanguageServer
from solidlsp.ls_config import Language, LanguageServerConfig
from solidlsp.ls_exceptions import SolidLSPException
from solidlsp.lsp_protocol_handler.lsp_types import InitializeParams
from solidlsp.lsp_protocol_handler.server import ProcessLaunchInfo
from solidlsp.settings import SolidLSPSettings
@@ -129,17 +127,11 @@ class FSharpLanguageServer(SolidLanguageServer):
# FsAutoComplete uses --lsp flag for LSP mode
return f"{fsautocomplete_path} --adaptive-lsp-server-enabled --project-graph-enabled --use-fcs-transparent-compiler"
def _get_initialize_params(self) -> InitializeParams:
def _create_base_initialize_params(self) -> dict:
"""
Returns the initialize params for the F# Language Server.
"""
root_uri = pathlib.Path(self.repository_root_path).as_uri()
initialize_params = {
"processId": os.getpid(),
"rootPath": self.repository_root_path,
"rootUri": root_uri,
"workspaceFolders": [{"name": "workspace", "uri": root_uri}],
"capabilities": {
"workspace": {
"applyEdit": True,
@@ -269,7 +261,7 @@ class FSharpLanguageServer(SolidLanguageServer):
"trace": "off",
}
return initialize_params # type: ignore
return initialize_params
def _get_dotnet_root(self) -> str:
"""
@@ -362,7 +354,7 @@ class FSharpLanguageServer(SolidLanguageServer):
raise SolidLSPException(f"Failed to start F# language server: {e}")
# Send initialization
initialize_params = self._get_initialize_params()
initialize_params = self._create_initialize_params()
log.info("Sending initialize request to F# language server")
try:
src/solidlsp/language_servers/godot_language_server.py
+7 -19
View File
@@ -8,14 +8,12 @@ The editor must be open with its built-in language server enabled (default).
import logging
import os
import pathlib
from collections.abc import Callable
from solidlsp.ls import SolidLanguageServer
from solidlsp.ls_config import LanguageServerConfig
from solidlsp.ls_process import LanguageServerInterface, TCPConnectionInfo, TCPLanguageServer
from solidlsp.lsp_protocol_handler.lsp_types import InitializeParams
from solidlsp.lsp_protocol_handler.server import ProcessLaunchInfo, StringDict
from solidlsp.lsp_protocol_handler.server import StringDict
from solidlsp.settings import SolidLSPSettings
log = logging.getLogger(__name__)
@@ -49,7 +47,7 @@ class GodotLanguageServer(SolidLanguageServer):
self._configured_request_timeout: float | None = None
# Dummy ProcessLaunchInfo — _create_language_server_interface() ignores it
super().__init__(config, repository_root_path, ProcessLaunchInfo(cmd=""), "gdscript", solidlsp_settings)
super().__init__(config, repository_root_path, None, "gdscript", solidlsp_settings)
def set_request_timeout(self, timeout: float | None) -> None:
"""Cap the timeout at the value configured in ls_specific_settings, if set."""
@@ -75,12 +73,8 @@ class GodotLanguageServer(SolidLanguageServer):
pass
return None
def _create_language_server_interface(
self,
process_launch_info: ProcessLaunchInfo,
logging_fn: Callable[[str, str, StringDict | str], None] | None,
) -> LanguageServerInterface:
settings: dict = self._custom_settings or {}
def _create_language_server_interface(self, logging_fn: Callable[[str, str, StringDict | str], None] | None) -> LanguageServerInterface:
settings = self._custom_settings or {}
port = settings.get("port", DEFAULT_GODOT_LS_PORT)
request_timeout = settings.get("request_timeout", DEFAULT_GODOT_REQUEST_TIMEOUT)
self._configured_request_timeout = settings.get("request_timeout")
@@ -93,14 +87,8 @@ class GodotLanguageServer(SolidLanguageServer):
request_timeout=request_timeout,
)
@staticmethod
def _get_initialize_params(repository_absolute_path: str) -> InitializeParams:
root_uri = pathlib.Path(repository_absolute_path).as_uri()
def _create_base_initialize_params(self) -> dict:
params = {
"processId": os.getpid(),
"rootUri": root_uri,
"rootPath": repository_absolute_path,
"workspaceFolders": [{"uri": root_uri, "name": os.path.basename(repository_absolute_path)}],
"capabilities": {
"textDocument": {
"synchronization": {"didSave": True, "dynamicRegistration": True},
@@ -125,7 +113,7 @@ class GodotLanguageServer(SolidLanguageServer):
},
},
}
return params # type: ignore
return params
def _start_server(self) -> None:
def do_nothing(params: dict) -> None:
@@ -143,7 +131,7 @@ class GodotLanguageServer(SolidLanguageServer):
log.info("Connecting to Godot LSP at %s:%d", self._conn_info.host, self._conn_info.port)
self.server.start()
initialize_params = self._get_initialize_params(self.repository_root_path)
initialize_params = self._create_initialize_params()
log.info("Sending LSP initialize request to Godot")
self.server.send.initialize(initialize_params)
self.server.notify.initialized({})
src/solidlsp/language_servers/gopls.py
+113 -34
View File
@@ -2,16 +2,16 @@ import hashlib
import json
import logging
import os
import pathlib
import re
import subprocess
from collections.abc import Hashable
from typing import Any, cast
from typing import Any
from overrides import override
from solidlsp.ls import RawDocumentSymbol, SolidLanguageServer
from solidlsp import ls_types
from solidlsp.ls import DocumentSymbols, LSPFileBuffer, RawDocumentSymbol, SolidLanguageServer, SymbolBodyFactory
from solidlsp.ls_config import LanguageServerConfig
from solidlsp.lsp_protocol_handler.lsp_types import InitializeParams
from solidlsp.lsp_protocol_handler.server import ProcessLaunchInfo
from solidlsp.settings import SolidLSPSettings
@@ -23,6 +23,12 @@ class Gopls(SolidLanguageServer):
Provides Go specific instantiation of the LanguageServer class using gopls.
"""
# matches a line prefix that consists solely of a leading `type`/`var`/`const` declaration
# keyword (with optional indentation and the whitespace before the declared identifier).
# gopls reports the symbol range of such single declarations starting at the identifier,
# i.e. after the keyword, whereas `func` declarations include the `func` keyword.
_LEADING_DECL_KEYWORD_RE = re.compile(r"(?P<indent>\s*)(?:type|var|const)\s+")
@classmethod
def supports_implementation_request(cls) -> bool:
return True
@@ -103,11 +109,10 @@ class Gopls(SolidLanguageServer):
super().__init__(config, repository_root_path, ProcessLaunchInfo(cmd="gopls", cwd=repository_root_path), "go", solidlsp_settings)
self.request_id = 0
def _get_initialize_params(self, repository_absolute_path: str) -> InitializeParams:
def _create_base_initialize_params(self) -> dict:
"""
Returns the initialize params for the Go Language Server.
"""
root_uri = pathlib.Path(repository_absolute_path).as_uri()
initialize_params: dict = {
"locale": "en",
"capabilities": {
@@ -122,15 +127,6 @@ class Gopls(SolidLanguageServer):
},
"workspace": {"workspaceFolders": True, "didChangeConfiguration": {"dynamicRegistration": True}},
},
"processId": os.getpid(),
"rootPath": repository_absolute_path,
"rootUri": root_uri,
"workspaceFolders": [
{
"uri": root_uri,
"name": os.path.basename(repository_absolute_path),
}
],
}
# Apply gopls-specific settings via initializationOptions
@@ -149,7 +145,7 @@ class Gopls(SolidLanguageServer):
log.debug("Applying gopls settings via initializationOptions: keys=%s", list(gopls_settings.keys()))
initialize_params["initializationOptions"] = gopls_settings
return cast(InitializeParams, initialize_params)
return initialize_params
def _validate_gopls_settings_dict(self, gopls_settings: object) -> dict:
if not isinstance(gopls_settings, dict):
@@ -173,12 +169,12 @@ class Gopls(SolidLanguageServer):
@override
def _document_symbols_cache_fingerprint(self) -> Hashable:
"""
Compute a deterministic fingerprint of the Go build context.
The fingerprint includes gopls_settings and selected env vars that affect symbol discovery.
"""
normalize_symbol_name_version = 1
request_document_symbols_impl_version = 2
return normalize_symbol_name_version, request_document_symbols_impl_version
@override
def _raw_document_symbols_cache_fingerprint(self) -> Hashable:
gopls_settings_raw = self._custom_settings.settings.get("gopls_settings")
gopls_settings: dict | None
@@ -197,23 +193,106 @@ class Gopls(SolidLanguageServer):
# Version processed symbols even when the build context itself is empty.
if gopls_settings is None and not env_subset:
return normalize_symbol_name_version
fingerprint_data: dict[str, object] = {
"env": env_subset,
"normalize_symbol_name_version": normalize_symbol_name_version,
}
if gopls_settings is not None:
fingerprint_data["gopls_settings"] = gopls_settings
canonical_json = self._canonical_json_or_raise(json, fingerprint_data)
return hashlib.sha256(canonical_json.encode("utf-8")).hexdigest()[:16]
return None
else:
fingerprint_data: dict[str, object] = {
"env": env_subset,
}
if gopls_settings is not None:
fingerprint_data["gopls_settings"] = gopls_settings
canonical_json = self._canonical_json_or_raise(json, fingerprint_data)
return hashlib.sha256(canonical_json.encode("utf-8")).hexdigest()[:16]
@override
def _normalize_symbol_name(self, symbol: RawDocumentSymbol, relative_file_path: str) -> str:
return symbol["name"].rsplit(".", 1)[-1]
@override
def request_document_symbols(self, relative_file_path: str, file_buffer: LSPFileBuffer | None = None) -> DocumentSymbols:
# Override to extend single `type`/`var`/`const` declaration ranges to include the leading
# keyword. gopls excludes the keyword from such ranges (unlike `func` declarations), which
# causes replace_symbol_body to drop the keyword from the symbol body and replacement range;
# a natural keyword-inclusive round-trip edit would then corrupt the file (e.g. `type Foo`
# becomes `type type Foo`). See _extend_go_symbol_range_to_include_leading_keyword.
document_symbols = super().request_document_symbols(relative_file_path, file_buffer=file_buffer)
if not document_symbols.root_symbols:
return document_symbols
# obtain the file lines and a body factory to recompute the bodies of extended symbols
with self._open_file_context(relative_file_path, file_buffer, open_in_ls=False) as file_data:
file_lines = file_data.split_lines()
body_factory = SymbolBodyFactory(file_data)
# extend ranges recursively, operating on copies so the cached symbols are not mutated.
# Children keep their `parent` back-pointer aimed at the original (un-extended) node;
# this is intentional and safe, because ancestor traversal only reads name/kind/overload
# index (see Symbol.iter_ancestors), none of which the extension changes. Rebinding the
# back-pointers would force a deep copy of every extended subtree for no observable gain.
def extend_symbol_and_children(symbol: ls_types.UnifiedSymbolInformation) -> ls_types.UnifiedSymbolInformation:
extended = self._extend_go_symbol_range_to_include_leading_keyword(symbol, file_lines, body_factory)
children = symbol.get("children")
if children:
if extended is symbol:
extended = symbol.copy()
extended["children"] = [extend_symbol_and_children(child) for child in children]
return extended
extended_root_symbols = [extend_symbol_and_children(sym) for sym in document_symbols.root_symbols]
return DocumentSymbols(extended_root_symbols)
def _extend_go_symbol_range_to_include_leading_keyword(
self,
symbol: ls_types.UnifiedSymbolInformation,
file_lines: list[str],
body_factory: SymbolBodyFactory,
) -> ls_types.UnifiedSymbolInformation:
"""
Extend a Go symbol's body range to include a leading `type`/`var`/`const` keyword.
gopls reports the range of a single `type`/`var`/`const` declaration starting at the
declared identifier (after the keyword), whereas the range of a `func` declaration includes
the `func` keyword. This asymmetry makes :meth:`replace_symbol_body` omit the keyword from
both the displayed body and the replacement range, so re-supplying the keyword in an edit
corrupts the file (e.g. ``type Foo`` becomes ``type type Foo``).
:param symbol: the symbol whose range may be extended.
:param file_lines: the lines of the file in which the symbol is defined.
:param body_factory: the factory used to recompute the symbol body from the extended range.
:return: a copy of the symbol with an extended range, or the original symbol if no leading
keyword precedes the identifier on the start line (e.g. for funcs or for grouped
declarations such as ``var ( ... )`` whose keyword sits on a separate line).
"""
# determine whether only a declaration keyword precedes the identifier on the start line
range_info = symbol["range"]
start_line = range_info["start"]["line"]
start_char = range_info["start"]["character"]
if start_line >= len(file_lines):
return symbol
prefix = file_lines[start_line][:start_char]
match = self._LEADING_DECL_KEYWORD_RE.fullmatch(prefix)
if match is None:
return symbol
# extend the range start back to the keyword (excluding indentation), updating both the
# symbol range and its location range so the replacement range covers the keyword
new_start = ls_types.Position(line=start_line, character=len(match.group("indent")))
extended = symbol.copy()
extended["range"] = ls_types.Range(start=new_start, end=range_info["end"])
location = extended.get("location")
if location:
location = location.copy()
if "range" in location:
location["range"] = ls_types.Range(start=new_start, end=location["range"]["end"])
extended["location"] = location
# recompute the body from the now-extended location range so the displayed body stays
# consistent with the replacement range; the stale body must be removed first, since the
# factory returns an existing SymbolBody as-is and otherwise reads the updated location range
extended.pop("body", None)
extended["body"] = body_factory.create_symbol_body(extended)
return extended
def _start_server(self) -> None:
"""Start gopls server process"""
@@ -233,7 +312,7 @@ class Gopls(SolidLanguageServer):
log.info("Starting gopls server process")
self.server.start()
initialize_params = self._get_initialize_params(self.repository_root_path)
initialize_params = self._create_initialize_params()
log.info("Sending initialize request from LSP client to LSP server and awaiting response")
init_response = self.server.send.initialize(initialize_params)
src/solidlsp/language_servers/groovy_language_server.py
+3 -20
View File
@@ -5,13 +5,11 @@ Provides Groovy specific instantiation of the LanguageServer class. Contains var
import dataclasses
import logging
import os
import pathlib
import shlex
from solidlsp.ls import SolidLanguageServer
from solidlsp.ls_config import Language, LanguageServerConfig
from solidlsp.ls_utils import FileUtils, PlatformUtils
from solidlsp.lsp_protocol_handler.lsp_types import InitializeParams
from solidlsp.lsp_protocol_handler.server import ProcessLaunchInfo
from solidlsp.settings import SolidLSPSettings
@@ -244,19 +242,11 @@ class GroovyLanguageServer(SolidLanguageServer):
" Ensure the JAR file is available at the configured path\n"
)
@staticmethod
def _get_initialize_params(repository_absolute_path: str) -> InitializeParams:
def _create_base_initialize_params(self) -> dict:
"""
Returns the initialize params for the Groovy Language Server.
"""
if not os.path.isabs(repository_absolute_path):
repository_absolute_path = os.path.abspath(repository_absolute_path)
root_uri = pathlib.Path(repository_absolute_path).as_uri()
initialize_params = {
"clientInfo": {"name": "Serena Groovy Client", "version": "1.0.0"},
"rootPath": repository_absolute_path,
"rootUri": root_uri,
"capabilities": {
"textDocument": {
"synchronization": {"dynamicRegistration": True, "didSave": True},
@@ -282,15 +272,8 @@ class GroovyLanguageServer(SolidLanguageServer):
}
},
},
"processId": os.getpid(),
"workspaceFolders": [
{
"uri": root_uri,
"name": os.path.basename(repository_absolute_path),
}
],
}
return initialize_params # type: ignore
return initialize_params
def _start_server(self) -> None:
"""
@@ -316,7 +299,7 @@ class GroovyLanguageServer(SolidLanguageServer):
log.info("Starting Groovy server process")
self.server.start()
initialize_params = self._get_initialize_params(self.repository_root_path)
initialize_params = self._create_initialize_params()
log.info("Sending initialize request from LSP client to LSP server and awaiting response")
init_response = self.server.send.initialize(initialize_params)
src/solidlsp/language_servers/haskell_language_server.py
+3 -17
View File
@@ -4,7 +4,6 @@ Provides Haskell specific instantiation of the LanguageServer class. Contains va
import logging
import os
import pathlib
import shutil
import time
from typing import Any
@@ -13,7 +12,6 @@ from overrides import override
from solidlsp.ls import SolidLanguageServer
from solidlsp.ls_config import LanguageServerConfig
from solidlsp.lsp_protocol_handler.lsp_types import InitializeParams
from solidlsp.lsp_protocol_handler.server import ProcessLaunchInfo
from solidlsp.settings import SolidLSPSettings
@@ -107,14 +105,11 @@ class HaskellLanguageServer(SolidLanguageServer):
def is_ignored_dirname(self, dirname: str) -> bool:
return super().is_ignored_dirname(dirname) or dirname in ["dist", "dist-newstyle", ".stack-work", ".cabal-sandbox"]
@staticmethod
def _get_initialize_params(repository_absolute_path: str) -> InitializeParams:
def _create_base_initialize_params(self) -> dict:
"""
Returns the initialize params for the Haskell Language Server.
"""
root_uri = pathlib.Path(repository_absolute_path).as_uri()
initialize_params = {
"clientInfo": {"name": "Serena", "version": "0.1.0"},
"locale": "en",
"capabilities": {
"workspace": {
@@ -299,17 +294,8 @@ class HaskellLanguageServer(SolidLanguageServer):
}
},
"trace": "verbose",
"processId": os.getpid(),
"rootPath": repository_absolute_path,
"rootUri": root_uri,
"workspaceFolders": [
{
"uri": root_uri,
"name": os.path.basename(repository_absolute_path),
}
],
}
return initialize_params # type: ignore
return initialize_params
def _start_server(self) -> None:
"""
@@ -364,7 +350,7 @@ class HaskellLanguageServer(SolidLanguageServer):
log.info("Starting Haskell Language Server process")
self.server.start()
initialize_params = self._get_initialize_params(self.repository_root_path)
initialize_params = self._create_initialize_params()
log.info("Sending initialize request from LSP client to LSP server and awaiting response")
init_response = self.server.send.initialize(initialize_params)
src/solidlsp/language_servers/haxe_language_server.py
+5 -16
View File
@@ -22,7 +22,7 @@ from solidlsp.ls import (
from solidlsp.ls_config import LanguageServerConfig
from solidlsp.ls_exceptions import SolidLSPException
from solidlsp.ls_types import Hover
from solidlsp.lsp_protocol_handler.lsp_types import DiagnosticSeverity, InitializeParams
from solidlsp.lsp_protocol_handler.lsp_types import DiagnosticSeverity
from solidlsp.settings import SolidLSPSettings
log = logging.getLogger(__name__)
@@ -202,13 +202,11 @@ class HaxeLanguageServer(SolidLanguageServer):
"dump",
]
def _get_initialize_params(self, repository_absolute_path: str) -> InitializeParams:
def _create_base_initialize_params(self) -> dict:
"""Return initialize params for the Haxe Language Server.
displayArguments are resolved from user-configured buildFile or auto-discovered .hxml files.
"""
root_uri = pathlib.Path(repository_absolute_path).as_uri()
# 1. Check for user-configured .hxml path
configured_build_file = self._custom_settings.get("buildFile")
if configured_build_file:
@@ -216,7 +214,7 @@ class HaxeLanguageServer(SolidLanguageServer):
display_arguments = [configured_build_file]
else:
# 2. Auto-discover .hxml files recursively
display_arguments = self._discover_hxml_file(repository_absolute_path)
display_arguments = self._discover_hxml_file(self.repository_root_path)
init_options: dict = {"displayArguments": display_arguments}
rename_source_folders = self._custom_settings.get("renameSourceFolders")
@@ -258,17 +256,8 @@ class HaxeLanguageServer(SolidLanguageServer):
},
},
"initializationOptions": init_options,
"processId": os.getpid(),
"rootPath": repository_absolute_path,
"rootUri": root_uri,
"workspaceFolders": [
{
"uri": root_uri,
"name": os.path.basename(repository_absolute_path),
}
],
}
return initialize_params # type: ignore[return-value]
return initialize_params
@staticmethod
def _discover_hxml_file(repository_absolute_path: str) -> list[str]:
@@ -377,7 +366,7 @@ class HaxeLanguageServer(SolidLanguageServer):
log.info("Starting Haxe server process")
self.server.start()
initialize_params = self._get_initialize_params(self.repository_root_path)
initialize_params = self._create_initialize_params()
log.info("Sending initialize request from LSP client to LSP server and awaiting response")
self.server.send.initialize(initialize_params)
src/solidlsp/language_servers/hlsl_language_server.py
+6 -14
View File
@@ -5,9 +5,8 @@ Supports HLSL, GLSL, and WGSL shader file formats.
import logging
import os
import pathlib
import shutil
from typing import Any, cast
from typing import Any
from overrides import override
@@ -17,7 +16,6 @@ from solidlsp.ls import (
SolidLanguageServer,
)
from solidlsp.ls_config import LanguageServerConfig
from solidlsp.lsp_protocol_handler.lsp_types import InitializeParams
from solidlsp.settings import SolidLSPSettings
from .common import RuntimeDependency, RuntimeDependencyCollection
@@ -83,7 +81,7 @@ class HlslLanguageServer(SolidLanguageServer):
base_url = f"{_GITHUB_RELEASE_BASE}/{tag}"
# macOS has no pre-built binaries; build from source via cargo install
cargo_install_cmd = ["cargo", "install", "shader_language_server", "--version", version, "--root", "."]
cargo_install_cmd = ["cargo", "install", "shader_language_server", "--version", version, "--locked", "--root", "."]
deps = RuntimeDependencyCollection(
[
@@ -143,7 +141,7 @@ class HlslLanguageServer(SolidLanguageServer):
raise FileNotFoundError(
"shader-language-server is not installed and no auto-install is available for your platform.\n"
"Please install it using one of the following methods:\n"
" cargo: cargo install shader_language_server\n"
" cargo: cargo install shader_language_server --locked\n"
" GitHub: Download from https://github.com/antaalt/shader-sense/releases\n"
"On macOS, install the Rust toolchain (https://rustup.rs) and Serena will build from source automatically.\n"
"See https://github.com/antaalt/shader-sense for more details."
@@ -167,13 +165,8 @@ class HlslLanguageServer(SolidLanguageServer):
def _create_launch_command(self, core_path: str) -> list[str]:
return [core_path, "--stdio"]
@staticmethod
def _get_initialize_params(repository_absolute_path: str) -> InitializeParams:
root_uri = pathlib.Path(repository_absolute_path).as_uri()
def _create_base_initialize_params(self) -> dict:
initialize_params = {
"processId": os.getpid(),
"rootPath": repository_absolute_path,
"rootUri": root_uri,
"locale": "en",
"capabilities": {
"textDocument": {
@@ -207,9 +200,8 @@ class HlslLanguageServer(SolidLanguageServer):
"configuration": True,
},
},
"workspaceFolders": [{"uri": root_uri, "name": os.path.basename(repository_absolute_path)}],
}
return cast(InitializeParams, initialize_params)
return initialize_params
@override
def _start_server(self) -> None:
@@ -237,7 +229,7 @@ class HlslLanguageServer(SolidLanguageServer):
log.info("Starting shader-language-server process")
self.server.start()
initialize_params = self._get_initialize_params(self.repository_root_path)
initialize_params = self._create_initialize_params()
log.info("Sending initialize request")
init_response = self.server.send.initialize(initialize_params)
src/solidlsp/language_servers/intelephense.py
+26 -15
View File
@@ -4,7 +4,6 @@ Provides PHP specific instantiation of the LanguageServer class using Intelephen
import logging
import os
import pathlib
import shutil
from time import sleep
@@ -13,7 +12,7 @@ from overrides import override
from solidlsp.ls import LanguageServerDependencyProvider, LanguageServerDependencyProviderSinglePath, SolidLanguageServer
from solidlsp.ls_config import LanguageServerConfig
from solidlsp.ls_utils import PlatformId, PlatformUtils
from solidlsp.lsp_protocol_handler.lsp_types import Definition, DefinitionParams, InitializeParams, LocationLink
from solidlsp.lsp_protocol_handler.lsp_types import Definition, DefinitionParams, DidChangeConfigurationParams, LocationLink
from solidlsp.settings import SolidLSPSettings
from ..lsp_protocol_handler import lsp_types
@@ -36,6 +35,11 @@ class Intelephense(SolidLanguageServer):
- maxMemory: sets intelephense.maxMemory
- maxFileSize: sets intelephense.files.maxSize
- ignore_vendor: whether or ignore directories named "vendor" (default: true)
- file_filter: list of additional file extensions (with leading dot) to treat as PHP
sources, e.g. [".module", ".inc"]. The extensions are added to the defaults (".php",
".phtml") of Serena's PHP source-file matcher and are pushed to Intelephense via
files.associations, such that the symbol tools and the language server treat the
same files as PHP sources (see #1710).
"""
@override
@@ -107,14 +111,19 @@ class Intelephense(SolidLanguageServer):
self._ignored_dirnames.add("vendor")
log.info(f"Ignoring the following directories for PHP projects: {', '.join(sorted(self._ignored_dirnames))}")
# extend the source-file matcher with any additional extensions configured via `file_filter`,
# such that Serena's symbol tools treat the added extensions as PHP sources as well (#1710)
file_filter = self._custom_settings.get("file_filter")
if file_filter:
self.language.get_source_fn_matcher().add_extensions(*file_filter)
def _create_dependency_provider(self) -> LanguageServerDependencyProvider:
return self.DependencyProvider(self._custom_settings, self._ls_resources_dir)
def _get_initialize_params(self, repository_absolute_path: str) -> InitializeParams:
def _create_base_initialize_params(self) -> dict:
"""
Returns the initialization params for the Intelephense Language Server.
"""
root_uri = pathlib.Path(repository_absolute_path).as_uri()
initialize_params = {
"locale": "en",
"capabilities": {
@@ -135,15 +144,6 @@ class Intelephense(SolidLanguageServer):
"symbol": {"dynamicRegistration": True},
},
},
"processId": os.getpid(),
"rootPath": repository_absolute_path,
"rootUri": root_uri,
"workspaceFolders": [
{
"uri": root_uri,
"name": os.path.basename(repository_absolute_path),
}
],
}
initialization_options = {}
# Add license key if provided via environment variable
@@ -159,7 +159,7 @@ class Intelephense(SolidLanguageServer):
initialization_options["intelephense.files.maxSize"] = max_file_size
initialize_params["initializationOptions"] = initialization_options
return initialize_params # type: ignore
return initialize_params
def _start_server(self) -> None:
"""Start Intelephense server process"""
@@ -180,7 +180,7 @@ class Intelephense(SolidLanguageServer):
log.info("Starting Intelephense server process")
self.server.start()
initialize_params = self._get_initialize_params(self.repository_root_path)
initialize_params = self._create_initialize_params()
log.info("Sending initialize request from LSP client to LSP server and awaiting response")
init_response = self.server.send.initialize(initialize_params)
@@ -195,6 +195,17 @@ class Intelephense(SolidLanguageServer):
self.server.notify.initialized({})
# push the file associations derived from the source-file matcher (the default extensions
# plus any configured via `file_filter`), such that Intelephense indexes the same set of
# files that Serena's symbol tools treat as PHP sources (#1710). This cannot be passed in
# the initialize request: Intelephense reads configuration exclusively from
# workspace/didChangeConfiguration (its initializationOptions only cover storagePath,
# globalStoragePath, clearCache and licenceKey).
associations = [f"*{ext}" for ext in self.language.get_source_fn_matcher().file_extensions]
intelephense_config: DidChangeConfigurationParams = {"settings": {"intelephense": {"files": {"associations": associations}}}}
log.info(f"Sending workspace/didChangeConfiguration with file associations: {associations}")
self.server.notify.workspace_did_change_configuration(intelephense_config)
# Intelephense server is typically ready immediately after initialization
# TODO: This is probably incorrect; the server does send an initialized notification, which we could wait for!
src/solidlsp/language_servers/jedi_server.py
+3 -19
View File
@@ -3,16 +3,12 @@ Provides Python specific instantiation of the LanguageServer class. Contains var
"""
import logging
import os
import pathlib
import threading
from typing import cast
from overrides import override
from solidlsp.ls import SolidLanguageServer
from solidlsp.ls_config import LanguageServerConfig
from solidlsp.lsp_protocol_handler.lsp_types import InitializeParams
from solidlsp.lsp_protocol_handler.server import ProcessLaunchInfo
from solidlsp.settings import SolidLSPSettings
@@ -40,18 +36,12 @@ class JediServer(SolidLanguageServer):
def is_ignored_dirname(self, dirname: str) -> bool:
return super().is_ignored_dirname(dirname) or dirname in ["venv", "__pycache__"]
@staticmethod
def _get_initialize_params(repository_absolute_path: str) -> InitializeParams:
def _create_base_initialize_params(self) -> dict:
"""
Returns the initialize params for the Jedi Language Server.
"""
root_uri = pathlib.Path(repository_absolute_path).as_uri()
initialize_params = {
"processId": os.getpid(),
"clientInfo": {"name": "Serena", "version": "0.1.0"},
"locale": "en",
"rootPath": repository_absolute_path,
"rootUri": root_uri,
# Note: this is not necessarily the minimal set of capabilities...
"capabilities": {
"workspace": {
@@ -143,14 +133,8 @@ class JediServer(SolidLanguageServer):
},
},
"trace": "verbose",
"workspaceFolders": [
{
"uri": root_uri,
"name": os.path.basename(repository_absolute_path),
}
],
}
return cast(InitializeParams, initialize_params)
return initialize_params
def _start_server(self) -> None:
"""
@@ -182,7 +166,7 @@ class JediServer(SolidLanguageServer):
log.info("Starting jedi-language-server server process")
self.server.start()
initialize_params = self._get_initialize_params(self.repository_root_path)
initialize_params = self._create_initialize_params()
log.info("Sending initialize request from LSP client to LSP server and awaiting response")
init_response = self.server.send.initialize(initialize_params)
Loaded 100 of 298 files, more files were not shown because too many files have changed in this diff. Show more