tiennm99/serena
1
0 Fork 0
mirror of https://github.com/tiennm99/serena.git synced 2026-10-15 08:17:15 +00:00
Code Issues Packages Projects Releases Wiki Activity

Merge branch 'main' into mypy_on_solidlsp

Browse Source 
Conflicts:
	src/solidlsp/language_servers/common.py
	src/solidlsp/language_servers/csharp_language_server.py
	src/solidlsp/language_servers/dart_language_server.py
	src/solidlsp/language_servers/intelephense.py
	src/solidlsp/language_servers/solargraph.py
	src/solidlsp/language_servers/typescript_language_server.py
	src/solidlsp/language_servers/vts_language_server.py
	src/solidlsp/ls.py
	src/solidlsp/ls_handler.py
This commit is contained in:
Dominik Jain authored and Dominik Jain committed 2025-10-23 20:54:42 +02:00
2 parents d777927716 1e06812d68
commit f5f1f267de
242 files changed
+20358 -2763
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

.github/FUNDING.yml
+3
View File
@@ -0,0 +1,3 @@
# These are supported funding model platforms
github: oraios
.github/ISSUE_TEMPLATE/config.yml
+1
View File
@@ -0,0 +1 @@
blank_issues_enabled: false
.github/ISSUE_TEMPLATE/feature_request.md
+20
View File
@@ -0,0 +1,20 @@
---
name: Feature request
about: Suggest an idea for this project
title: ''
labels: ''
assignees: ''
---
**Is your feature request related to a problem? Please describe.**
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]
**Describe the solution you'd like**
A clear and concise description of what you want to happen.
**Describe alternatives you've considered**
A clear and concise description of any alternative solutions or features you've considered.
**Additional context**
Add any other context or screenshots about the feature request here.
.github/ISSUE_TEMPLATE/issue--bug--performance-problem--question-.md
+27
View File
@@ -0,0 +1,27 @@
---
name: Issue (bug, performance problem, question)
about: General Issue
title: ''
labels: ''
assignees: ''
---
I have:
- [ ] Read the readme and verified that the issue cannot be solved by adjusting configuration
- [ ] Understood that Serena's dashboard can be disabled through the config
- [ ] Understood that by default a client session will start a separate instance of a Serena server.
- [ ] Understood that for multi-agent setups, the SSE mode should be used.
- [ ] Verified that non-project files are ignored using either gitignore or the corresponding setting in `.serena/project.yml`
- [ ] Have looked for similar issues and discussions, including closed ones
- [ ] Made sure it's an actual issue and not a question - those should be opened as discussion instead.
If you have encountered a real and new issue:
- [ ] I performed `<uv invocation> serena project health-check`
- [ ] I indexed the project as described in the readme
- [ ] Added sufficient explanation of my setup: the MCP client, the OS, the programming language, any config adjustments or relevant project specifics
- [ ] Explained how the issue arose, added instructions on how to reproduce it (if possible)
- [ ] If the issue happens on an open source project, I have added the link
- [ ] Wrote a meaningful title and description
.github/workflows/junie.yml
+21
View File
@@ -0,0 +1,21 @@
name: Junie
run-name: Junie run ${{ inputs.run_id }}
permissions:
contents: write
on:
workflow_dispatch:
inputs:
run_id:
description: "id of workflow process"
required: true
workflow_params:
description: "stringified params"
required: true
jobs:
call-workflow-passing-data:
uses: jetbrains-junie/junie-workflows/.github/workflows/ej-issue.yml@main
with:
workflow_params: ${{ inputs.workflow_params }}
.github/workflows/pytest.yml
+244 -9
View File
@@ -27,7 +27,17 @@ jobs:
python-version: "${{ matrix.python-version }}"
- uses: actions/setup-go@v5
with:
go-version: '>=1.17.0'
go-version: ">=1.17.0"
- name: Set up Node.js
uses: actions/setup-node@v4
with:
node-version: '20.x'
- name: Ensure cached directory exist before calling cache-related actions
shell: bash
run: |
mkdir -p $HOME/.serena/language_servers/static
mkdir -p $HOME/.cache/go-build
mkdir -p $HOME/go/bin
# Add Go bin directory to PATH for this workflow
# GITHUB_PATH is a special file that GitHub Actions uses to modify PATH
# Writing to this file adds the directory to the PATH for subsequent steps
@@ -47,23 +57,245 @@ jobs:
if: runner.os != 'Windows'
uses: erlef/setup-beam@v1
with:
elixir-version: '1.18.4'
otp-version: '26.1'
- name: Prepare java
uses: actions/setup-java@v3
with:
distribution: "temurin"
java-version: "17"
elixir-version: "1.18.4"
otp-version: "26.1"
# Erlang currently not tested in CI, random hangings on macos, always hangs on ubuntu
# In local tests, erlang seems to work though
# - name: Install Erlang Language Server
# if: runner.os != 'Windows'
# shell: bash
# run: |
# # Install rebar3 if not already available
# which rebar3 || (curl -fsSL https://github.com/erlang/rebar3/releases/download/3.23.0/rebar3 -o /tmp/rebar3 && chmod +x /tmp/rebar3 && sudo mv /tmp/rebar3 /usr/local/bin/rebar3)
# # Clone and build erlang_ls
# git clone https://github.com/erlang-ls/erlang_ls.git /tmp/erlang_ls
# cd /tmp/erlang_ls
# make install PREFIX=/usr/local
# # Ensure erlang_ls is in PATH
# echo "$HOME/.local/bin" >> $GITHUB_PATH
- name: Install clojure tools
uses: DeLaGuardo/setup-clojure@13.4
with:
cli: latest
- name: Setup Java (for JVM based languages)
uses: actions/setup-java@v4
with:
distribution: 'temurin'
java-version: '17'
- name: Install Terraform
uses: hashicorp/setup-terraform@v3
with:
terraform_version: "1.5.0"
terraform_wrapper: false
# - name: Install swift
# if: runner.os != 'Windows'
# uses: swift-actions/setup-swift@v2
# Installation of swift with the action screws with installation of ruby on macOS for some reason
# We can try again when version 3 of the action is released, where they will also use swiftly
# Until then, we use custom code to install swift. Sourcekit-lsp is installed automatically with swift
- name: Install Swift with swiftly (macOS)
if: runner.os == 'macOS'
run: |
echo "=== Installing swiftly on macOS ==="
curl -O https://download.swift.org/swiftly/darwin/swiftly.pkg && \
installer -pkg swiftly.pkg -target CurrentUserHomeDirectory && \
~/.swiftly/bin/swiftly init --quiet-shell-followup && \
. "${SWIFTLY_HOME_DIR:-$HOME/.swiftly}/env.sh" && \
hash -r
swiftly install --use 6.1.2
swiftly use 6.1.2
echo "~/.swiftly/bin" >> $GITHUB_PATH
echo "Swiftly installed successfully"
# Verify sourcekit-lsp is working before proceeding
echo "=== Verifying sourcekit-lsp installation ==="
which sourcekit-lsp || echo "Warning: sourcekit-lsp not found in PATH"
sourcekit-lsp --help || echo "Warning: sourcekit-lsp not responding"
- name: Install Swift with swiftly (Ubuntu)
if: runner.os == 'Linux'
run: |
echo "=== Installing swiftly on Ubuntu ==="
# Install dependencies BEFORE Swift to avoid exit code 1
sudo apt-get update
sudo apt-get -y install libcurl4-openssl-dev
curl -O https://download.swift.org/swiftly/linux/swiftly-$(uname -m).tar.gz && \
tar zxf swiftly-$(uname -m).tar.gz && \
./swiftly init --quiet-shell-followup && \
. "${SWIFTLY_HOME_DIR:-$HOME/.local/share/swiftly}/env.sh" && \
hash -r
swiftly install --use 6.1.2
swiftly use 6.1.2
echo "=== Adding Swift toolchain to PATH ==="
echo "$HOME/.local/share/swiftly/bin" >> $GITHUB_PATH
echo "Swiftly installed successfully!"
# Verify sourcekit-lsp is working before proceeding
echo "=== Verifying sourcekit-lsp installation ==="
which sourcekit-lsp || echo "Warning: sourcekit-lsp not found in PATH"
sourcekit-lsp --help || echo "Warning: sourcekit-lsp not responding"
- name: Install Ruby
uses: ruby/setup-ruby@v1
with:
ruby-version: '3.4'
- name: Install Ruby language server
shell: bash
run: gem install ruby-lsp
- name: Install R
uses: r-lib/actions/setup-r@v2
with:
r-version: '4.4.2'
use-public-rspm: true
- name: Install R language server
shell: bash
run: |
Rscript -e "install.packages('languageserver', repos='https://cloud.r-project.org')"
- name: Set up Julia
uses: julia-actions/setup-julia@v2
with:
version: '1.10'
- name: Install Julia LanguageServer
shell: bash
run: julia -e 'using Pkg; Pkg.add("LanguageServer")'
- name: Install Zig
uses: goto-bus-stop/setup-zig@v2
with:
version: 0.14.1
- name: Install ZLS (Zig Language Server)
shell: bash
run: |
if [[ "${{ runner.os }}" == "Linux" ]]; then
wget https://github.com/zigtools/zls/releases/download/0.14.0/zls-x86_64-linux.tar.xz
tar -xf zls-x86_64-linux.tar.xz
sudo mv zls /usr/local/bin/
rm zls-x86_64-linux.tar.xz
elif [[ "${{ runner.os }}" == "macOS" ]]; then
wget https://github.com/zigtools/zls/releases/download/0.14.0/zls-x86_64-macos.tar.xz
tar -xf zls-x86_64-macos.tar.xz
sudo mv zls /usr/local/bin/
rm zls-x86_64-macos.tar.xz
elif [[ "${{ runner.os }}" == "Windows" ]]; then
curl -L -o zls.zip https://github.com/zigtools/zls/releases/download/0.14.0/zls-x86_64-windows.zip
unzip -o zls.zip
mkdir -p "$HOME/bin"
mv zls.exe "$HOME/bin/"
echo "$HOME/bin" >> $GITHUB_PATH
rm zls.zip
fi
- name: Install Lua Language Server
shell: bash
run: |
LUA_LS_VERSION="3.15.0"
LUA_LS_DIR="$HOME/.serena/language_servers/lua"
mkdir -p "$LUA_LS_DIR"
if [[ "${{ runner.os }}" == "Linux" ]]; then
if [[ "$(uname -m)" == "x86_64" ]]; then
wget https://github.com/LuaLS/lua-language-server/releases/download/${LUA_LS_VERSION}/lua-language-server-${LUA_LS_VERSION}-linux-x64.tar.gz
tar -xzf lua-language-server-${LUA_LS_VERSION}-linux-x64.tar.gz -C "$LUA_LS_DIR"
else
wget https://github.com/LuaLS/lua-language-server/releases/download/${LUA_LS_VERSION}/lua-language-server-${LUA_LS_VERSION}-linux-arm64.tar.gz
tar -xzf lua-language-server-${LUA_LS_VERSION}-linux-arm64.tar.gz -C "$LUA_LS_DIR"
fi
chmod +x "$LUA_LS_DIR/bin/lua-language-server"
# Create wrapper script instead of symlink to ensure supporting files are found
echo '#!/bin/bash' | sudo tee /usr/local/bin/lua-language-server > /dev/null
echo 'cd "${HOME}/.serena/language_servers/lua/bin"' | sudo tee -a /usr/local/bin/lua-language-server > /dev/null
echo 'exec ./lua-language-server "$@"' | sudo tee -a /usr/local/bin/lua-language-server > /dev/null
sudo chmod +x /usr/local/bin/lua-language-server
rm lua-language-server-*.tar.gz
elif [[ "${{ runner.os }}" == "macOS" ]]; then
if [[ "$(uname -m)" == "x86_64" ]]; then
wget https://github.com/LuaLS/lua-language-server/releases/download/${LUA_LS_VERSION}/lua-language-server-${LUA_LS_VERSION}-darwin-x64.tar.gz
tar -xzf lua-language-server-${LUA_LS_VERSION}-darwin-x64.tar.gz -C "$LUA_LS_DIR"
else
wget https://github.com/LuaLS/lua-language-server/releases/download/${LUA_LS_VERSION}/lua-language-server-${LUA_LS_VERSION}-darwin-arm64.tar.gz
tar -xzf lua-language-server-${LUA_LS_VERSION}-darwin-arm64.tar.gz -C "$LUA_LS_DIR"
fi
chmod +x "$LUA_LS_DIR/bin/lua-language-server"
# Create wrapper script instead of symlink to ensure supporting files are found
echo '#!/bin/bash' | sudo tee /usr/local/bin/lua-language-server > /dev/null
echo 'cd "${HOME}/.serena/language_servers/lua/bin"' | sudo tee -a /usr/local/bin/lua-language-server > /dev/null
echo 'exec ./lua-language-server "$@"' | sudo tee -a /usr/local/bin/lua-language-server > /dev/null
sudo chmod +x /usr/local/bin/lua-language-server
rm lua-language-server-*.tar.gz
elif [[ "${{ runner.os }}" == "Windows" ]]; then
curl -L -o lua-ls.zip https://github.com/LuaLS/lua-language-server/releases/download/${LUA_LS_VERSION}/lua-language-server-${LUA_LS_VERSION}-win32-x64.zip
unzip -o lua-ls.zip -d "$LUA_LS_DIR"
# For Windows, we'll add the bin directory directly to PATH
# The lua-language-server.exe can find its supporting files relative to its location
echo "$LUA_LS_DIR/bin" >> $GITHUB_PATH
rm lua-ls.zip
fi
- name: Install Perl::LanguageServer
if: runner.os != 'Windows'
shell: bash
run: |
if [[ "${{ runner.os }}" == "Linux" ]]; then
sudo apt-get update
sudo apt-get install -y cpanminus build-essential libanyevent-perl libio-aio-perl
elif [[ "${{ runner.os }}" == "macOS" ]]; then
brew install cpanminus
fi
PERL_MM_USE_DEFAULT=1 cpanm --notest --force Perl::LanguageServer
# Set up Perl local::lib environment for subsequent steps
echo "PERL5LIB=$HOME/perl5/lib/perl5${PERL5LIB:+:${PERL5LIB}}" >> $GITHUB_ENV
echo "PERL_LOCAL_LIB_ROOT=$HOME/perl5${PERL_LOCAL_LIB_ROOT:+:${PERL_LOCAL_LIB_ROOT}}" >> $GITHUB_ENV
echo "PERL_MB_OPT=--install_base \"$HOME/perl5\"" >> $GITHUB_ENV
echo "PERL_MM_OPT=INSTALL_BASE=$HOME/perl5" >> $GITHUB_ENV
echo "$HOME/perl5/bin" >> $GITHUB_PATH
- name: Install Elm
shell: bash
run: npm install -g elm@0.19.1-6
- name: Install Nix
if: runner.os != 'Windows' # Nix doesn't support Windows natively
uses: cachix/install-nix-action@v30
with:
nix_path: nixpkgs=channel:nixos-unstable
- name: Install nixd (Nix Language Server)
if: runner.os != 'Windows' # Skip on Windows since Nix isn't available
shell: bash
run: |
# Install nixd using nix
nix profile install github:nix-community/nixd
# Verify nixd is installed and working
if ! command -v nixd &> /dev/null; then
echo "nixd installation failed or not in PATH"
exit 1
fi
echo "$HOME/.nix-profile/bin" >> $GITHUB_PATH
- name: Verify Nix package build
if: runner.os != 'Windows' # Nix only supported on Linux/macOS
shell: bash
run: |
# Verify the flake builds successfully
nix build --no-link
- name: Install Regal (Rego Language Server)
shell: bash
run: |
REGAL_VERSION="0.36.1"
if [[ "${{ runner.os }}" == "Linux" ]]; then
if [[ "$(uname -m)" == "x86_64" ]]; then
curl -L -o regal https://github.com/StyraInc/regal/releases/download/v${REGAL_VERSION}/regal_Linux_x86_64
else
curl -L -o regal https://github.com/StyraInc/regal/releases/download/v${REGAL_VERSION}/regal_Linux_arm64
fi
chmod +x regal
sudo mv regal /usr/local/bin/
elif [[ "${{ runner.os }}" == "macOS" ]]; then
if [[ "$(uname -m)" == "x86_64" ]]; then
curl -L -o regal https://github.com/StyraInc/regal/releases/download/v${REGAL_VERSION}/regal_Darwin_x86_64
else
curl -L -o regal https://github.com/StyraInc/regal/releases/download/v${REGAL_VERSION}/regal_Darwin_arm64
fi
chmod +x regal
sudo mv regal /usr/local/bin/
elif [[ "${{ runner.os }}" == "Windows" ]]; then
curl -L -o regal.exe https://github.com/StyraInc/regal/releases/download/v${REGAL_VERSION}/regal_Windows_x86_64.exe
mkdir -p "$HOME/bin"
mv regal.exe "$HOME/bin/"
echo "$HOME/bin" >> $GITHUB_PATH
fi
- name: Install uv
shell: bash
run: curl -LsSf https://astral.sh/uv/install.sh | sh
@@ -90,6 +322,9 @@ jobs:
- name: Install dependencies
shell: bash
run: uv pip install -e ".[dev]"
- name: Check formatting
shell: bash
run: uv run poe lint
- name: Test with pytest
shell: bash
run: uv run poe test
.gitignore
+22
View File
@@ -227,3 +227,25 @@ tmp/
/test/resources/repos/elixir/test_repo/deps
# Exception: Don't ignore Elixir test repository lib directory (contains source code)
!/test/resources/repos/elixir/test_repo/lib
# Exception: Don't ignore Nix test repository lib directory (contains source code)
!/test/resources/repos/nix/test_repo/lib
# Exception: Don't ignore Julia test repository lib directory (contains source code)
!/test/resources/repos/julia/test_repo/lib
# Swift
/test/resources/repos/swift/test_repo/.build
/test/resources/repos/swift/test_repo/.swiftpm
# Elm
/test/resources/repos/elm/test_repo/.elm/
/test/resources/repos/elm/test_repo/elm-stuff/
# Scala
.metals/
.bsp/
.scala-build/
.bloop/
bootstrap
test/resources/repos/scala/.bloop/
.serena/memories/adding_new_language_support_guide.md
+17 -1
View File
@@ -5,6 +5,7 @@ This guide explains how to add support for a new programming language to Serena.
## Overview
Adding a new language involves:
1. **Language Server Implementation** - Creating a language-specific server class
2. **Language Registration** - Adding the language to enums and configurations
3. **Test Repository** - Creating a minimal test project
@@ -154,6 +155,7 @@ test/resources/repos/new_language/test_repo/
### 3.2 Example Source Files
Create meaningful source files that demonstrate:
- **Classes/Types** - For symbol testing
- **Functions/Methods** - For reference finding
- **Imports/Dependencies** - For cross-file operations
@@ -184,9 +186,22 @@ class Program {
## Step 4: Test Suite
Testing the language server implementation is of crucial importance, and the tests will
form the main part of the review process. Make sure that the tests are up to the standard
of Serena to make the review go smoother.
General rules for tests:
1. Tests for symbols and references should always check that the expected symbol names and references were actually found.
Just testing that a list came back or that the result is not None is insufficient.
2. Tests should never be skipped, the only exception is skipping based on some package being available or on an unsupported OS.
3. Tests should run in CI, check if there is a suitable GitHub action for installing the dependencies.
### 4.1 Basic Tests
Create `test/solidlsp/new_language/test_new_language_basic.py`. You should at least test:
Create `test/solidlsp/new_language/test_new_language_basic.py`.
Have a look at the structure of existing tests, for example, in `test/solidlsp/php/test_php_basic.py`
You should at least test:
1. Finding symbols
2. Finding within-file references
@@ -203,6 +218,7 @@ Consider adding new cases to the parametrized tests in `test_serena_agent.py` fo
### 5 Documentation
Update:
- **README.md** - Add language to supported languages list
- **CHANGELOG.md** - Document the new language support
- **Language-specific docs** - Installation requirements, known issues
.serena/project.yml
+1 -1
View File
@@ -33,7 +33,7 @@ read_only: false
# * `find_referencing_symbols`: Finds symbols that reference the symbol at the given location (optionally filtered by type).
# * `find_symbol`: Performs a global (or local) search for symbols with/containing a given name/substring (optionally filtered by type).
# * `get_current_config`: Prints the current configuration of the agent, including the active and available projects, tools, contexts, and modes.
# * `get_symbols_overview`: Gets an overview of the top-level symbols defined in a given file or directory.
# * `get_symbols_overview`: Gets an overview of the top-level symbols defined in a given file.
# * `initial_instructions`: Gets the initial instructions for the current project.
# Should only be used in settings where the system prompt cannot be set,
# e.g. in clients you have no control over, like Claude Desktop.
CHANGELOG.md
+68 -2
View File
@@ -1,9 +1,75 @@
# Latest
# latest
Status of the `main` branch. Changes prior to the next official version change will appear here.
* Tools:
* Added `RenameSymbolTool` for renaming symbols across the codebase (if LS supports this operation).
* Language support:
* **Add support for Scala** via Metals language server (requires some [manual setup](docs/scala_setup_guide_for_serena.md))
* **Add support for Elm** via @elm-tooling/elm-language-server (automatically downloads if not installed; requires Elm compiler)
* **Add support for Perl** via Perl::LanguageServer with LSP integration for .pl, .pm, and .t files
* **Add support for AL (Application Language)** for Microsoft Dynamics 365 Business Central development. Requires VS Code AL extension (ms-dynamics-smb.al).
* **Add support for R** via the R languageserver package with LSP integration, performance optimizations, and fallback symbol extraction
* **Add support for Zig** via ZLS (cross-file references may not fully work on Windows)
* **Add support for Lua** via lua-language-server
* **Add support for Nix** requires nixd installation (Windows not supported)
* **Dart now officially supported**: Dart was always working, but now tests were added, and it is promoted to "officially supported"
* **Rust now uses already installed rustup**: The rust-analyzer is no longer bundled with Serena. Instead, it uses the rust-analyzer from your Rust toolchain managed by rustup. This ensures compatibility with your Rust version and eliminates outdated bundled binaries.
* **Kotlin now officially supported**: We now use the official Kotlin LS, tests run through and performance is good, even though the LS is in an early development stage.
* **Add support for Erlang** experimental, may hang or be slow, uses the recently archived [erlang_ls](https://github.com/erlang-ls/erlang_ls)
* **Ruby dual language server support**: Added ruby-lsp as the modern primary Ruby language server. Solargraph remains available as an experimental legacy option. ruby-lsp supports both .rb and .erb files, while Solargraph supports .rb files only.
* Client support:
* New mode `oaicompat-agent` and extensions in the openai tool compatibility, **permitting Serena to work with llama.cpp**
* General:
* Various fixes related to indexing, special paths and determation of ignored paths
* Decreased `TOOL_DEFAULT_MAX_ANSWER_LENGTH` to be in accordance with (below) typical max-tokens configurations
* Allow passing language server specific settings through `ls_specific_settings` field (in `serena_config.yml`)
# 0.1.4
## Summary
This likely is the last release before the stable version 1.0.0 which will come together with the jetbrains IDE extension.
We release it for users who install Serena from a tag, since the last tag cannot be installed due to a breaking change in the mcp dependency (see #381).
Since the last release, several new languages were supported, and the Serena CLI and configurability were significantly extended.
We thank all external contributors who made a lot of the improvements possible!
* General:
* **Initial instructions no longer need to be loaded by the user**
* Significantly extended CLI
* Removed `replace_regex` tool from `ide-assistant` and `codex` contexts.
The current string replacement tool in Claude Code seems to be sufficiently efficient and is better
integrated with the IDE. Users who want to enable `replace_regex` can do so by customizing the context.
* Configuration:
* Simplify customization of modes and contexts, including CLI support.
* Possibility to customize the system prompt and outputs of simple tools, including CLI support.
* Possibility to override tool descriptions through the context YAML.
* Prompt templates are now automatically adapted to the enabled tools.
* Several tools are now excluded by default, need to be included explicitly.
* New context for ChatGPT
* Language servers:
* Reliably detect language server termination and propagate the respective error all the way
back to the tool application, where an unexpected termination is handled by restarting the language server
and subsequently retrying the tool application.
* **Add support for Swift**
* **Add support for Bash**
* Enhance Solargraph (Ruby) integration
* Automatic Rails project detection via config/application.rb, Rakefile, and Gemfile analysis
* Ruby/Rails-specific exclude patterns for improved indexing performance (vendor/, .bundle/, tmp/, log/, coverage/)
* Enhanced error handling with detailed diagnostics and Ruby manager-specific installation instructions (rbenv, RVM, asdf)
* Improved LSP capability negotiation and analysis completion detection
* Better Bundler and Solargraph installation error messages with clear resolution steps
Fixes:
* Ignore `.git` in check for ignored paths and improve performance of `find_all_non_ignored_files`
* Fix language server startup issues on Windows when using Claude Code (which was due to
default shell reconfiguration imposed by Claude Code)
* Additional wait for initialization in C# language server before requesting references, allowing cross-file references to be found.
# 0.1.3
CLAUDE.md
+2 -2
View File
@@ -13,7 +13,7 @@ This file provides guidance to Claude Code (claude.ai/code) when working with co
**Test Markers:**
Available pytest markers for selective testing:
- `python`, `go`, `java`, `rust`, `typescript`, `php`, `csharp`, `elixir`, `terraform`, `clojure`
- `python`, `go`, `java`, `rust`, `typescript`, `php`, `perl`, `csharp`, `elixir`, `terraform`, `clojure`, `swift`, `bash`, `ruby`, `ruby_solargraph`
- `snapshot` - for symbolic editing operation tests
**Project Management:**
@@ -100,7 +100,7 @@ Configuration is loaded from (in order of precedence):
- **Symbol-based editing** - Uses LSP for precise code manipulation
- **Caching strategy** - Reduces language server overhead
- **Error recovery** - Automatic language server restart on crashes
- **Multi-language support** - 13+ languages with LSP integration
- **Multi-language support** - 16+ languages with LSP integration
- **MCP protocol** - Exposes tools to AI agents via Model Context Protocol
- **Async operation** - Non-blocking language server interactions
CONTRIBUTING.md
+4 -39
View File
@@ -2,7 +2,7 @@
Serena is under active development. We are just discovering what it can do and where the limitations lie.
Feel free to share your learnings by opening open new issues, feature requests and extensions.
Feel free to share your learnings by opening new issues, feature requests and extensions.
## Developer Environment Setup
@@ -21,30 +21,10 @@ You can install a virtual environment with the required as follows
1. Create a new virtual environment: `uv venv`
2. Activate the environment:
* On Linux/Unix/macOS: `source .venv/bin/activate`
* On Windows: `.venv\Scripts\activate.bat` (in cmd/ps) or `source .venv/Scripts/activate` (in git-bash)
* On Linux/Unix/macOS or Windows with Git Bash: `source .venv/bin/activate`
* On Windows outside of Git Bash: `.venv\Scripts\activate.bat` (in cmd/ps) or `source .venv/Scripts/activate` (in git-bash)
3. Install the required packages with all extras: `uv pip install --all-extras -r pyproject.toml -e .`
### Docker setup
Build the docker image with
```shell
docker build -t serena .
```
and run it with the repository mounted as a volume:
```shell
docker run -it --rm -v "$(pwd)":/workspace serena
```
You can also just run `bash docker_build_and_run.sh`, which will do both things
for you.
Note: For the Windows subsystem for Linux (WSL), you may need to adjust the path for the
volume.
## Running Tools Locally
The Serena tools (and in fact all Serena code) can be executed without an LLM, and also without
@@ -54,19 +34,4 @@ An example script for running tools is provided in [scripts/demo_run_tools.py](s
## Adding a New Supported Language
Serena interacts with code through language servers which are included in
the `solidlsp` package. It is rather easy to include a new supported language
if an LSP implementation for it exists. You just need to:
1. create a new subclass of `SolidLanguageServer`
2. add a new value to the `Language` enum
3. make a new `elif` case in the `SolidLanguageServer.create` method
4. add a test repo of the new language to `test/resources/repos/<new_language>/test_repo`
and new tests in `test/solidlsp/<new_language>`. Similar to existing tests for other languages
5. also add a new case to the parameterized tests in `test/serena/test_serena_agent`
The subclasses are typically easy to write, have a look at the
[PyrightLanguageServer](src/multilspy/language_servers/pyright_language_server/pyright_server.py)
for an example, or at any other implementation to see how non-python
dependencies for language servers are handled there.
There are also some tips from the multilspy admin [here](https://github.com/microsoft/multilspy/issues/5).
See the corresponding [memory](.serena/memories/adding_new_language_support_guide.md).
DOCKER.md
+21
View File
@@ -68,6 +68,8 @@ SERENA_DASHBOARD_PORT=8080 docker-compose up serena
docker-compose up serena-dev
```
Note: Edit the `compose.yaml` file to customize volume mounts for your projects.
### Using Docker directly
```bash
@@ -83,6 +85,24 @@ docker run -it --rm \
serena
```
### Using Docker Compose with Merge Compose files
To use Docker Compose with merge files, you can create a `compose.override.yml` file to customize the configuration:
```yaml
services:
serena:
# To work with projects, you must mount them as volumes:
volumes:
- ./my-project:/workspace/my-project
- /path/to/another/project:/workspace/another-project
# Add the context for the IDE assistant option:
command:
- "uv run --directory . serena-mcp-server --transport sse --port 9121 --host 0.0.0.0 --context ide-assistant"
```
See the [Docker Merge Compose files documentation](https://docs.docker.com/compose/how-tos/multiple-compose-files/merge/) for more details on using merge files.
## Accessing the Dashboard
Once running, access the web dashboard at:
@@ -105,6 +125,7 @@ volumes:
- `SERENA_DOCKER=1`: Set automatically to indicate Docker environment
- `SERENA_PORT`: MCP server port (default: 9121)
- `SERENA_DASHBOARD_PORT`: Web dashboard port (default: 24282)
- `INTELEPHENSE_LICENSE_KEY`: License key for Intelephense PHP LSP premium features (optional)
## Troubleshooting
Dockerfile
+28 -2
View File
@@ -3,27 +3,53 @@ FROM python:3.11-slim AS base
SHELL ["/bin/bash", "-c"]
# Set environment variables to make Python print directly to the terminal and avoid .pyc files.
ENV PYTHONUNBUFFERED 1
ENV PYTHONDONTWRITEBYTECODE 1
ENV PYTHONUNBUFFERED=1
ENV PYTHONDONTWRITEBYTECODE=1
# Install system dependencies required for package manager and build tools.
# sudo, wget, zip needed for some assistants, like junie
RUN apt-get update && apt-get install -y --no-install-recommends \
curl \
build-essential \
git \
ssh \
sudo \
wget \
zip \
unzip \
git \
&& rm -rf /var/lib/apt/lists/*
# Install pipx.
RUN python3 -m pip install --no-cache-dir pipx \
&& pipx ensurepath
# Install nodejs
ENV NVM_VERSION=0.40.3
ENV NODE_VERSION=22.18.0
RUN curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v${NVM_VERSION}/install.sh | bash
# standard location
ENV NVM_DIR=/root/.nvm
RUN . "$NVM_DIR/nvm.sh" && nvm install ${NODE_VERSION}
RUN . "$NVM_DIR/nvm.sh" && nvm use v${NODE_VERSION}
RUN . "$NVM_DIR/nvm.sh" && nvm alias default v${NODE_VERSION}
ENV PATH="${NVM_DIR}/versions/node/v${NODE_VERSION}/bin/:${PATH}"
# Add local bin to the path
ENV PATH="${PATH}:/root/.local/bin"
# Install the latest version of uv
RUN curl -LsSf https://astral.sh/uv/install.sh | sh
# Install Rust and rustup for rust-analyzer support (minimal profile)
ENV RUSTUP_HOME=/usr/local/rustup
ENV CARGO_HOME=/usr/local/cargo
ENV PATH="${CARGO_HOME}/bin:${PATH}"
RUN curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y \
--default-toolchain stable \
--profile minimal \
&& rustup component add rust-analyzer
# Set the working directory
WORKDIR /workspaces/serena
README.md
+392 -357
View File
File diff suppressed because it is too large. Load diff
compose.yaml
+6
View File
@@ -1,6 +1,10 @@
services:
serena:
image: serena:latest
# To work with projects, you must mount them as volumes:
# volumes:
# - ./my-project:/workspace/my-project
# - /path/to/another/project:/workspace/another-project
build:
context: ./
dockerfile: Dockerfile
@@ -12,6 +16,8 @@ services:
- SERENA_DOCKER=1
command:
- "uv run --directory . serena-mcp-server --transport sse --port 9121 --host 0.0.0.0"
# Add the context for the IDE assistant
# - "uv run --directory . serena-mcp-server --transport sse --port 9121 --host 0.0.0.0 --context ide-assistant"
serena-dev:
image: serena:dev
docs/custom_agent.md
+67
View File
@@ -0,0 +1,67 @@
# Custom Agents with Serena
As a reference implementation, we provide an integration with the [Agno](https://docs.agno.com/introduction/playground) agent framework.
Agno is a model-agnostic agent framework that allows you to turn Serena into an agent
(independent of the MCP technology) with a large number of underlying LLMs. While Agno has recently
added support for MCP servers out of the box, our Agno integration predates this and is a good illustration of how
easy it is to integrate Serena into an arbitrary agent framework.
Here's how it works:
1. Download the agent-ui code with npx
```shell
npx create-agent-ui@latest
```
or, alternatively, clone it manually:
```shell
git clone https://github.com/agno-agi/agent-ui.git
cd agent-ui
pnpm install
pnpm dev
```
2. Install serena with the optional requirements:
```shell
# You can also only select agno,google or agno,anthropic instead of all-extras
uv pip install --all-extras -r pyproject.toml -e .
```
3. Copy `.env.example` to `.env` and fill in the API keys for the provider(s) you
intend to use.
4. Start the agno agent app with
```shell
uv run python scripts/agno_agent.py
```
By default, the script uses Claude as the model, but you can choose any model
supported by Agno (which is essentially any existing model).
5. In a new terminal, start the agno UI with
```shell
cd agent-ui
pnpm dev
```
Connect the UI to the agent you started above and start chatting. You will have
the same tools as in the MCP server version.
Here is a short demo of Serena performing a small analysis task with the newest Gemini model:
https://github.com/user-attachments/assets/ccfcb968-277d-4ca9-af7f-b84578858c62
⚠️ IMPORTANT: In contrast to the MCP server approach, tool execution in the Agno UI does
not ask for the user's permission. The shell tool is particularly critical, as it can perform arbitrary code execution.
While we have never encountered any issues with
this in our testing with Claude, allowing this may not be entirely safe.
You may choose to disable certain tools for your setup in your Serena project's
configuration file (`.yml`).
## Other Agent Frameworks
It should be straightforward to incorporate Serena into any
agent framework (like [pydantic-ai](https://ai.pydantic.dev/), [langgraph](https://langchain-ai.github.io/langgraph/tutorials/introduction/) or others).
Typically, you need only to write an adapter for Serena's tools to the tool representation in the framework of your choice,
as was done by us for Agno with [SerenaAgnoToolkit](/src/serena/agno.py).
docs/scala_setup_guide_for_serena.md
+67
View File
@@ -0,0 +1,67 @@
# Scala Setup Guide for Serena
This guide explains how to prepare a Scala project so that Serena can provide reliable code intelligence via Metals (Scala LSP) and how to run Scala tests manually.
Serena automatically bootstraps the Metals language server using Coursier when needed. Your project, however, must be importable by a build server (BSP) — typically via Bloop or sbt’s built‑in BSP — so that Metals can compile and index your code.
---
## Prerequisites
Install the following on your system and ensure they are available on `PATH`:
- Java Development Kit (JDK). A modern LTS (e.g., 17 or 21) is recommended.
- `sbt`
- Coursier command (`cs`) or the legacy `coursier` launcher
- Serena uses `cs` if available; if only `coursier` exists, it will attempt to install `cs`. If neither is present, install Coursier first.
---
## Quick Start (Recommended: VS Code + Metals auto‑import)
1. Open your Scala project in VS Code.
2. When prompted by Metals, accept “Import build”. Wait until the import and initial compile/indexing finish.
3. Run the “Connect to build server” command (id: `build.connect`).
4. Once the import completes, start Serena in your project root and use it.
This flow ensures the `.bloop/` and (if applicable) `.metals/` directories are created and your build is known to the build server that Metals uses.
---
## Manual Setup (No VS Code)
Follow these steps if you prefer a manual setup or you are not using VS Code:
These instructions cover the setup for projects that use sbt as the build tool, with Bloop as the BSP server.
1. Add Bloop to `project/plugins.sbt` in your Scala project:
```scala
// project/plugins.sbt
addSbtPlugin("ch.epfl.scala" % "sbt-bloop" % "<version>")
```
Replace `<version>` with an appropriate current version from the Metals documentation.
2. Export Bloop configuration with sources:
```bash
sbt -Dbloop.export-jar-classifiers=sources bloopInstall
```
This creates a `.bloop/` directory containing your project’s build metadata for the BSP server.
3. Compile from sbt to verify the build:
```bash
sbt compile
```
4. Start Serena in your project root. Serena will bootstrap Metals (if not already present) and connect to the build server using the configuration exported above.
---
## Using Serena with Scala
- Serena automatically detects Scala files (`*.scala`, `*.sbt`) and will start a Metals process per project when needed.
- On first run, you may see messages like “Bootstrapping metals…” in the Serena logs — this is expected.
- Optimal results require that your project compiles successfully via the build server (BSP). If compilation fails, fix build errors in `sbt` first.
Notes:
- Ensure you completed the manual or auto‑import steps so that the build is compiled and indexed; otherwise, code navigation and references may be incomplete until the first successful compile.
## Reference
- Metals + sbt: [https://scalameta.org/metals/docs/build-tools/sbt](https://scalameta.org/metals/docs/build-tools/sbt)
docs/serena_on_chatgpt.md
+102
View File
@@ -0,0 +1,102 @@
# Connecting Serena MCP Server to ChatGPT via MCPO & Cloudflare Tunnel
This guide explains how to expose a **locally running Serena MCP server** (powered by MCPO) to the internet using **Cloudflare Tunnel**, and how to connect it to **ChatGPT as a Custom GPT with tool access**.
Once configured, ChatGPT becomes a powerful **coding agent** with direct access to your codebase, shell, and file system — so **read the security notes carefully**.
---
## Prerequisites
Make sure you have [uv](https://docs.astral.sh/uv/getting-started/installation/)
and [cloudflared](https://developers.cloudflare.com/cloudflare-one/connections/connect-networks/downloads/) installed.
## 1. Start the Serena MCP Server via MCPO
Run the following command to launch Serena as http server (assuming port 8000):
```bash
uvx mcpo --port 8000 --api-key <YOUR_SECRET_KEY> -- \
uvx --from git+https://github.com/oraios/serena \
serena start-mcp-server --context chatgpt --project $(pwd)
```
- `--api-key` is required to secure the server.
- `--project` should point to the root of your codebase.
You can also use other options, and you don't have to pass `--project` if you want to work on multiple projects
or want to activate it later. See
```shell
uvx --from git+https://github.com/oraios/serena serena start-mcp-server --help
```
---
## 2. Expose the Server Using Cloudflare Tunnel
Run:
```bash
cloudflared tunnel --url http://localhost:8000
```
This will give you a **public HTTPS URL** like:
```
https://serena-agent-tunnel.trycloudflare.com
```
Your server is now securely exposed to the internet.
---
## 3. Connect It to ChatGPT (Custom GPT)
### Steps:
1. Go to [ChatGPT → Explore GPTs → Create](https://chat.openai.com/gpts/editor)
2. During setup, click **“Add APIs”**
3. Set up **API Key authentication** with the auth type as **Bearer** and enter the api key you used to start the MCPO server.
4. In the **Schema** section, click on **import from URL** and paste `<cloudflared_url>/openapi.json` with the URL you got from the previous step.
5. Add the following line to the top of the imported JSON schema:
```
"servers": ["url": "<cloudflared_url>"],
```
**Important**: don't include a trailing slash at the end of the URL!
ChatGPT will read the schema and create functions automatically.
---
## Security Warning — Read Carefully
Depending on your configuration and enabled tools, Serena's MCP server may:
- Execute **arbitrary shell commands**
- Read, write, and modify **files in your codebase**
This gives ChatGPT the same powers as a remote developer on your machine.
### ⚠️ Key Rules:
- **NEVER expose your API key**
- **Only expose this server when needed**, and monitor its use.
In your project’s `.serena/project.yml` or global config, you can disable tools like:
```yaml
excluded_tools:
- execute_shell_command
- ...
read_only: true
```
This is strongly recommended if you want a read-only or safer agent.
---
## Final Thoughts
With this setup, ChatGPT becomes a coding assistant **running on your local code** — able to index, search, edit, and even run shell commands depending on your configuration.
Use responsibly, and keep security in mind.
flake.lock
Generated
+133
View File
@@ -0,0 +1,133 @@
{
"nodes": {
"flake-utils": {
"inputs": {
"systems": "systems"
},
"locked": {
"lastModified": 1731533236,
"narHash": "sha256-l0KFg5HjrsfsO/JpG+r7fRrqm12kzFHyUHqHCVpMMbI=",
"owner": "numtide",
"repo": "flake-utils",
"rev": "11707dc2f618dd54ca8739b309ec4fc024de578b",
"type": "github"
},
"original": {
"owner": "numtide",
"repo": "flake-utils",
"type": "github"
}
},
"nixpkgs": {
"locked": {
"lastModified": 1755615617,
"narHash": "sha256-HMwfAJBdrr8wXAkbGhtcby1zGFvs+StOp19xNsbqdOg=",
"owner": "NixOS",
"repo": "nixpkgs",
"rev": "20075955deac2583bb12f07151c2df830ef346b4",
"type": "github"
},
"original": {
"owner": "NixOS",
"ref": "nixos-unstable",
"repo": "nixpkgs",
"type": "github"
}
},
"pyproject-build-systems": {
"inputs": {
"nixpkgs": [
"nixpkgs"
],
"pyproject-nix": [
"pyproject-nix"
],
"uv2nix": [
"uv2nix"
]
},
"locked": {
"lastModified": 1755484659,
"narHash": "sha256-2FfbqsaHVQd12XFFUAinIMAuGO3853LONmva1gT3vKw=",
"owner": "pyproject-nix",
"repo": "build-system-pkgs",
"rev": "9778e87c2361810ff15e287ca5895c9da4a0e900",
"type": "github"
},
"original": {
"owner": "pyproject-nix",
"repo": "build-system-pkgs",
"type": "github"
}
},
"pyproject-nix": {
"inputs": {
"nixpkgs": [
"nixpkgs"
]
},
"locked": {
"lastModified": 1755886095,
"narHash": "sha256-oMQ1YERKqiac28LB1cKjl4d0gjvK98SbGejFMfK0WGE=",
"owner": "pyproject-nix",
"repo": "pyproject.nix",
"rev": "7ea903d50c20d4d3ea45ccaad7ee0d21996780c9",
"type": "github"
},
"original": {
"owner": "pyproject-nix",
"repo": "pyproject.nix",
"type": "github"
}
},
"root": {
"inputs": {
"flake-utils": "flake-utils",
"nixpkgs": "nixpkgs",
"pyproject-build-systems": "pyproject-build-systems",
"pyproject-nix": "pyproject-nix",
"uv2nix": "uv2nix"
}
},
"systems": {
"locked": {
"lastModified": 1681028828,
"narHash": "sha256-Vy1rq5AaRuLzOxct8nz4T6wlgyUR7zLU309k9mBC768=",
"owner": "nix-systems",
"repo": "default",
"rev": "da67096a3b9bf56a91d16901293e51ba5b49a27e",
"type": "github"
},
"original": {
"owner": "nix-systems",
"repo": "default",
"type": "github"
}
},
"uv2nix": {
"inputs": {
"nixpkgs": [
"nixpkgs"
],
"pyproject-nix": [
"pyproject-nix"
]
},
"locked": {
"lastModified": 1755887980,
"narHash": "sha256-MD1IhwuoDXQSYcQ/JfEpMxrk6pPl+aHPXSqZLOXLHks=",
"owner": "pyproject-nix",
"repo": "uv2nix",
"rev": "76778b5f457e416688324d19d8f02ec5e22cbe0c",
"type": "github"
},
"original": {
"owner": "pyproject-nix",
"repo": "uv2nix",
"type": "github"
}
}
},
"root": "root",
"version": 7
}
flake.nix
+108
View File
@@ -0,0 +1,108 @@
{
description = "A powerful coding agent toolkit providing semantic retrieval and editing capabilities (MCP server & Agno integration)";
inputs = {
nixpkgs.url = "github:NixOS/nixpkgs/nixos-unstable";
flake-utils = {
url = "github:numtide/flake-utils";
};
pyproject-nix = {
url = "github:pyproject-nix/pyproject.nix";
inputs.nixpkgs.follows = "nixpkgs";
};
uv2nix = {
url = "github:pyproject-nix/uv2nix";
inputs = {
pyproject-nix.follows = "pyproject-nix";
nixpkgs.follows = "nixpkgs";
};
};
pyproject-build-systems = {
url = "github:pyproject-nix/build-system-pkgs";
inputs = {
pyproject-nix.follows = "pyproject-nix";
uv2nix.follows = "uv2nix";
nixpkgs.follows = "nixpkgs";
};
};
};
outputs = {
nixpkgs,
uv2nix,
pyproject-nix,
pyproject-build-systems,
flake-utils,
...
}:
flake-utils.lib.eachDefaultSystem (system: let
pkgs = import nixpkgs {inherit system;};
inherit (pkgs) lib;
workspace = uv2nix.lib.workspace.loadWorkspace {workspaceRoot = ./.;};
overlay = workspace.mkPyprojectOverlay {
sourcePreference = "wheel"; # or sourcePreference = "sdist";
};
pyprojectOverrides = final: prev: {
# Add setuptools for packages that need it during build
ruamel-yaml-clib = prev.ruamel-yaml-clib.overrideAttrs (old: {
nativeBuildInputs = (old.nativeBuildInputs or []) ++ [
final.setuptools
];
});
};
python = pkgs.python311;
pythonSet =
(pkgs.callPackage pyproject-nix.build.packages {
inherit python;
}).overrideScope
(
lib.composeManyExtensions [
pyproject-build-systems.overlays.default
overlay
pyprojectOverrides
]
);
in rec {
formatter = pkgs.alejandra;
packages = {
serena = pythonSet.mkVirtualEnv "serena" workspace.deps.default;
default = packages.serena;
};
apps.default = {
type = "app";
program = "${packages.default}/bin/serena";
};
devShells = {
default = pkgs.mkShell {
packages = [
python
pkgs.uv
];
env =
{
UV_PYTHON_DOWNLOADS = "never";
UV_PYTHON = python.interpreter;
}
// lib.optionalAttrs pkgs.stdenv.isLinux {
LD_LIBRARY_PATH = lib.makeLibraryPath pkgs.pythonManylinuxPackages.manylinux1;
};
shellHook = ''
unset PYTHONPATH
'';
};
};
});
}
pyproject.toml
+27 -6
View File
@@ -4,7 +4,7 @@ requires = ["hatchling"]
[project]
name = "serena-agent"
version = "0.1.3"
version = "0.1.4"
description = ""
authors = [{ name = "Oraios AI", email = "info@oraios-ai.de" }]
readme = "README.md"
@@ -18,9 +18,9 @@ dependencies = [
"pyright>=1.1.396,<2",
"overrides>=7.7.0,<8",
"python-dotenv>=1.0.0, <2",
"mcp>=1.5.0",
"mcp==1.12.3",
"flask>=3.0.0",
"sensai-utils>=1.4.0",
"sensai-utils>=1.5.0",
"pydantic>=2.10.6",
"types-pyyaml>=6.0.12.20241230",
"pyyaml>=6.0.2",
@@ -45,7 +45,7 @@ explicit = true
[project.scripts]
serena = "serena.cli:top_level"
serena-mcp-server = "serena.cli:start_mcp_server"
index-project = "serena.cli:index_project" # deprecated
index-project = "serena.cli:index_project" # deprecated
[project.license]
text = "MIT"
@@ -63,6 +63,7 @@ dev = [
"toml-sort>=0.24.2",
"types-pyyaml>=6.0.12.20241230",
"syrupy>=4.9.1",
"types-requests>=2.32.4.20241230",
]
agno = ["agno>=1.2.6", "sqlalchemy>=2.0.40"]
google = ["google-genai>=1.8.0"]
@@ -112,7 +113,7 @@ PYDEVD_DISABLE_FILE_VALIDATION = "1"
# Uses PYTEST_MARKERS env var for default markers
# For custom markers, one can either adjust the env var or just use -m option in the command line,
# as the second -m option will override the first one.
test = "pytest test -vv -m \"${PYTEST_MARKERS:-not java and not rust}\""
test = "pytest test -vv -m \"${PYTEST_MARKERS:-not java and not rust and not erlang}\""
_black_check = "black --check src scripts test"
_ruff_check = "ruff check src scripts test"
_black_format = "black src scripts test"
@@ -166,6 +167,7 @@ ignore = [
"PLC0415",
"RUF002",
"RUF005",
"RUF059",
"SIM118",
"SIM108",
"E501",
@@ -235,6 +237,7 @@ ignore = [
"SIM117", # forbids nested with statements
"C400", # wants to unnecessarily force use of list comprehension
"UP037", # can incorrectly (!) convert quoted type to unquoted type, causing an error
"UP045", # imposes T | None instead of Optional[T]
]
unfixable = ["F841", "F601", "F602", "B018"]
extend-fixable = ["F401", "B905", "W291"]
@@ -253,13 +256,31 @@ markers = [
"python: language server running for Python",
"go: language server running for Go",
"java: language server running for Java",
"kotlin: language server running for kotlin",
"rust: language server running for Rust",
"typescript: language server running for TypeScript",
"php: language server running for PHP",
"perl: language server running for Perl",
"csharp: language server running for C#",
"elixir: language server running for Elixir",
"elm: language server running for Elm",
"terraform: language server running for Terraform",
"swift: language server running for Swift",
"bash: language server running for Bash",
"r: language server running for R",
"snapshot: snapshot tests for symbolic editing operations",
"ruby: language server running for Ruby (uses ruby-lsp)",
"zig: language server running for Zig",
"lua: language server running for Lua",
"nix: language server running for Nix",
"dart: language server running for Dart",
"erlang: language server running for Erlang",
"scala: language server running for Scala",
"scala: language server running for Scala",
"al: language server running for AL (Microsoft Dynamics 365 Business Central)",
"rego: language server running for Rego",
"markdown: language server running for Markdown",
"julia: Julia language server tests",
]
[tool.codespell]
@@ -267,4 +288,4 @@ markers = [
skip = '.git*,*.svg,*.lock,*.min.*'
check-hidden = true
# ignore-regex = ''
# ignore-words-list = ''
# ignore-words-list = ''
resources/vscode_sponsor_logo.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 13 KiB

scripts/demo_run_tools.py
+3 -7
View File
@@ -9,7 +9,7 @@ from pprint import pprint
from serena.agent import SerenaAgent
from serena.config.serena_config import SerenaConfig
from serena.constants import REPO_ROOT
from serena.tools import FindFileTool, FindReferencingSymbolsTool, SearchForPatternTool
from serena.tools import FindFileTool, FindReferencingSymbolsTool, GetSymbolsOverviewTool, SearchForPatternTool
if __name__ == "__main__":
agent = SerenaAgent(project=REPO_ROOT, serena_config=SerenaConfig(gui_log_window_enabled=False, web_dashboard=False))
@@ -18,13 +18,9 @@ if __name__ == "__main__":
find_refs_tool = agent.get_tool(FindReferencingSymbolsTool)
find_file_tool = agent.get_tool(FindFileTool)
search_pattern_tool = agent.get_tool(SearchForPatternTool)
overview_tool = agent.get_tool(GetSymbolsOverviewTool)
result = agent.execute_task(
lambda: search_pattern_tool.apply(
r"def request_full_.*?\).*?\)",
restrict_search_to_code_files=False,
relative_path="src/solidlsp",
paths_include_glob="**/ls.py",
)
lambda: overview_tool.apply("src/solidlsp/ls.py"),
)
pprint(json.loads(result))
scripts/gen_prompt_factory.py
+2 -2
View File
@@ -7,14 +7,14 @@ from pathlib import Path
from sensai.util import logging
from interprompt import autogenerate_prompt_factory_module
from serena.constants import PROMPT_TEMPLATES_DIR, REPO_ROOT
from serena.constants import PROMPT_TEMPLATES_DIR_INTERNAL, REPO_ROOT
log = logging.getLogger(__name__)
def main():
autogenerate_prompt_factory_module(
prompts_dir=PROMPT_TEMPLATES_DIR,
prompts_dir=PROMPT_TEMPLATES_DIR_INTERNAL,
target_module_path=str(Path(REPO_ROOT) / "src" / "serena" / "generated" / "generated_prompt_factory.py"),
)
src/interprompt/.syncCommitId.remote
+1 -1
View File
@@ -1 +1 @@
6bdf19b0fd0fa5a809e1989b334bbfe9d8a7051a
059d50b7d4d7e8fb9e7a13df7f7f33bae1aed5e2
src/interprompt/.syncCommitId.this
+1 -1
View File
@@ -1 +1 @@
6d9d3adac8e77fa72e55e2675c1bcd3a7331cde1
53ee7f47c08f29ae336567bcfaf89f79e7d447a2
src/interprompt/multilang_prompt.py
+84 -12
View File
@@ -1,7 +1,7 @@
import logging
import os
from enum import Enum
from typing import Any, Generic, TypeVar
from typing import Any, Generic, Literal, TypeVar
import yaml
from sensai.util.string import ToStringMixin
@@ -92,6 +92,9 @@ class _MultiLangContainer(Generic[T], ToStringMixin):
raise KeyError(f"Item for language '{lang_code}' already registered for name '{self.name}'")
self._lang2item[lang_code] = item
def has_item(self, lang_code: str = DEFAULT_LANG_CODE) -> bool:
return lang_code in self._lang2item
def get_item(self, lang: str = DEFAULT_LANG_CODE, fallback_mode: LanguageFallbackMode = LanguageFallbackMode.EXCEPTION) -> T:
"""
Gets the item for the given language.
@@ -183,6 +186,9 @@ class MultiLangPromptTemplate(ParameterizedTemplateInterface):
prompt_template = self.get_prompt_template(lang_code, fallback_mode)
return prompt_template.render(**params)
def has_item(self, lang_code: str = DEFAULT_LANG_CODE) -> bool:
return self._prompts_container.has_item(lang_code)
class MultiLangPromptList(_MultiLangContainer[PromptList]):
pass
@@ -212,44 +218,96 @@ class MultiLangPromptCollection:
The prompt names must be unique (for the same language) within the collection.
"""
def __init__(self, prompts_dir: str, fallback_mode: LanguageFallbackMode = LanguageFallbackMode.EXCEPTION) -> None:
def __init__(self, prompts_dir: str | list[str], fallback_mode: LanguageFallbackMode = LanguageFallbackMode.EXCEPTION) -> None:
"""
:param prompts_dir: the directory containing the prompt templates and prompt lists
:param prompts_dir: the directory containing the prompt templates and prompt lists.
If a list is provided, will look for prompt templates in the dirs from left to right
(first one containing the desired template wins).
:param fallback_mode: the fallback mode to use when a prompt template or prompt list is not found for the requested language.
May be reset after initialization.
"""
self._multi_lang_prompt_templates: dict[str, MultiLangPromptTemplate] = {}
self._multi_lang_prompt_lists: dict[str, MultiLangPromptList] = {}
self._load_from_disc(prompts_dir)
if isinstance(prompts_dir, str):
prompts_dir = [prompts_dir]
# Add prompts from multiple directories, prioritizing names from the left.
# If name collisions appear in the first directory, an error is raised (so the first directory should have no
# internal collisions, this helps in avoiding errors)
# For all following directories, on a collision the new value will be ignored.
# This also means that for the following directories, there is no error check on collisions internal to them.
# We assume that they are correct (i.e., they have no internal collisions).
first_prompts_dir, fallback_prompt_dirs = prompts_dir[0], prompts_dir[1:]
self._load_from_disc(first_prompts_dir, on_name_collision="raise")
for fallback_prompt_dir in fallback_prompt_dirs:
# already loaded prompts have priority
self._load_from_disc(fallback_prompt_dir, on_name_collision="skip")
self.fallback_mode = fallback_mode
def _add_prompt_template(self, name: str, template_str: str, lang_code: str = DEFAULT_LANG_CODE) -> None:
def _add_prompt_template(
self,
name: str,
template_str: str,
lang_code: str = DEFAULT_LANG_CODE,
on_name_collision: Literal["skip", "overwrite", "raise"] = "raise",
) -> None:
"""
:param name: name of the prompt template
:param template_str: the Jinja template string
:param lang_code: the language code for which to add the prompt template.
:param on_name_collision: how to deal with name/lang_code collisions
"""
allow_overwrite = False
prompt_template = PromptTemplate(name, template_str)
mlpt = self._multi_lang_prompt_templates.get(name)
if mlpt is None:
mlpt = MultiLangPromptTemplate(name)
self._multi_lang_prompt_templates[name] = mlpt
mlpt.add_prompt_template(prompt_template, lang_code=lang_code)
if mlpt.has_item(lang_code):
if on_name_collision == "raise":
raise KeyError(f"Prompt '{name}' for {lang_code} already exists!")
if on_name_collision == "skip":
log.debug(f"Skipping prompt '{name}' since it already exists.")
return
elif on_name_collision == "overwrite":
allow_overwrite = True
mlpt.add_prompt_template(prompt_template, lang_code=lang_code, allow_overwrite=allow_overwrite)
def _add_prompt_list(self, name: str, prompt_list: list[str], lang_code: str = DEFAULT_LANG_CODE) -> None:
def _add_prompt_list(
self,
name: str,
prompt_list: list[str],
lang_code: str = DEFAULT_LANG_CODE,
on_name_collision: Literal["skip", "overwrite", "raise"] = "raise",
) -> None:
"""
:param name: name of the prompt list
:param prompt_list: a list of prompts
:param lang_code: the language code for which to add the prompt list.
:param on_name_collision: how to deal with name/lang_code collisions
"""
allow_overwrite = False
multilang_prompt_list = self._multi_lang_prompt_lists.get(name)
if multilang_prompt_list is None:
multilang_prompt_list = MultiLangPromptList(name)
self._multi_lang_prompt_lists[name] = multilang_prompt_list
multilang_prompt_list.add_item(PromptList(prompt_list), lang_code=lang_code)
if multilang_prompt_list.has_item(lang_code):
if on_name_collision == "raise":
raise KeyError(f"Prompt '{name}' for {lang_code} already exists!")
if on_name_collision == "skip":
log.debug(f"Skipping prompt '{name}' since it already exists.")
return
elif on_name_collision == "overwrite":
allow_overwrite = True
multilang_prompt_list.add_item(PromptList(prompt_list), lang_code=lang_code, allow_overwrite=allow_overwrite)
def _load_from_disc(self, prompts_dir: str) -> None:
"""Loads all prompt templates and prompt lists from yaml files in the given directory."""
def _load_from_disc(self, prompts_dir: str, on_name_collision: Literal["skip", "overwrite", "raise"] = "raise") -> None:
"""Loads all prompt templates and prompt lists from yaml files in the given directory.
:param prompts_dir:
:param on_name_collision: how to deal with name/lang_code collisions
"""
for fn in os.listdir(prompts_dir):
if not fn.endswith((".yml", ".yaml")):
log.debug(f"Skipping non-YAML file: {fn}")
@@ -266,9 +324,11 @@ class MultiLangPromptCollection:
# add the data to the collection
for prompt_name, prompt_template_or_list in prompts_data.items():
if isinstance(prompt_template_or_list, list):
self._add_prompt_list(prompt_name, prompt_template_or_list, lang_code=lang_code)
self._add_prompt_list(prompt_name, prompt_template_or_list, lang_code=lang_code, on_name_collision=on_name_collision)
elif isinstance(prompt_template_or_list, str):
self._add_prompt_template(prompt_name, prompt_template_or_list, lang_code=lang_code)
self._add_prompt_template(
prompt_name, prompt_template_or_list, lang_code=lang_code, on_name_collision=on_name_collision
)
else:
raise ValueError(
f"Invalid prompt type for {prompt_name} in file {path} (should be str or list): {prompt_template_or_list}"
@@ -306,6 +366,18 @@ class MultiLangPromptCollection:
"""The PromptList object for the given prompt name and language code."""
return self.get_multilang_prompt_list(prompt_name).get_item(lang_code)
def _has_prompt_list(self, prompt_name: str, lang_code: str = DEFAULT_LANG_CODE) -> bool:
multi_lang_prompt_list = self._multi_lang_prompt_lists.get(prompt_name)
if multi_lang_prompt_list is None:
return False
return multi_lang_prompt_list.has_item(lang_code)
def _has_prompt_template(self, prompt_name: str, lang_code: str = DEFAULT_LANG_CODE) -> bool:
multi_lang_prompt_template = self._multi_lang_prompt_templates.get(prompt_name)
if multi_lang_prompt_template is None:
return False
return multi_lang_prompt_template.has_item(lang_code)
def render_prompt_template(
self,
prompt_name: str,
src/interprompt/prompt_factory.py
+4 -2
View File
@@ -10,9 +10,11 @@ log = logging.getLogger(__name__)
class PromptFactoryBase:
"""Base class for auto-generated prompt factory classes."""
def __init__(self, prompts_dir: str, lang_code: str = DEFAULT_LANG_CODE, fallback_mode=LanguageFallbackMode.EXCEPTION):
def __init__(self, prompts_dir: str | list[str], lang_code: str = DEFAULT_LANG_CODE, fallback_mode=LanguageFallbackMode.EXCEPTION):
"""
:param prompts_dir: the directory containing the prompt templates and prompt lists
:param prompts_dir: the directory containing the prompt templates and prompt lists.
If a list is provided, will look for prompt templates in the dirs from left to right
(first one containing the desired template wins).
:param lang_code: the language code to use for retrieving the prompt templates and prompt lists.
Leave as `default` for single-language use cases.
:param fallback_mode: the fallback mode to use when a prompt template or prompt list is not found for the requested language.
src/serena/__init__.py
+1 -1
View File
@@ -1,4 +1,4 @@
__version__ = "0.1.3"
__version__ = "0.1.4"
import logging
src/serena/agent.py
+75 -75
View File
@@ -8,24 +8,24 @@ import platform
import sys
import threading
import webbrowser
from collections import defaultdict
from collections.abc import Callable
from concurrent.futures import Future, ThreadPoolExecutor
from logging import Logger
from pathlib import Path
from typing import TYPE_CHECKING, Any, Optional, TypeVar
from sensai.util import logging
from sensai.util.logging import LogTime
from interprompt.jinja_template import JinjaTemplate
from serena import serena_version
from serena.analytics import RegisteredTokenCountEstimator, ToolUsageStats
from serena.config.context_mode import RegisteredContext, SerenaAgentContext, SerenaAgentMode
from serena.config.serena_config import SerenaConfig, ToolInclusionDefinition, ToolSet, get_serena_managed_in_project_dir
from serena.config.serena_config import SerenaConfig, ToolInclusionDefinition, ToolSet
from serena.dashboard import SerenaDashboardAPI
from serena.project import Project
from serena.prompt_factory import SerenaPromptFactory
from serena.tools import ActivateProjectTool, Tool, ToolRegistry
from serena.tools import ActivateProjectTool, GetCurrentConfigTool, Tool, ToolMarker, ToolRegistry
from serena.util.inspection import iter_subclasses
from serena.util.logging import MemoryLogHandler
from solidlsp import SolidLanguageServer
@@ -42,53 +42,21 @@ class ProjectNotFoundError(Exception):
pass
class LinesRead:
def __init__(self) -> None:
self.files: dict[str, set[tuple[int, int]]] = defaultdict(lambda: set())
class AvailableTools:
def __init__(self, tools: list[Tool]):
"""
:param tools: the list of available tools
"""
self.tools = tools
self.tool_names = [tool.get_name_from_cls() for tool in tools]
self.tool_marker_names = set()
for marker_class in iter_subclasses(ToolMarker):
for tool in tools:
if isinstance(tool, marker_class):
self.tool_marker_names.add(marker_class.__name__)
def add_lines_read(self, relative_path: str, lines: tuple[int, int]) -> None:
self.files[relative_path].add(lines)
def were_lines_read(self, relative_path: str, lines: tuple[int, int]) -> bool:
lines_read_in_file = self.files[relative_path]
return lines in lines_read_in_file
def invalidate_lines_read(self, relative_path: str) -> None:
if relative_path in self.files:
del self.files[relative_path]
class MemoriesManager:
def __init__(self, project_root: str):
self._memory_dir = Path(get_serena_managed_in_project_dir(project_root)) / "memories"
self._memory_dir.mkdir(parents=True, exist_ok=True)
def _get_memory_file_path(self, name: str) -> Path:
# strip all .md from the name. Models tend to get confused, sometimes passing the .md extension and sometimes not.
name = name.replace(".md", "")
filename = f"{name}.md"
return self._memory_dir / filename
def load_memory(self, name: str) -> str:
memory_file_path = self._get_memory_file_path(name)
if not memory_file_path.exists():
return f"Memory file {name} not found, consider creating it with the `write_memory` tool if you need it."
with open(memory_file_path, encoding="utf-8") as f:
return f.read()
def save_memory(self, name: str, content: str) -> str:
memory_file_path = self._get_memory_file_path(name)
with open(memory_file_path, "w", encoding="utf-8") as f:
f.write(content)
return f"Memory {name} written."
def list_memories(self) -> list[str]:
return [f.name.replace(".md", "") for f in self._memory_dir.iterdir() if f.is_file()]
def delete_memory(self, name: str) -> str:
memory_file_path = self._get_memory_file_path(name)
memory_file_path.unlink()
return f"Memory {name} deleted."
def __len__(self) -> int:
return len(self.tools)
class SerenaAgent:
@@ -116,9 +84,13 @@ class SerenaAgent:
# obtain serena configuration using the decoupled factory function
self.serena_config = serena_config or SerenaConfig.from_config_file()
# project-specific instances, which will be initialized upon project activation
self._active_project: Project | None = None
self.language_server: SolidLanguageServer | None = None
# adjust log level
serena_log_level = self.serena_config.log_level
if Logger.root.level > serena_log_level:
if Logger.root.level != serena_log_level:
log.info(f"Changing the root logger level to {serena_log_level}")
Logger.root.setLevel(serena_log_level)
@@ -164,7 +136,7 @@ class SerenaAgent:
# start the dashboard (web frontend), registering its log handler
if self.serena_config.web_dashboard:
self._dashboard_thread, port = SerenaDashboardAPI(
get_memory_log_handler(), tool_names, tool_usage_stats=self._tool_usage_stats
get_memory_log_handler(), tool_names, agent=self, tool_usage_stats=self._tool_usage_stats
).run_in_thread()
dashboard_url = f"http://127.0.0.1:{port}/dashboard/index.html"
log.info("Serena web dashboard started at %s", dashboard_url)
@@ -181,16 +153,18 @@ class SerenaAgent:
log.info("Available projects: {}".format(", ".join(self.serena_config.project_names)))
log.info(f"Loaded tools ({len(self._all_tools)}): {', '.join([tool.get_name_from_cls() for tool in self._all_tools.values()])}")
self._check_shell_settings()
# determine the base toolset defining the set of exposed tools (which e.g. the MCP shall see),
# limited by the Serena config, the context (which is fixed for the session) and JetBrains mode
tool_inclusion_definitions: list[ToolInclusionDefinition] = [self.serena_config, self._context]
if self._context.name == RegisteredContext.IDE_ASSISTANT.value:
tool_inclusion_definitions.extend(self._ide_context_tool_inclusion_definitions(project))
tool_inclusion_definitions.extend(self._ide_assistant_context_tool_inclusion_definitions(project))
if self.serena_config.jetbrains:
tool_inclusion_definitions.append(SerenaAgentMode.from_name_internal("jetbrains"))
self._base_tool_set = ToolSet.default().apply(*tool_inclusion_definitions)
self._exposed_tools = {tc: t for tc, t in self._all_tools.items() if self._base_tool_set.includes_name(t.get_name())}
self._exposed_tools = AvailableTools([t for t in self._all_tools.values() if self._base_tool_set.includes_name(t.get_name())])
log.info(f"Number of exposed tools: {len(self._exposed_tools)}")
# create executor for starting the language server and running tools in another thread
@@ -203,13 +177,6 @@ class SerenaAgent:
self.prompt_factory = SerenaPromptFactory()
self._project_activation_callback = project_activation_callback
# project-specific instances, which will be initialized upon project activation
self._active_project: Project | None = None
self._active_project_root: str | None = None
self.language_server: SolidLanguageServer | None = None
self.memories_manager: MemoriesManager | None = None
self.lines_read: LinesRead | None = None
# set the active modes
if modes is None:
modes = SerenaAgentMode.load_default_modes()
@@ -225,11 +192,28 @@ class SerenaAgent:
except Exception as e:
log.error(f"Error activating project '{project}' at startup: {e}", exc_info=e)
def _ide_context_tool_inclusion_definitions(self, project_root_or_name: str | None) -> list[ToolInclusionDefinition]:
def get_context(self) -> SerenaAgentContext:
return self._context
def get_tool_description_override(self, tool_name: str) -> str | None:
return self._context.tool_description_overrides.get(tool_name, None)
def _check_shell_settings(self) -> None:
# On Windows, Claude Code sets COMSPEC to Git-Bash (often even with a path containing spaces),
# which causes all sorts of trouble, preventing language servers from being launched correctly.
# So we make sure that COMSPEC is unset if it has been set to bash specifically.
if platform.system() == "Windows":
comspec = os.environ.get("COMSPEC", "")
if "bash" in comspec:
os.environ["COMSPEC"] = "" # force use of default shell
log.info("Adjusting COMSPEC environment variable to use the default shell instead of '%s'", comspec)
def _ide_assistant_context_tool_inclusion_definitions(self, project_root_or_name: str | None) -> list[ToolInclusionDefinition]:
"""
In the IDE assistant context, the agent is assumed to work on a single project, and we thus
want to apply that project's tool exclusions/inclusions from the get-go, limiting the set
of tools that will be exposed to the client.
Furthermore, we disable tools that are only relevant for project activation.
So if the project exists, we apply all the aforementioned exclusions.
:param project_root_or_name: the project root path or project name
@@ -242,7 +226,11 @@ class SerenaAgent:
# and provide responses to the client immediately.
project = self.load_project_from_path_or_name(project_root_or_name, autogenerate=False)
if project is not None:
tool_inclusion_definitions.append(ToolInclusionDefinition(excluded_tools=[ActivateProjectTool.get_name_from_cls()]))
tool_inclusion_definitions.append(
ToolInclusionDefinition(
excluded_tools=[ActivateProjectTool.get_name_from_cls(), GetCurrentConfigTool.get_name_from_cls()]
)
)
tool_inclusion_definitions.append(project.project_config)
return tool_inclusion_definitions
@@ -289,7 +277,7 @@ class SerenaAgent:
If a client should attempt to use a tool that is dynamically disabled
(e.g. because a project is activated that disables it), it will receive an error.
"""
return list(self._exposed_tools.values())
return list(self._exposed_tools.tools)
def get_active_project(self) -> Project | None:
"""
@@ -323,12 +311,27 @@ class SerenaAgent:
"""
return list(self._modes)
def _format_prompt(self, prompt_template: str) -> str:
template = JinjaTemplate(prompt_template)
return template.render(available_tools=self._exposed_tools.tool_names, available_markers=self._exposed_tools.tool_marker_names)
def create_system_prompt(self) -> str:
return self.prompt_factory.create_system_prompt(
context_system_prompt=self._context.prompt,
mode_system_prompts=[mode.prompt for mode in self._modes],
available_markers = self._exposed_tools.tool_marker_names
log.info("Generating system prompt with available_tools=(see exposed tools), available_markers=%s", available_markers)
system_prompt = self.prompt_factory.create_system_prompt(
context_system_prompt=self._format_prompt(self._context.prompt),
mode_system_prompts=[self._format_prompt(mode.prompt) for mode in self._modes],
available_tools=self._exposed_tools.tool_names,
available_markers=available_markers,
)
# If a project is active at startup, append its activation message
if self._active_project is not None:
system_prompt += "\n\n" + self._active_project.get_activation_message()
log.info("System prompt:\n%s", system_prompt)
return system_prompt
def _update_active_tools(self) -> None:
"""
Update the active tools based on enabled modes and the active project.
@@ -391,10 +394,6 @@ class SerenaAgent:
self._active_project = project
self._update_active_tools()
# initialize project-specific instances which do not depend on the language server
self.memories_manager = MemoriesManager(project.project_root)
self.lines_read = LinesRead()
def init_language_server() -> None:
# start the language server
with LogTime("Language server initialization", logger=log):
@@ -538,6 +537,7 @@ class SerenaAgent:
log_level=self.serena_config.log_level,
ls_timeout=ls_timeout,
trace_lsp_communication=self.serena_config.trace_lsp_communication,
ls_specific_settings=self.serena_config.ls_specific_settings,
)
log.info(f"Starting the language server for {self._active_project.project_name}")
self.language_server.start()
@@ -552,10 +552,6 @@ class SerenaAgent:
def print_tool_overview(self) -> None:
ToolRegistry().print_tool_overview(self._active_tools.values())
def mark_file_modified(self, relativ_path: str) -> None:
assert self.lines_read is not None
self.lines_read.invalidate_lines_read(relativ_path)
def __del__(self) -> None:
"""
Destructor to clean up the language server instance and GUI logger
@@ -571,3 +567,7 @@ class SerenaAgent:
if self._gui_log_viewer:
log.info("Stopping the GUI log window ...")
self._gui_log_viewer.stop()
def get_tool_by_name(self, tool_name: str) -> Tool:
tool_class = ToolRegistry().get_tool_class_by_name(tool_name)
return self.get_tool(tool_class)
src/serena/cli.py
+364 -24
View File
@@ -1,3 +1,5 @@
import glob
import json
import os
import shutil
import subprocess
@@ -8,6 +10,7 @@ from typing import Any, Literal
import click
from sensai.util import logging
from sensai.util.logging import FileLoggerContext, datetime_tag
from tqdm import tqdm
from serena.agent import SerenaAgent
@@ -16,6 +19,8 @@ from serena.config.serena_config import ProjectConfig, SerenaConfig, SerenaPaths
from serena.constants import (
DEFAULT_CONTEXT,
DEFAULT_MODES,
PROMPT_TEMPLATES_DIR_IN_USER_HOME,
PROMPT_TEMPLATES_DIR_INTERNAL,
SERENA_LOG_FORMAT,
SERENA_MANAGED_DIR_IN_HOME,
SERENAS_OWN_CONTEXT_YAMLS_DIR,
@@ -23,11 +28,12 @@ from serena.constants import (
USER_CONTEXT_YAMLS_DIR,
USER_MODE_YAMLS_DIR,
)
from serena.mcp import SerenaMCPFactorySingleProcess
from serena.mcp import SerenaMCPFactory, SerenaMCPFactorySingleProcess
from serena.project import Project
from serena.tools import ToolRegistry
from serena.tools import FindReferencingSymbolsTool, FindSymbolTool, GetSymbolsOverviewTool, SearchForPatternTool, ToolRegistry
from serena.util.logging import MemoryLogHandler
from solidlsp.ls_config import Language
from solidlsp.util.subprocess_util import subprocess_kwargs
log = logging.getLogger(__name__)
@@ -37,18 +43,19 @@ log = logging.getLogger(__name__)
def _open_in_editor(path: str) -> None:
"""Open the given file in the system's default editor or viewer."""
editor = os.environ.get("EDITOR")
run_kwargs = subprocess_kwargs()
try:
if editor:
subprocess.run([editor, path], check=False)
subprocess.run([editor, path], check=False, **run_kwargs)
elif sys.platform.startswith("win"):
try:
os.startfile(path)
except OSError:
subprocess.run(["notepad.exe", path], check=False)
subprocess.run(["notepad.exe", path], check=False, **run_kwargs)
elif sys.platform == "darwin":
subprocess.run(["open", path], check=False)
subprocess.run(["open", path], check=False, **run_kwargs)
else:
subprocess.run(["xdg-open", path], check=False)
subprocess.run(["xdg-open", path], check=False, **run_kwargs)
except Exception as e:
print(f"Failed to open {path}: {e}")
@@ -112,7 +119,13 @@ class TopLevelCommands(AutoRegisteringGroup):
show_default=True,
help="Built-in mode names or paths to custom mode YAMLs.",
)
@click.option("--transport", type=click.Choice(["stdio", "sse"]), default="stdio", show_default=True, help="Transport protocol.")
@click.option(
"--transport",
type=click.Choice(["stdio", "sse", "streamable-http"]),
default="stdio",
show_default=True,
help="Transport protocol.",
)
@click.option("--host", type=str, default="0.0.0.0", show_default=True)
@click.option("--port", type=int, default=8000, show_default=True)
@click.option("--enable-web-dashboard", type=bool, is_flag=False, default=None, help="Override dashboard setting in config.")
@@ -130,7 +143,7 @@ class TopLevelCommands(AutoRegisteringGroup):
project_file_arg: str | None,
context: str,
modes: tuple[str, ...],
transport: Literal["stdio", "sse"],
transport: Literal["stdio", "sse", "streamable-http"],
host: str,
port: int,
enable_web_dashboard: bool | None,
@@ -427,32 +440,240 @@ class ProjectCommands(AutoRegisteringGroup):
default="WARNING",
help="Log level for indexing.",
)
def index(project: str, log_level: str = "WARNING") -> None:
ProjectCommands._index_project(project, log_level)
@click.option("--timeout", type=float, default=10, help="Timeout for indexing a single file.")
def index(project: str, log_level: str, timeout: float) -> None:
ProjectCommands._index_project(project, log_level, timeout=timeout)
@staticmethod
@click.command("index-deprecated", help="Deprecated alias for 'serena project index'.")
@click.argument("project", type=click.Path(exists=True), default=os.getcwd(), required=False)
@click.option("--log-level", type=click.Choice(["DEBUG", "INFO", "WARNING", "ERROR", "CRITICAL"]), default="WARNING")
def index_deprecated(project: str, log_level: str = "WARNING") -> None:
click.echo("Deprecated! Use `project index` instead.")
ProjectCommands._index_project(project, log_level)
@click.option("--timeout", type=float, default=10, help="Timeout for indexing a single file.")
def index_deprecated(project: str, log_level: str, timeout: float) -> None:
click.echo("Deprecated! Use `serena project index` instead.")
ProjectCommands._index_project(project, log_level, timeout=timeout)
@staticmethod
def _index_project(project: str, log_level: str) -> None:
def _index_project(project: str, log_level: str, timeout: float) -> None:
lvl = logging.getLevelNamesMapping()[log_level.upper()]
logging.configure(level=lvl)
serena_config = SerenaConfig.from_config_file()
proj = Project.load(os.path.abspath(project))
print(f"Indexing symbols in project {project}…")
ls = proj.create_language_server(log_level=lvl)
click.echo(f"Indexing symbols in project {project}…")
ls = proj.create_language_server(log_level=lvl, ls_timeout=timeout, ls_specific_settings=serena_config.ls_specific_settings)
log_file = os.path.join(project, ".serena", "logs", "indexing.txt")
collected_exceptions: list[Exception] = []
files_failed = []
with ls.start_server():
files = proj.gather_source_files()
for i, f in enumerate(tqdm(files, desc="Indexing")):
ls.request_document_symbols(f, include_body=False)
ls.request_document_symbols(f, include_body=True)
try:
ls.request_document_symbols(f, include_body=False)
ls.request_document_symbols(f, include_body=True)
except Exception as e:
log.error(f"Failed to index {f}, continuing.")
collected_exceptions.append(e)
files_failed.append(f)
if (i + 1) % 10 == 0:
ls.save_cache()
ls.save_cache()
print(f"Symbols saved to {ls.cache_path}")
click.echo(f"Symbols saved to {ls.cache_path}")
if len(files_failed) > 0:
os.makedirs(os.path.dirname(log_file), exist_ok=True)
with open(log_file, "w") as f:
for file, exception in zip(files_failed, collected_exceptions, strict=True):
f.write(f"{file}\n")
f.write(f"{exception}\n")
click.echo(f"Failed to index {len(files_failed)} files, see:\n{log_file}")
@staticmethod
@click.command("is_ignored_path", help="Check if a path is ignored by the project configuration.")
@click.argument("path", type=click.Path(exists=False, file_okay=True, dir_okay=True))
@click.argument("project", type=click.Path(exists=True, file_okay=False, dir_okay=True), default=os.getcwd())
def is_ignored_path(path: str, project: str) -> None:
"""
Check if a given path is ignored by the project configuration.
:param path: The path to check.
:param project: The path to the project directory, defaults to the current working directory.
"""
proj = Project.load(os.path.abspath(project))
if os.path.isabs(path):
path = os.path.relpath(path, start=proj.project_root)
is_ignored = proj.is_ignored_path(path)
click.echo(f"Path '{path}' IS {'ignored' if is_ignored else 'IS NOT ignored'} by the project configuration.")
@staticmethod
@click.command("index-file", help="Index a single file by saving its symbols to the LSP cache.")
@click.argument("file", type=click.Path(exists=True, file_okay=True, dir_okay=False))
@click.argument("project", type=click.Path(exists=True, file_okay=False, dir_okay=True), default=os.getcwd())
@click.option("--verbose", "-v", is_flag=True, help="Print detailed information about the indexed symbols.")
def index_file(file: str, project: str, verbose: bool) -> None:
"""
Index a single file by saving its symbols to the LSP cache, useful for debugging.
:param file: path to the file to index, must be inside the project directory.
:param project: path to the project directory, defaults to the current working directory.
:param verbose: if set, prints detailed information about the indexed symbols.
"""
proj = Project.load(os.path.abspath(project))
if os.path.isabs(file):
file = os.path.relpath(file, start=proj.project_root)
if proj.is_ignored_path(file, ignore_non_source_files=True):
click.echo(f"'{file}' is ignored or declared as non-code file by the project configuration, won't index.")
exit(1)
ls = proj.create_language_server()
with ls.start_server():
symbols, _ = ls.request_document_symbols(file, include_body=False)
ls.request_document_symbols(file, include_body=True)
if verbose:
click.echo(f"Symbols in file '{file}':")
for symbol in symbols:
click.echo(f" - {symbol['name']} at line {symbol['selectionRange']['start']['line']} of kind {symbol['kind']}")
ls.save_cache()
click.echo(f"Successfully indexed file '{file}', {len(symbols)} symbols saved to {ls.cache_path}.")
@staticmethod
@click.command("health-check", help="Perform a comprehensive health check of the project's tools and language server.")
@click.argument("project", type=click.Path(exists=True, file_okay=False, dir_okay=True), default=os.getcwd())
def health_check(project: str) -> None:
"""
Perform a comprehensive health check of the project's tools and language server.
:param project: path to the project directory, defaults to the current working directory.
"""
# NOTE: completely written by Claude Code, only functionality was reviewed, not implementation
logging.configure(level=logging.INFO)
project_path = os.path.abspath(project)
proj = Project.load(project_path)
# Create log file with timestamp
timestamp = datetime_tag()
log_dir = os.path.join(project_path, ".serena", "logs", "health-checks")
os.makedirs(log_dir, exist_ok=True)
log_file = os.path.join(log_dir, f"health_check_{timestamp}.log")
with FileLoggerContext(log_file, append=False, enabled=True):
log.info("Starting health check for project: %s", project_path)
try:
# Create SerenaAgent with dashboard disabled
log.info("Creating SerenaAgent with disabled dashboard...")
config = SerenaConfig(gui_log_window_enabled=False, web_dashboard=False)
agent = SerenaAgent(project=project_path, serena_config=config)
log.info("SerenaAgent created successfully")
# Find first non-empty file that can be analyzed
log.info("Searching for analyzable files...")
files = proj.gather_source_files()
target_file = None
for file_path in files:
try:
full_path = os.path.join(project_path, file_path)
if os.path.getsize(full_path) > 0:
target_file = file_path
log.info("Found analyzable file: %s", target_file)
break
except (OSError, FileNotFoundError):
continue
if not target_file:
log.error("No analyzable files found in project")
click.echo("❌ Health check failed: No analyzable files found")
click.echo(f"Log saved to: {log_file}")
return
# Get tools from agent
overview_tool = agent.get_tool(GetSymbolsOverviewTool)
find_symbol_tool = agent.get_tool(FindSymbolTool)
find_refs_tool = agent.get_tool(FindReferencingSymbolsTool)
search_pattern_tool = agent.get_tool(SearchForPatternTool)
# Test 1: Get symbols overview
log.info("Testing GetSymbolsOverviewTool on file: %s", target_file)
overview_result = agent.execute_task(lambda: overview_tool.apply(target_file))
overview_data = json.loads(overview_result)
log.info("GetSymbolsOverviewTool returned %d symbols", len(overview_data))
if not overview_data:
log.error("No symbols found in file %s", target_file)
click.echo("❌ Health check failed: No symbols found in target file")
click.echo(f"Log saved to: {log_file}")
return
# Extract suitable symbol (prefer class or function over variables)
# LSP symbol kinds: 5=class, 12=function, 6=method, 9=constructor
preferred_kinds = [5, 12, 6, 9] # class, function, method, constructor
selected_symbol = None
for symbol in overview_data:
if symbol.get("kind") in preferred_kinds:
selected_symbol = symbol
break
# If no preferred symbol found, use first available
if not selected_symbol:
selected_symbol = overview_data[0]
log.info("No class or function found, using first available symbol")
symbol_name = selected_symbol.get("name_path", "unknown")
symbol_kind = selected_symbol.get("kind", "unknown")
log.info("Using symbol for testing: %s (kind: %d)", symbol_name, symbol_kind)
# Test 2: FindSymbolTool
log.info("Testing FindSymbolTool for symbol: %s", symbol_name)
find_symbol_result = agent.execute_task(
lambda: find_symbol_tool.apply(symbol_name, relative_path=target_file, include_body=True)
)
find_symbol_data = json.loads(find_symbol_result)
log.info("FindSymbolTool found %d matches for symbol %s", len(find_symbol_data), symbol_name)
# Test 3: FindReferencingSymbolsTool
log.info("Testing FindReferencingSymbolsTool for symbol: %s", symbol_name)
try:
find_refs_result = agent.execute_task(lambda: find_refs_tool.apply(symbol_name, relative_path=target_file))
find_refs_data = json.loads(find_refs_result)
log.info("FindReferencingSymbolsTool found %d references for symbol %s", len(find_refs_data), symbol_name)
except Exception as e:
log.warning("FindReferencingSymbolsTool failed for symbol %s: %s", symbol_name, str(e))
find_refs_data = []
# Test 4: SearchForPatternTool to verify references
log.info("Testing SearchForPatternTool for pattern: %s", symbol_name)
try:
search_result = agent.execute_task(
lambda: search_pattern_tool.apply(substring_pattern=symbol_name, restrict_search_to_code_files=True)
)
search_data = json.loads(search_result)
pattern_matches = sum(len(matches) for matches in search_data.values())
log.info("SearchForPatternTool found %d pattern matches for %s", pattern_matches, symbol_name)
except Exception as e:
log.warning("SearchForPatternTool failed for pattern %s: %s", symbol_name, str(e))
pattern_matches = 0
# Verify tools worked as expected
tools_working = True
if not find_symbol_data:
log.error("FindSymbolTool returned no results")
tools_working = False
if len(find_refs_data) == 0 and pattern_matches == 0:
log.warning("Both FindReferencingSymbolsTool and SearchForPatternTool found no matches - this might indicate an issue")
log.info("Health check completed successfully")
if tools_working:
click.echo("✅ Health check passed - All tools working correctly")
else:
click.echo("⚠️ Health check completed with warnings - Check log for details")
except Exception as e:
log.exception("Health check failed with exception: %s", str(e))
click.echo(f"❌ Health check failed: {e!s}")
finally:
click.echo(f"Log saved to: {log_file}")
class ToolCommands(AutoRegisteringGroup):
@@ -465,15 +686,133 @@ class ToolCommands(AutoRegisteringGroup):
)
@staticmethod
@click.command("list", help="Prints an overview of all tools implemented in Serena (not just the active ones for your project).")
@click.command(
"list",
help="Prints an overview of the tools that are active by default (not just the active ones for your project). For viewing all tools, pass `--all / -a`",
)
@click.option("--quiet", "-q", is_flag=True)
def list(quiet: bool = False) -> None:
@click.option("--all", "-a", "include_optional", is_flag=True, help="List all tools, including those not enabled by default.")
@click.option("--only-optional", is_flag=True, help="List only optional tools (those not enabled by default).")
def list(quiet: bool = False, include_optional: bool = False, only_optional: bool = False) -> None:
tool_registry = ToolRegistry()
if quiet:
for tool_name in tool_registry.get_tool_names_default_enabled():
if only_optional:
tool_names = tool_registry.get_tool_names_optional()
elif include_optional:
tool_names = tool_registry.get_tool_names()
else:
tool_names = tool_registry.get_tool_names_default_enabled()
for tool_name in tool_names:
click.echo(tool_name)
else:
ToolRegistry().print_tool_overview()
ToolRegistry().print_tool_overview(include_optional=include_optional, only_optional=only_optional)
@staticmethod
@click.command(
"description",
help="Print the description of a tool, optionally with a specific context (the latter may modify the default description).",
)
@click.argument("tool_name", type=str)
@click.option("--context", type=str, default=None, help="Context name or path to context file.")
def description(tool_name: str, context: str | None = None) -> None:
# Load the context
serena_context = None
if context:
serena_context = SerenaAgentContext.load(context)
agent = SerenaAgent(
project=None,
serena_config=SerenaConfig(web_dashboard=False, log_level=logging.INFO),
context=serena_context,
)
tool = agent.get_tool_by_name(tool_name)
mcp_tool = SerenaMCPFactory.make_mcp_tool(tool)
click.echo(mcp_tool.description)
class PromptCommands(AutoRegisteringGroup):
def __init__(self) -> None:
super().__init__(name="prompts", help="Commands related to Serena's prompts that are outside of contexts and modes.")
@staticmethod
def _get_user_prompt_yaml_path(prompt_yaml_name: str) -> str:
os.makedirs(PROMPT_TEMPLATES_DIR_IN_USER_HOME, exist_ok=True)
return os.path.join(PROMPT_TEMPLATES_DIR_IN_USER_HOME, prompt_yaml_name)
@staticmethod
@click.command("list", help="Lists yamls that are used for defining prompts.")
def list() -> None:
serena_prompt_yaml_names = [os.path.basename(f) for f in glob.glob(PROMPT_TEMPLATES_DIR_INTERNAL + "/*.yml")]
for prompt_yaml_name in serena_prompt_yaml_names:
user_prompt_yaml_path = PromptCommands._get_user_prompt_yaml_path(prompt_yaml_name)
if os.path.exists(user_prompt_yaml_path):
click.echo(f"{user_prompt_yaml_path} merged with default prompts in {prompt_yaml_name}")
else:
click.echo(prompt_yaml_name)
@staticmethod
@click.command("create-override", help="Create an override of an internal prompts yaml for customizing Serena's prompts")
@click.argument("prompt_yaml_name")
def create_override(prompt_yaml_name: str) -> None:
"""
:param prompt_yaml_name: The yaml name of the prompt you want to override. Call the `list` command for discovering valid prompt yaml names.
:return:
"""
# for convenience, we can pass names without .yml
if not prompt_yaml_name.endswith(".yml"):
prompt_yaml_name = prompt_yaml_name + ".yml"
user_prompt_yaml_path = PromptCommands._get_user_prompt_yaml_path(prompt_yaml_name)
if os.path.exists(user_prompt_yaml_path):
raise FileExistsError(f"{user_prompt_yaml_path} already exists.")
serena_prompt_yaml_path = os.path.join(PROMPT_TEMPLATES_DIR_INTERNAL, prompt_yaml_name)
shutil.copyfile(serena_prompt_yaml_path, user_prompt_yaml_path)
_open_in_editor(user_prompt_yaml_path)
@staticmethod
@click.command("edit-override", help="Edit an existing prompt override file")
@click.argument("prompt_yaml_name")
def edit_override(prompt_yaml_name: str) -> None:
"""
:param prompt_yaml_name: The yaml name of the prompt override to edit.
:return:
"""
# for convenience, we can pass names without .yml
if not prompt_yaml_name.endswith(".yml"):
prompt_yaml_name = prompt_yaml_name + ".yml"
user_prompt_yaml_path = PromptCommands._get_user_prompt_yaml_path(prompt_yaml_name)
if not os.path.exists(user_prompt_yaml_path):
click.echo(f"Override file '{prompt_yaml_name}' not found. Create it with: prompts create-override {prompt_yaml_name}")
return
_open_in_editor(user_prompt_yaml_path)
@staticmethod
@click.command("list-overrides", help="List existing prompt override files")
def list_overrides() -> None:
os.makedirs(PROMPT_TEMPLATES_DIR_IN_USER_HOME, exist_ok=True)
serena_prompt_yaml_names = [os.path.basename(f) for f in glob.glob(PROMPT_TEMPLATES_DIR_INTERNAL + "/*.yml")]
override_files = glob.glob(os.path.join(PROMPT_TEMPLATES_DIR_IN_USER_HOME, "*.yml"))
for file_path in override_files:
if os.path.basename(file_path) in serena_prompt_yaml_names:
click.echo(file_path)
@staticmethod
@click.command("delete-override", help="Delete a prompt override file")
@click.argument("prompt_yaml_name")
def delete_override(prompt_yaml_name: str) -> None:
"""
:param prompt_yaml_name: The yaml name of the prompt override to delete."
:return:
"""
# for convenience, we can pass names without .yml
if not prompt_yaml_name.endswith(".yml"):
prompt_yaml_name = prompt_yaml_name + ".yml"
user_prompt_yaml_path = PromptCommands._get_user_prompt_yaml_path(prompt_yaml_name)
if not os.path.exists(user_prompt_yaml_path):
click.echo(f"Override file '{prompt_yaml_name}' not found.")
return
os.remove(user_prompt_yaml_path)
click.echo(f"Deleted override file '{prompt_yaml_name}'.")
# Expose groups so we can reference them in pyproject.toml
@@ -482,6 +821,7 @@ context = ContextCommands()
project = ProjectCommands()
config = SerenaConfigCommands()
tools = ToolCommands()
prompts = PromptCommands()
# Expose toplevel commands for the same reason
top_level = TopLevelCommands()
@@ -489,7 +829,7 @@ start_mcp_server = top_level.start_mcp_server
index_project = project.index_deprecated
# needed for the help script to work - register all subcommands to the top-level group
for subgroup in (mode, context, project, config, tools):
for subgroup in (mode, context, project, config, tools, prompts):
top_level.add_command(subgroup)
src/serena/code_editor.py
+80 -7
View File
@@ -4,13 +4,15 @@ import os
from abc import ABC, abstractmethod
from collections.abc import Iterable, Iterator, Reversible
from contextlib import contextmanager
from typing import TYPE_CHECKING, Generic, Optional, TypeVar
from typing import TYPE_CHECKING, Generic, Optional, TypeVar, cast
from serena.symbol import JetBrainsSymbol, LanguageServerSymbol, LanguageServerSymbolRetriever, PositionInFile, Symbol
from solidlsp import SolidLanguageServer
from solidlsp import SolidLanguageServer, ls_types
from solidlsp.ls import LSPFileBuffer
from solidlsp.ls_utils import TextUtils
from solidlsp.ls_types import extract_text_edits
from solidlsp.ls_utils import PathUtils, TextUtils
from .constants import DEFAULT_SOURCE_FILE_ENCODING
from .project import Project
from .tools.jetbrains_plugin_client import JetBrainsPluginClient
@@ -27,6 +29,14 @@ class CodeEditor(Generic[TSymbol], ABC):
self.project_root = project_root
self.agent = agent
# set encoding based on active project, if available
encoding = DEFAULT_SOURCE_FILE_ENCODING
if agent is not None:
project = agent.get_active_project()
if project is not None:
encoding = project.project_config.encoding
self.encoding = encoding
class EditedFile(ABC):
@abstractmethod
def get_contents(self) -> str:
@@ -58,11 +68,8 @@ class CodeEditor(Generic[TSymbol], ABC):
yield edited_file
# save the file
abs_path = os.path.join(self.project_root, relative_path)
with open(abs_path, "w", encoding="utf-8") as f:
with open(abs_path, "w", encoding=self.encoding) as f:
f.write(edited_file.get_contents())
# notify agent (if provided)
if self.agent is not None:
self.agent.mark_file_modified(relative_path)
@abstractmethod
def _find_unique_symbol(self, name_path: str, relative_file_path: str) -> TSymbol:
@@ -210,6 +217,17 @@ class CodeEditor(Generic[TSymbol], ABC):
with self._edited_file_context(relative_file_path) as edited_file:
edited_file.delete_text_between_positions(start_pos, end_pos)
@abstractmethod
def rename_symbol(self, name_path: str, relative_file_path: str, new_name: str) -> str:
"""
Renames the symbol with the given name throughout the codebase.
:param name_path: the name path of the symbol to rename
:param relative_file_path: the relative path of the file containing the symbol
:param new_name: the new name for the symbol
:return: a status message
"""
class LanguageServerCodeEditor(CodeEditor[LanguageServerSymbol]):
def __init__(self, symbol_retriever: LanguageServerSymbolRetriever, agent: Optional["SerenaAgent"] = None):
@@ -235,6 +253,9 @@ class LanguageServerCodeEditor(CodeEditor[LanguageServerSymbol]):
def insert_text_at_position(self, pos: PositionInFile, text: str) -> None:
self._lang_server.insert_text_at_position(self._relative_path, pos.line, pos.col, text)
def apply_text_edits(self, text_edits: list[ls_types.TextEdit]) -> None:
return self._lang_server.apply_text_edits_to_file(self._relative_path, text_edits)
@contextmanager
def _open_file_context(self, relative_path: str) -> Iterator["CodeEditor.EditedFile"]:
with self._lang_server.open_file(relative_path) as file_buffer:
@@ -255,6 +276,47 @@ class LanguageServerCodeEditor(CodeEditor[LanguageServerSymbol]):
)
return symbol_candidates[0]
def _apply_workspace_edit(self, workspace_edit: ls_types.WorkspaceEdit) -> list[str]:
"""
Apply a WorkspaceEdit by making the changes to files.
:param workspace_edit: The WorkspaceEdit containing the changes to apply
:return: List of relative file paths that were modified
"""
uri_to_edits = extract_text_edits(workspace_edit)
modified_relative_paths = []
# Handle the 'changes' format (URI -> list of TextEdits)
for uri, edits in uri_to_edits.items():
file_path = PathUtils.uri_to_path(uri)
relative_path = os.path.relpath(file_path, self._lang_server.repository_root_path)
modified_relative_paths.append(relative_path)
with self._edited_file_context(relative_path) as edited_file:
edited_file = cast(LanguageServerCodeEditor.EditedFile, edited_file)
edited_file.apply_text_edits(edits)
return modified_relative_paths
def rename_symbol(self, name_path: str, relative_file_path: str, new_name: str) -> str:
symbol = self._find_unique_symbol(name_path, relative_file_path)
if not symbol.location.has_position_in_file():
raise ValueError(f"Symbol '{name_path}' does not have a valid position in file for renaming")
# After has_position_in_file check, line and column are guaranteed to be non-None
assert symbol.location.line is not None
assert symbol.location.column is not None
rename_result = self._lang_server.request_rename_symbol_edit(
relative_file_path=relative_file_path, line=symbol.location.line, column=symbol.location.column, new_name=new_name
)
if rename_result is None:
raise ValueError(
f"Language server for {self._lang_server.language_id} returned no rename edits for symbol '{name_path}'. "
f"The symbol might not support renaming."
)
modified_files = self._apply_workspace_edit(rename_result)
msg = f"Successfully renamed '{name_path}' to '{new_name}' in {len(modified_files)} file(s)"
return msg
class JetBrainsCodeEditor(CodeEditor[JetBrainsSymbol]):
def __init__(self, project: Project, agent: Optional["SerenaAgent"] = None) -> None:
@@ -295,3 +357,14 @@ class JetBrainsCodeEditor(CodeEditor[JetBrainsSymbol]):
"Their locations are: \n " + json.dumps([s["location"] for s in symbols], indent=2)
)
return JetBrainsSymbol(symbols[0], self._project)
def rename_symbol(self, name_path: str, relative_file_path: str, new_name: str) -> str:
with JetBrainsPluginClient.from_project(self._project) as client:
client.rename_symbol(
name_path=name_path,
relative_path=relative_file_path,
new_name=new_name,
rename_in_comments=False,
rename_in_text_occurrences=False,
)
return "Success"
src/serena/config/context_mode.py
+17 -26
View File
@@ -3,8 +3,7 @@ Context and Mode configuration loader
"""
import os
from copy import copy
from dataclasses import asdict, dataclass
from dataclasses import dataclass, field
from enum import Enum
from pathlib import Path
from typing import TYPE_CHECKING, Self
@@ -18,6 +17,7 @@ from serena.constants import (
DEFAULT_CONTEXT,
DEFAULT_MODES,
INTERNAL_MODE_YAMLS_DIR,
SERENA_FILE_ENCODING,
SERENAS_OWN_CONTEXT_YAMLS_DIR,
SERENAS_OWN_MODE_YAMLS_DIR,
USER_CONTEXT_YAMLS_DIR,
@@ -39,22 +39,15 @@ class SerenaAgentMode(ToolInclusionDefinition, ToStringMixin):
name: str
prompt: str
"""
a Jinja2 template for the generation of the system prompt.
It is formatted by the agent (see SerenaAgent._format_prompt()).
"""
description: str = ""
def _tostring_includes(self) -> list[str]:
return ["name"]
def to_json_dict(self) -> dict[str, str | list[str]]:
result = asdict(self)
result["excluded_tools"] = list(result["excluded_tools"])
return result
@classmethod
def from_json_dict(cls, data: dict) -> Self:
data = copy(data)
data["excluded_tools"] = set(data["excluded_tools"])
return cls(**data)
def print_overview(self) -> None:
"""Print an overview of the mode."""
print(f"{self.name}:\n {self.description}")
@@ -64,7 +57,7 @@ class SerenaAgentMode(ToolInclusionDefinition, ToStringMixin):
@classmethod
def from_yaml(cls, yaml_path: str | Path) -> Self:
"""Load a mode from a YAML file."""
with open(yaml_path, encoding="utf-8") as f:
with open(yaml_path, encoding=SERENA_FILE_ENCODING) as f:
data = yaml.safe_load(f)
name = data.pop("name", Path(yaml_path).stem)
return cls(name=name, **data)
@@ -133,28 +126,26 @@ class SerenaAgentContext(ToolInclusionDefinition, ToStringMixin):
name: str
prompt: str
"""
a Jinja2 template for the generation of the system prompt.
It is formatted by the agent (see SerenaAgent._format_prompt()).
"""
description: str = ""
tool_description_overrides: dict[str, str] = field(default_factory=dict)
"""Maps tool names to custom descriptions, default descriptions are extracted from the tool docstrings."""
def _tostring_includes(self) -> list[str]:
return ["name"]
def to_json_dict(self) -> dict[str, str | list[str]]:
result = asdict(self)
result["excluded_tools"] = list(result["excluded_tools"])
return result
@classmethod
def from_json_dict(cls, data: dict) -> Self:
data = copy(data)
data["excluded_tools"] = set(data["excluded_tools"])
return cls(**data)
@classmethod
def from_yaml(cls, yaml_path: str | Path) -> Self:
"""Load a context from a YAML file."""
with open(yaml_path, encoding="utf-8") as f:
with open(yaml_path, encoding=SERENA_FILE_ENCODING) as f:
data = yaml.safe_load(f)
name = data.pop("name", Path(yaml_path).stem)
# Ensure backwards compatibility for tool_description_overrides
if "tool_description_overrides" not in data:
data["tool_description_overrides"] = {}
return cls(name=name, **data)
@classmethod
src/serena/config/serena_config.py
+21 -11
View File
@@ -19,10 +19,11 @@ from sensai.util.logging import LogTime, datetime_tag
from sensai.util.string import ToStringMixin
from serena.constants import (
DEFAULT_ENCODING,
DEFAULT_SOURCE_FILE_ENCODING,
PROJECT_TEMPLATE_FILE,
REPO_ROOT,
SELENA_CONFIG_TEMPLATE_FILE,
SERENA_CONFIG_TEMPLATE_FILE,
SERENA_FILE_ENCODING,
SERENA_MANAGED_DIR_IN_HOME,
SERENA_MANAGED_DIR_NAME,
)
@@ -99,7 +100,7 @@ class ToolSet:
for excluded_tool in definition.excluded_tools:
if not registry.is_valid_tool_name(excluded_tool):
raise ValueError(f"Invalid tool name '{excluded_tool}' provided for exclusion")
if excluded_tool in self._tool_names:
if excluded_tool in tool_names:
tool_names.remove(excluded_tool)
excluded_tools.append(excluded_tool)
if included_tools:
@@ -166,7 +167,7 @@ class ProjectConfig(ToolInclusionDefinition, ToStringMixin):
read_only: bool = False
ignore_all_files_in_gitignore: bool = True
initial_prompt: str = ""
encoding: str = DEFAULT_ENCODING
encoding: str = DEFAULT_SOURCE_FILE_ENCODING
SERENA_DEFAULT_PROJECT_FILE = "project.yml"
@@ -198,12 +199,12 @@ class ProjectConfig(ToolInclusionDefinition, ToStringMixin):
raise ValueError(
f"No source files found in {project_root}\n\n"
f"To use Serena with this project, you need to either:\n"
f"1. Add source files in one of the supported languages (Python, JavaScript/TypeScript, Java, C#, Rust, Go, Ruby, C++, PHP)\n"
f"1. Add source files in one of the supported languages (Python, JavaScript/TypeScript, Java, C#, Rust, Go, Ruby, C++, PHP, Swift, Elixir, Terraform, Bash)\n"
f"2. Create a project configuration file manually at:\n"
f" {os.path.join(project_root, cls.rel_path_to_project_yml())}\n\n"
f"Example project.yml:\n"
f" project_name: {project_name}\n"
f" language: python # or typescript, java, csharp, rust, go, ruby, cpp, php\n"
f" language: python # or typescript, java, csharp, rust, go, ruby, cpp, php, swift, elixir, terraform, bash\n"
)
# find the language with the highest percentage
dominant_language = max(language_composition.keys(), key=lambda lang: language_composition[lang])
@@ -244,7 +245,7 @@ class ProjectConfig(ToolInclusionDefinition, ToStringMixin):
read_only=data.get("read_only", False),
ignore_all_files_in_gitignore=data.get("ignore_all_files_in_gitignore", True),
initial_prompt=data.get("initial_prompt", ""),
encoding=data.get("encoding", DEFAULT_ENCODING),
encoding=data.get("encoding", DEFAULT_SOURCE_FILE_ENCODING),
)
@classmethod
@@ -259,7 +260,7 @@ class ProjectConfig(ToolInclusionDefinition, ToStringMixin):
return cls.autogenerate(project_root)
else:
raise FileNotFoundError(f"Project configuration file not found: {yaml_path}")
with open(yaml_path, encoding="utf-8") as f:
with open(yaml_path, encoding=SERENA_FILE_ENCODING) as f:
yaml_data = yaml.safe_load(f)
if "project_name" not in yaml_data:
yaml_data["project_name"] = project_root.name
@@ -350,6 +351,13 @@ class SerenaConfig(ToolInclusionDefinition, ToStringMixin):
on the first run, which can take some time and require internet access. Others, like the Anthropic ones, may require an API key
and rate limits may apply.
"""
default_max_tool_answer_chars: int = 150_000
"""Used as default for tools where the apply method has a default maximal answer length.
Even though the value of the max_answer_chars can be changed when calling the tool, it may make sense to adjust this default
through the global configuration.
"""
ls_specific_settings: dict = field(default_factory=dict)
"""Advanced configuration option allowing to configure language server implementation specific options, see SolidLSPSettings for more info."""
CONFIG_FILE = "serena_config.yml"
CONFIG_FILE_DOCKER = "serena_config.docker.yml" # Docker-specific config file; auto-generated if missing, mounted via docker-compose for user customization
@@ -365,7 +373,7 @@ class SerenaConfig(ToolInclusionDefinition, ToStringMixin):
:param config_file_path: the path where the configuration file should be generated
"""
log.info(f"Auto-generating Serena configuration file in {config_file_path}")
loaded_commented_yaml = load_yaml(SELENA_CONFIG_TEMPLATE_FILE, preserve_comments=True)
loaded_commented_yaml = load_yaml(SERENA_CONFIG_TEMPLATE_FILE, preserve_comments=True)
save_yaml(config_file_path, loaded_commented_yaml, preserve_comments=True)
@classmethod
@@ -453,6 +461,8 @@ class SerenaConfig(ToolInclusionDefinition, ToStringMixin):
instance.token_count_estimator = loaded_commented_yaml.get(
"token_count_estimator", RegisteredTokenCountEstimator.TIKTOKEN_GPT4O.name
)
instance.default_max_tool_answer_chars = loaded_commented_yaml.get("default_max_tool_answer_chars", 150_000)
instance.ls_specific_settings = loaded_commented_yaml.get("ls_specific_settings", {})
# re-save the configuration file if any migrations were performed
if num_project_migrations > 0:
@@ -474,11 +484,11 @@ class SerenaConfig(ToolInclusionDefinition, ToStringMixin):
"""
log.info(f"Found legacy project configuration file {path}, migrating to in-project configuration.")
try:
with open(path, encoding="utf-8") as f:
with open(path, encoding=SERENA_FILE_ENCODING) as f:
project_config_data = yaml.safe_load(f)
if "project_name" not in project_config_data:
project_name = path.stem
with open(path, "a", encoding="utf-8") as f:
with open(path, "a", encoding=SERENA_FILE_ENCODING) as f:
f.write(f"\nproject_name: {project_name}")
project_root = project_config_data["project_root"]
shutil.move(str(path), str(Path(project_root) / ProjectConfig.rel_path_to_project_yml()))
src/serena/constants.py
+8 -3
View File
@@ -10,7 +10,8 @@ SERENA_MANAGED_DIR_IN_HOME = str(_serena_in_home_managed_dir)
# TODO: Path-related constants should be moved to SerenaPaths; don't add further constants here.
REPO_ROOT = str(_repo_root_path)
PROMPT_TEMPLATES_DIR = str(_serena_pkg_path / "resources" / "config" / "prompt_templates")
PROMPT_TEMPLATES_DIR_INTERNAL = str(_serena_pkg_path / "resources" / "config" / "prompt_templates")
PROMPT_TEMPLATES_DIR_IN_USER_HOME = str(_serena_in_home_managed_dir / "prompt_templates")
SERENAS_OWN_CONTEXT_YAMLS_DIR = str(_serena_pkg_path / "resources" / "config" / "contexts")
"""The contexts that are shipped with the Serena package, i.e. the default contexts."""
USER_CONTEXT_YAMLS_DIR = str(_serena_in_home_managed_dir / "contexts")
@@ -24,11 +25,15 @@ INTERNAL_MODE_YAMLS_DIR = str(_serena_pkg_path / "resources" / "config" / "inter
SERENA_DASHBOARD_DIR = str(_serena_pkg_path / "resources" / "dashboard")
SERENA_ICON_DIR = str(_serena_pkg_path / "resources" / "icons")
DEFAULT_ENCODING = "utf-8"
DEFAULT_SOURCE_FILE_ENCODING = "utf-8"
"""The default encoding assumed for project source files."""
DEFAULT_CONTEXT = "desktop-app"
DEFAULT_MODES = ("interactive", "editing")
SERENA_FILE_ENCODING = "utf-8"
"""The encoding used for Serena's own files, such as configuration files and memories."""
PROJECT_TEMPLATE_FILE = str(_serena_pkg_path / "resources" / "project.template.yml")
SELENA_CONFIG_TEMPLATE_FILE = str(_serena_pkg_path / "resources" / "serena_config.template.yml")
SERENA_CONFIG_TEMPLATE_FILE = str(_serena_pkg_path / "resources" / "serena_config.template.yml")
SERENA_LOG_FORMAT = "%(levelname)-5s %(asctime)-15s [%(threadName)s] %(name)s:%(funcName)s:%(lineno)d - %(message)s"
src/serena/dashboard.py
+10 -2
View File
@@ -2,7 +2,7 @@ import os
import socket
import threading
from collections.abc import Callable
from typing import Any
from typing import TYPE_CHECKING, Any
from flask import Flask, Response, request, send_from_directory
from pydantic import BaseModel
@@ -12,6 +12,9 @@ from serena.analytics import ToolUsageStats
from serena.constants import SERENA_DASHBOARD_DIR
from serena.util.logging import MemoryLogHandler
if TYPE_CHECKING:
from serena.agent import SerenaAgent
log = logging.getLogger(__name__)
# disable Werkzeug's logging to avoid cluttering the output
@@ -25,6 +28,7 @@ class RequestLog(BaseModel):
class ResponseLog(BaseModel):
messages: list[str]
max_idx: int
active_project: str | None = None
class ResponseToolNames(BaseModel):
@@ -42,11 +46,13 @@ class SerenaDashboardAPI:
self,
memory_log_handler: MemoryLogHandler,
tool_names: list[str],
agent: "SerenaAgent",
shutdown_callback: Callable[[], None] | None = None,
tool_usage_stats: ToolUsageStats | None = None,
) -> None:
self._memory_log_handler = memory_log_handler
self._tool_names = tool_names
self._agent = agent
self._shutdown_callback = shutdown_callback
self._app = Flask(__name__)
self._tool_usage_stats = tool_usage_stats
@@ -106,7 +112,9 @@ class SerenaDashboardAPI:
def _get_log_messages(self, request_log: RequestLog) -> ResponseLog:
all_messages = self._memory_log_handler.get_log_messages()
requested_messages = all_messages[request_log.start_idx :] if request_log.start_idx <= len(all_messages) else []
return ResponseLog(messages=requested_messages, max_idx=len(all_messages) - 1)
project = self._agent.get_active_project()
project_name = project.project_name if project else None
return ResponseLog(messages=requested_messages, max_idx=len(all_messages) - 1, active_project=project_name)
def _get_tool_names(self) -> ResponseToolNames:
return ResponseToolNames(tool_names=self._tool_names)
src/serena/generated/generated_prompt_factory.py
+3 -1
View File
@@ -32,5 +32,7 @@ class PromptFactory(PromptFactoryBase):
def create_prepare_for_new_conversation(self) -> str:
return self._render_prompt("prepare_for_new_conversation", locals())
def create_system_prompt(self, *, context_system_prompt: Any, mode_system_prompts: Any) -> str:
def create_system_prompt(
self, *, available_markers: Any, available_tools: Any, context_system_prompt: Any, mode_system_prompts: Any
) -> str:
return self._render_prompt("system_prompt", locals())
src/serena/mcp.py
+144 -10
View File
@@ -6,6 +6,7 @@ import sys
from abc import abstractmethod
from collections.abc import AsyncIterator, Iterator, Sequence
from contextlib import asynccontextmanager
from copy import deepcopy
from dataclasses import dataclass
from typing import Any, Literal, cast
@@ -37,7 +38,7 @@ def configure_logging(*args, **kwargs) -> None: # type: ignore
# patch the logging configuration function in fastmcp, because it's hard-coded and broken
server.configure_logging = configure_logging
server.configure_logging = configure_logging # type: ignore
@dataclass
@@ -57,21 +58,144 @@ class SerenaMCPFactory:
self.project = project
@staticmethod
def make_mcp_tool(tool: Tool) -> MCPTool:
def _sanitize_for_openai_tools(schema: dict) -> dict:
"""
This method was written by GPT-5, I have not reviewed it in detail.
Only called when `openai_tool_compatible` is True.
Make a Pydantic/JSON Schema object compatible with OpenAI tool schema.
- 'integer' -> 'number' (+ multipleOf: 1)
- remove 'null' from union type arrays
- coerce integer-only enums to number
- best-effort simplify oneOf/anyOf when they only differ by integer/number
"""
s = deepcopy(schema)
def walk(node): # type: ignore
if not isinstance(node, dict):
# lists get handled by parent calls
return node
# ---- handle type ----
t = node.get("type")
if isinstance(t, str):
if t == "integer":
node["type"] = "number"
# preserve existing multipleOf but ensure it's integer-like
if "multipleOf" not in node:
node["multipleOf"] = 1
elif isinstance(t, list):
# remove 'null' (OpenAI tools don't support nullables)
t2 = [x if x != "integer" else "number" for x in t if x != "null"]
if not t2:
# fall back to object if it somehow becomes empty
t2 = ["object"]
node["type"] = t2[0] if len(t2) == 1 else t2
if "integer" in t or "number" in t2:
# if integers were present, keep integer-like restriction
node.setdefault("multipleOf", 1)
# ---- enums of integers -> number ----
if "enum" in node and isinstance(node["enum"], list):
vals = node["enum"]
if vals and all(isinstance(v, int) for v in vals):
node.setdefault("type", "number")
# keep them as ints; JSON 'number' covers ints
node.setdefault("multipleOf", 1)
# ---- simplify anyOf/oneOf if they only differ by integer/number ----
for key in ("oneOf", "anyOf"):
if key in node and isinstance(node[key], list):
# Special case: anyOf or oneOf with "type X" and "null"
if len(node[key]) == 2:
types = [sub.get("type") for sub in node[key]]
if "null" in types:
non_null_type = next(t for t in types if t != "null")
if isinstance(non_null_type, str):
node["type"] = non_null_type
node.pop(key, None)
continue
simplified = []
changed = False
for sub in node[key]:
sub = walk(sub) # recurse
simplified.append(sub)
# If all subs are the same after integer→number, collapse
try:
import json
canon = [json.dumps(x, sort_keys=True) for x in simplified]
if len(set(canon)) == 1:
# copy the single schema up
only = simplified[0]
node.pop(key, None)
for k, v in only.items():
if k not in node:
node[k] = v
changed = True
except Exception:
pass
if not changed:
node[key] = simplified
# ---- recurse into known schema containers ----
for child_key in ("properties", "patternProperties", "definitions", "$defs"):
if child_key in node and isinstance(node[child_key], dict):
for k, v in list(node[child_key].items()):
node[child_key][k] = walk(v)
# arrays/items
if "items" in node:
node["items"] = walk(node["items"])
# allOf/if/then/else - pass through with integer→number conversions applied inside
for key in ("allOf",):
if key in node and isinstance(node[key], list):
node[key] = [walk(x) for x in node[key]]
if "if" in node:
node["if"] = walk(node["if"])
if "then" in node:
node["then"] = walk(node["then"])
if "else" in node:
node["else"] = walk(node["else"])
return node
return walk(s)
@staticmethod
def make_mcp_tool(tool: Tool, openai_tool_compatible: bool = True) -> MCPTool:
"""
Create an MCP tool from a Serena Tool instance.
:param tool: The Serena Tool instance to convert.
:param openai_tool_compatible: whether to process the tool schema to be compatible with OpenAI tools
(doesn't accept integer, needs number instead, etc.). This allows using Serena MCP within codex.
"""
func_name = tool.get_name()
func_doc = tool.get_apply_docstring() or ""
func_arg_metadata = tool.get_apply_fn_metadata()
is_async = False
parameters = func_arg_metadata.arg_model.model_json_schema()
if openai_tool_compatible:
parameters = SerenaMCPFactory._sanitize_for_openai_tools(parameters)
docstring = docstring_parser.parse(func_doc)
# Mount the tool description as a combination of the docstring description and
# the return value description, if it exists.
if docstring.description:
func_doc = f"{docstring.description.strip().strip('.')}."
overridden_description = tool.agent.get_context().tool_description_overrides.get(func_name, None)
if overridden_description is not None:
func_doc = overridden_description
elif docstring.description:
func_doc = docstring.description
else:
func_doc = ""
func_doc = func_doc.strip().strip(".")
if func_doc:
func_doc += "."
if docstring.returns and (docstring_returns_descr := docstring.returns.description):
# Only add a space before "Returns" if func_doc is not empty
prefix = " " if func_doc else ""
@@ -106,13 +230,14 @@ class SerenaMCPFactory:
pass
# noinspection PyProtectedMember
def _set_mcp_tools(self, mcp: FastMCP) -> None:
def _set_mcp_tools(self, mcp: FastMCP, openai_tool_compatible: bool = False) -> None:
"""Update the tools in the MCP server"""
if mcp is not None:
mcp._tool_manager._tools = {}
for tool in self._iter_tools():
mcp_tool = self.make_mcp_tool(tool)
mcp_tool = self.make_mcp_tool(tool, openai_tool_compatible=openai_tool_compatible)
mcp._tool_manager._tools[tool.get_name()] = mcp_tool
log.info(f"Starting MCP server with {len(mcp._tool_manager._tools)} tools: {list(mcp._tool_manager._tools.keys())}")
@abstractmethod
def _instantiate_agent(self, serena_config: SerenaConfig, modes: list[SerenaAgentMode]) -> None:
@@ -170,9 +295,8 @@ class SerenaMCPFactory:
# `.env` files (e.g. containing LOG_LEVEL) that are not supposed to override the MCP settings;
# retain only FASTMCP_ prefix for already set environment variables.
Settings.model_config = SettingsConfigDict(env_prefix="FASTMCP_")
mcp_settings: Settings = Settings(lifespan=self.server_lifespan, host=host, port=port)
mcp = FastMCP(**mcp_settings.model_dump())
instructions = self._get_initial_instructions()
mcp = FastMCP(lifespan=self.server_lifespan, host=host, port=port, instructions=instructions)
return mcp
@asynccontextmanager
@@ -181,6 +305,10 @@ class SerenaMCPFactory:
"""Manage server startup and shutdown lifecycle."""
yield None # ensures MyPy understands we yield None
@abstractmethod
def _get_initial_instructions(self) -> str:
pass
class SerenaMCPFactorySingleProcess(SerenaMCPFactory):
"""
@@ -207,8 +335,14 @@ class SerenaMCPFactorySingleProcess(SerenaMCPFactory):
assert self.agent is not None
yield from self.agent.get_exposed_tool_instances()
def _get_initial_instructions(self) -> str:
assert self.agent is not None
# we don't use the tool (which at the time of writing calls this method), since the tool may be disabled by the config
return self.agent.create_system_prompt()
@asynccontextmanager
async def server_lifespan(self, mcp_server: FastMCP) -> AsyncIterator[None]:
self._set_mcp_tools(mcp_server)
openai_tool_compatible = self.context.name in ["chatgpt", "codex", "oaicompat-agent"]
self._set_mcp_tools(mcp_server, openai_tool_compatible=openai_tool_compatible)
log.info("MCP server lifetime setup complete")
yield
src/serena/project.py
+143 -40
View File
@@ -1,26 +1,72 @@
import json
import logging
import os
from pathlib import Path
from typing import Any
import pathspec
from serena.config.serena_config import DEFAULT_TOOL_TIMEOUT, ProjectConfig
from serena.constants import SERENA_MANAGED_DIR_IN_HOME
from serena.config.serena_config import DEFAULT_TOOL_TIMEOUT, ProjectConfig, get_serena_managed_in_project_dir
from serena.constants import SERENA_FILE_ENCODING, SERENA_MANAGED_DIR_IN_HOME, SERENA_MANAGED_DIR_NAME
from serena.text_utils import MatchedConsecutiveLines, search_files
from serena.util.file_system import GitignoreParser, match_path
from solidlsp import SolidLanguageServer
from solidlsp.ls_config import Language, LanguageServerConfig
from solidlsp.ls_logger import LanguageServerLogger
from solidlsp.ls_utils import FileUtils
from solidlsp.settings import SolidLSPSettings
log = logging.getLogger(__name__)
class MemoriesManager:
def __init__(self, project_root: str):
self._memory_dir = Path(get_serena_managed_in_project_dir(project_root)) / "memories"
self._memory_dir.mkdir(parents=True, exist_ok=True)
self._encoding = SERENA_FILE_ENCODING
def _get_memory_file_path(self, name: str) -> Path:
# strip all .md from the name. Models tend to get confused, sometimes passing the .md extension and sometimes not.
name = name.replace(".md", "")
filename = f"{name}.md"
return self._memory_dir / filename
def load_memory(self, name: str) -> str:
memory_file_path = self._get_memory_file_path(name)
if not memory_file_path.exists():
return f"Memory file {name} not found, consider creating it with the `write_memory` tool if you need it."
with open(memory_file_path, encoding=self._encoding) as f:
return f.read()
def save_memory(self, name: str, content: str) -> str:
memory_file_path = self._get_memory_file_path(name)
with open(memory_file_path, "w", encoding=self._encoding) as f:
f.write(content)
return f"Memory {name} written."
def list_memories(self) -> list[str]:
return [f.name.replace(".md", "") for f in self._memory_dir.iterdir() if f.is_file()]
def delete_memory(self, name: str) -> str:
memory_file_path = self._get_memory_file_path(name)
memory_file_path.unlink()
return f"Memory {name} deleted."
class Project:
def __init__(self, project_root: str, project_config: ProjectConfig, is_newly_created: bool = False):
self.project_root = project_root
self.project_config = project_config
self.is_newly_created = is_newly_created
self.memories_manager = MemoriesManager(project_root)
self._is_newly_created = is_newly_created
# create .gitignore file in the project's Serena data folder if not yet present
serena_data_gitignore_path = os.path.join(self.path_to_serena_data_folder(), ".gitignore")
if not os.path.exists(serena_data_gitignore_path):
os.makedirs(os.path.dirname(serena_data_gitignore_path), exist_ok=True)
log.info(f"Creating .gitignore file in {serena_data_gitignore_path}")
with open(serena_data_gitignore_path, "w", encoding="utf-8") as f:
f.write(f"/{SolidLanguageServer.CACHE_FOLDER_NAME}\n")
# gather ignored paths from the project configuration and gitignore files
ignored_patterns = project_config.ignored_paths
@@ -28,9 +74,7 @@ class Project:
log.info(f"Using {len(ignored_patterns)} ignored paths from the explicit project configuration.")
log.debug(f"Ignored paths: {ignored_patterns}")
if project_config.ignore_all_files_in_gitignore:
log.info(f"Parsing all gitignore files in {self.project_root}")
gitignore_parser = GitignoreParser(self.project_root)
log.info(f"Found {len(gitignore_parser.get_ignore_specs())} gitignore files.")
for spec in gitignore_parser.get_ignore_specs():
log.debug(f"Adding {len(spec.patterns)} patterns from {spec.file_path} to the ignored paths.")
ignored_patterns.extend(spec.patterns)
@@ -62,9 +106,31 @@ class Project:
project_config = ProjectConfig.load(project_root, autogenerate=autogenerate)
return Project(project_root=str(project_root), project_config=project_config)
def path_to_serena_data_folder(self) -> str:
return os.path.join(self.project_root, SERENA_MANAGED_DIR_NAME)
def path_to_project_yml(self) -> str:
return os.path.join(self.project_root, self.project_config.rel_path_to_project_yml())
def get_activation_message(self) -> str:
"""
:return: a message providing information about the project upon activation (e.g. programming language, memories, initial prompt)
"""
if self._is_newly_created:
msg = f"Created and activated a new project with name '{self.project_name}' at {self.project_root}. "
else:
msg = f"The project with name '{self.project_name}' at {self.project_root} is activated."
msg += f"\nProgramming language: {self.project_config.language.value}; file encoding: {self.project_config.encoding}"
memories = self.memories_manager.list_memories()
if memories:
msg += (
f"\nAvailable project memories: {json.dumps(memories)}\n"
+ "Use the `read_memory` tool to read these memories later if they are relevant to the task."
)
if self.project_config.initial_prompt:
msg += f"\nAdditional project-specific instructions:\n {self.project_config.initial_prompt}"
return msg
def read_file(self, relative_path: str) -> str:
"""
Reads a file relative to the project root.
@@ -73,9 +139,7 @@ class Project:
:return: the content of the file
"""
abs_path = Path(self.project_root) / relative_path
if not abs_path.exists():
raise FileNotFoundError(f"File not found: {abs_path}")
return abs_path.read_text(encoding=self.project_config.encoding)
return FileUtils.read_file(str(abs_path), self.project_config.encoding)
def get_ignore_spec(self) -> pathspec.PathSpec:
"""
@@ -84,12 +148,10 @@ class Project:
"""
return self._ignore_spec
def _is_ignored_dirname(self, dirname: str) -> bool:
return dirname.startswith(".")
def _is_ignored_relative_path(self, relative_path: str, ignore_non_source_files: bool = True) -> bool:
def _is_ignored_relative_path(self, relative_path: str | Path, ignore_non_source_files: bool = True) -> bool:
"""
Determine whether a path should be ignored based on file type and ignore patterns.
Determine whether an existing path should be ignored based on file type and ignore patterns.
Raises `FileNotFoundError` if the path does not exist.
:param relative_path: Relative path to check
:param ignore_non_source_files: whether files that are not source files (according to the file masks
@@ -97,6 +159,12 @@ class Project:
:return: whether the path should be ignored
"""
# special case, never ignore the project root itself
# If the user ignores hidden files, "." might match against the corresponding PathSpec pattern.
# The empty string also points to the project root and should never be ignored.
if str(relative_path) in [".", ""]:
return False
abs_path = os.path.join(self.project_root, relative_path)
if not os.path.exists(abs_path):
raise FileNotFoundError(f"File {abs_path} not found, the ignore check cannot be performed")
@@ -111,35 +179,33 @@ class Project:
# Create normalized path for consistent handling
rel_path = Path(relative_path)
# Check each part of the path against always fulfilled ignore conditions
dir_parts = rel_path.parts
if is_file:
dir_parts = dir_parts[:-1]
for part in dir_parts:
if not part: # Skip empty parts (e.g., from leading '/')
continue
if self._is_ignored_dirname(part):
return True
# always ignore paths inside .git
if len(rel_path.parts) > 0 and rel_path.parts[0] == ".git":
return True
return match_path(relative_path, self.get_ignore_spec(), root_path=self.project_root)
return match_path(str(relative_path), self.get_ignore_spec(), root_path=self.project_root)
def is_ignored_path(self, path: str | Path) -> bool:
def is_ignored_path(self, path: str | Path, ignore_non_source_files: bool = False) -> bool:
"""
Checks whether the given path is ignored
:param path: the path to check, can be absolute or relative
:param ignore_non_source_files: whether to ignore files that are not source files
(according to the file masks determined by the project's programming language)
"""
path = Path(path)
if path.is_absolute():
relative_path = path.relative_to(self.project_root)
try:
relative_path = path.relative_to(self.project_root)
except ValueError:
# If the path is not relative to the project root, we consider it as an absolute path outside the project
# (which we ignore)
log.warning(f"Path {path} is not relative to the project root {self.project_root} and was therefore ignored")
return True
else:
relative_path = path
# always ignore paths inside .git
if len(relative_path.parts) > 0 and relative_path.parts[0] == ".git":
return True
return match_path(str(relative_path), self.get_ignore_spec(), root_path=self.project_root)
return self._is_ignored_relative_path(str(relative_path), ignore_non_source_files=ignore_non_source_files)
def is_path_in_project(self, path: str | Path) -> bool:
"""
@@ -154,16 +220,32 @@ class Project:
path = path.resolve()
return path.is_relative_to(_proj_root)
def validate_relative_path(self, relative_path: str) -> None:
def relative_path_exists(self, relative_path: str) -> bool:
"""
Validates that the given relative path is safe to read or edit,
meaning it's inside the project directory and is not ignored by git.
Checks if the given relative path exists in the project directory.
:param relative_path: the path to check, relative to the project root
:return: True if the path exists, False otherwise
"""
abs_path = Path(self.project_root) / relative_path
return abs_path.exists()
def validate_relative_path(self, relative_path: str, require_not_ignored: bool = False) -> None:
"""
Validates that the given relative path to an existing file/dir is safe to read or edit,
meaning it's inside the project directory.
Passing a path to a non-existing file will lead to a `FileNotFoundError`.
:param relative_path: the path to validate, relative to the project root
:param require_not_ignored: if True, the path must not be ignored according to the project's ignore settings
"""
if not self.is_path_in_project(relative_path):
raise ValueError(f"{relative_path=} points to path outside of the repository root; cannot access for safety reasons")
if self.is_ignored_path(relative_path):
raise ValueError(f"Path {relative_path} is ignored; cannot access for safety reasons")
if require_not_ignored:
if self.is_ignored_path(relative_path):
raise ValueError(f"Path {relative_path} is ignored; cannot access for safety reasons")
def gather_source_files(self, relative_path: str = "") -> list[str]:
"""Retrieves relative paths of all source files, optionally limited to the given path
@@ -178,15 +260,27 @@ class Project:
return [relative_path]
else:
for root, dirs, files in os.walk(start_path, followlinks=True):
dirs[:] = [d for d in dirs if not self._is_ignored_relative_path(os.path.join(root, d))]
# prevent recursion into ignored directories
dirs[:] = [d for d in dirs if not self.is_ignored_path(os.path.join(root, d))]
# collect non-ignored files
for file in files:
rel_file_path = os.path.relpath(os.path.join(root, file), start=self.project_root)
abs_file_path = os.path.join(root, file)
try:
if not self._is_ignored_relative_path(rel_file_path):
if not self.is_ignored_path(abs_file_path, ignore_non_source_files=True):
try:
rel_file_path = os.path.relpath(abs_file_path, start=self.project_root)
except Exception:
log.warning(
"Ignoring path '%s' because it appears to be outside of the project root (%s)",
abs_file_path,
self.project_root,
)
continue
rel_file_paths.append(rel_file_path)
except FileNotFoundError:
log.warning(
f"File {rel_file_path} not found (possibly due it being a symlink), skipping it in request_parsed_files",
f"File {abs_file_path} not found (possibly due it being a symlink), skipping it in request_parsed_files",
)
return rel_file_paths
@@ -215,6 +309,7 @@ class Project:
relative_file_paths,
pattern,
root_path=self.project_root,
file_reader=self.read_file,
context_lines_before=context_lines_before,
context_lines_after=context_lines_after,
paths_include_glob=paths_include_glob,
@@ -248,6 +343,7 @@ class Project:
log_level: int = logging.INFO,
ls_timeout: float | None = DEFAULT_TOOL_TIMEOUT - 5,
trace_lsp_communication: bool = False,
ls_specific_settings: dict[Language, Any] | None = None,
) -> SolidLanguageServer:
"""
Create a language server for a project. Note that you will have to start it
@@ -258,12 +354,15 @@ class Project:
:param log_level: the log level for the language server
:param ls_timeout: the timeout for the language server
:param trace_lsp_communication: whether to trace LSP communication
:param ls_specific_settings: optional LS specific configuration of the language server,
see docstrings in the inits of subclasses of SolidLanguageServer to see what values may be passed.
:return: the language server
"""
ls_config = LanguageServerConfig(
code_language=self.language,
ignored_paths=self._ignored_patterns,
trace_lsp_communication=trace_lsp_communication,
encoding=self.project_config.encoding,
)
ls_logger = LanguageServerLogger(log_level=log_level)
@@ -273,5 +372,9 @@ class Project:
ls_logger,
self.project_root,
timeout=ls_timeout,
solidlsp_settings=SolidLSPSettings(solidlsp_dir=SERENA_MANAGED_DIR_IN_HOME),
solidlsp_settings=SolidLSPSettings(
solidlsp_dir=SERENA_MANAGED_DIR_IN_HOME,
project_data_relative_path=SERENA_MANAGED_DIR_NAME,
ls_specific_settings=ls_specific_settings or {},
),
)
src/serena/prompt_factory.py
+5 -2
View File
@@ -1,4 +1,6 @@
from serena.constants import PROMPT_TEMPLATES_DIR
import os
from serena.constants import PROMPT_TEMPLATES_DIR_IN_USER_HOME, PROMPT_TEMPLATES_DIR_INTERNAL
from serena.generated.generated_prompt_factory import PromptFactory
@@ -8,4 +10,5 @@ class SerenaPromptFactory(PromptFactory):
"""
def __init__(self) -> None:
super().__init__(PROMPT_TEMPLATES_DIR)
os.makedirs(PROMPT_TEMPLATES_DIR_IN_USER_HOME, exist_ok=True)
super().__init__(prompts_dir=[PROMPT_TEMPLATES_DIR_IN_USER_HOME, PROMPT_TEMPLATES_DIR_INTERNAL])
src/serena/resources/config/contexts/agent.yml
+3 -1
View File
@@ -3,4 +3,6 @@ prompt: |
You are running in agent context where the system prompt is provided externally. You should use symbolic
tools when possible for code understanding and modification.
excluded_tools:
- initial_instructions
- initial_instructions
tool_description_overrides: {}
src/serena/resources/config/contexts/chatgpt.yml
+28
View File
@@ -0,0 +1,28 @@
description: A configuration specific for chatgpt, which has a limit of 30 tools and requires short descriptions.
prompt: |
You are running in desktop app context where the tools give you access to the code base as well as some
access to the file system, if configured. You interact with the user through a chat interface that is separated
from the code base. As a consequence, if you are in interactive mode, your communication with the user should
involve high-level thinking and planning as well as some summarization of any code edits that you make.
For viewing the code edits the user will view them in a separate code editor window, and the back-and-forth
between the chat and the code editor should be minimized as well as facilitated by you.
If complex changes have been made, advise the user on how to review them in the code editor.
If complex relationships that the user asked for should be visualized or explained, consider creating
a diagram in addition to your text-based communication. Note that in the chat interface you have various rendering
options for text, html, and mermaid diagrams, as has been explained to you in your initial instructions.
excluded_tools: []
included_optional_tools:
- switch_modes
tool_description_overrides:
find_symbol: |
Retrieves symbols matching `name_path` in a file.
Use `depth > 0` to include children. `name_path` can be: "foo": any symbol named "foo"; "foo/bar": "bar" within "foo"; "/foo/bar": only top-level "foo/bar"
replace_regex: |
Replaces text using regular expressions. Preferred for smaller edits where symbol-level tools aren't appropriate.
Use wildcards (.*?) to match large sections efficiently: "beginning.*?end" instead of specifying exact content.
Essential for multi-line replacements.
search_for_pattern: |
Flexible pattern search across codebase. Prefer symbolic operations when possible.
Uses DOTALL matching. Use non-greedy quantifiers (.*?) to avoid over-matching.
Supports file filtering via globs and code-only restriction.
src/serena/resources/config/contexts/codex.yml
+28
View File
@@ -0,0 +1,28 @@
description: Non-symbolic editing tools and general shell tool are excluded
prompt: |
You are running in IDE assistant context where file operations, basic (line-based) edits and reads,
and shell commands are handled by your own, internal tools.
The initial instructions and the current config inform you on which tools are available to you,
and how to use them.
Don't attempt to use any excluded tools, instead rely on your own internal tools
for achieving the basic file or shell operations.
If serena's tools can be used for achieving your task,
you should prioritize them. In particular, it is important that you avoid reading entire source code files,
unless it is strictly necessary! Instead, for exploring and reading code in a token-efficient manner,
you should use serena's overview and symbolic search tools. The call of the read_file tool on an entire source code
file should only happen in exceptional cases, usually you should first explore the file (by itself or as part of exploring
the directory containing it) using the symbol_overview tool, and then make targeted reads using find_symbol and other symbolic tools.
For non-code files or for reads where you don't know the symbol's name path you can use the patterns searching tool,
using the read_file as a last resort.
excluded_tools:
- create_text_file
- read_file
- execute_shell_command
- prepare_for_new_conversation
- replace_regex
tool_description_overrides: {}
src/serena/resources/config/contexts/context.template.yml
+8 -1
View File
@@ -1,4 +1,11 @@
# See Serena's documentation for more details on concept of contexts.
description: Description of the context, not used in the code.
prompt: Prompt that will form part of the system prompt/initial instructions for agents started in this context.
excluded_tools: []
excluded_tools: []
# several tools are excluded by default and have to be explicitly included by the user
included_optional_tools: []
# mapping of tool names to an override of their descriptions (the default description is the docstring of the Tool's apply method).
# Sometimes, tool descriptions are too long (e.g., for ChatGPT), or users may want to override them for another reason.
tool_description_overrides: {}
src/serena/resources/config/contexts/desktop-app.yml
+5 -1
View File
@@ -10,4 +10,8 @@ prompt: |
If complex relationships that the user asked for should be visualized or explained, consider creating
a diagram in addition to your text-based communication. Note that in the chat interface you have various rendering
options for text, html, and mermaid diagrams, as has been explained to you in your initial instructions.
excluded_tools: []
excluded_tools: []
included_optional_tools:
- switch_modes
tool_description_overrides: {}
src/serena/resources/config/contexts/ide-assistant.yml
+11 -9
View File
@@ -10,16 +10,18 @@ prompt: |
If serena's tools can be used for achieving your task,
you should prioritize them. In particular, it is important that you avoid reading entire source code files,
unless it is strictly necessary! Instead, for exploring and reading code in a token-efficient manner,
you should use serena's overview and symbolic search tools. The call of the read_file tool on an entire source code
file should only happen in exceptional cases, usually you should first explore the file (by itself or as part of exploring
the directory containing it) using the symbol_overview tool, and then make targeted reads using find_symbol and other symbolic tools.
For non-code files or for reads where you don't know the symbol's name path you can use the patterns searching tool,
using the read_file as a last resort.
you should use serena's overview and symbolic search tools.
Before reading a full file, it is usually best to first explore the file using the symbol_overview tool,
and then make targeted reads using find_symbol and other symbolic tools.
For non-code files or for reads where you don't know the symbol's name path, you can use the pattern searching tool,
and read full files only as a last resort.
excluded_tools:
- create_text_file
- read_file
- delete_lines
- replace_lines
- insert_at_line
- execute_shell_command
- execute_shell_command
- prepare_for_new_conversation
- replace_regex
tool_description_overrides: {}
src/serena/resources/config/contexts/oaicompat-agent.yml
+8
View File
@@ -0,0 +1,8 @@
description: All tools except InitialInstructionsTool for agent context, uses OpenAI compatible tool definitions
prompt: |
You are running in agent context where the system prompt is provided externally. You should use symbolic
tools when possible for code understanding and modification.
excluded_tools:
- initial_instructions
tool_description_overrides: {}
src/serena/resources/config/modes/editing.yml
+7 -5
View File
@@ -14,10 +14,10 @@ prompt: |
use the regex-based approach that is described below.
Let us first discuss the symbol-based approach.
Symbols are identified by their name path and relative file path, see the description of the `find_symbols` tool for more details
Symbols are identified by their name path and relative file path, see the description of the `find_symbol` tool for more details
on how the `name_path` matches symbols.
You can get information about available symbols by using the `get_symbols_overview` tool for finding top-level symbols in a file
or directory, or by using `find_symbol` if you already know the symbol's name path. You generally try to read as little code as possible
You can get information about available symbols by using the `get_symbols_overview` tool for finding top-level symbols in a file,
or by using `find_symbol` if you already know the symbol's name path. You generally try to read as little code as possible
while still solving your task, meaning you only read the bodies when you need to, and after you have found the symbol you want to edit.
Before calling symbolic reading tools, you should have a basic understanding of the repository structure that you can get from memories
or by using the `list_dir` and `find_file` tools (or similar).
@@ -34,7 +34,8 @@ prompt: |
You will generally be able to use the info from the snippets and the regex-based approach to adjust the references as well.
You can assume that all symbol editing tools are reliable, so you don't need to verify the results if the tool returns without error.
Now let us discuss the regex-based approach.
{% if 'replace_regex' in available_tools %}
Let us discuss the regex-based approach.
The regex-based approach is your primary tool for editing code whenever replacing or deleting a whole symbol would be a more expensive operation.
This is the case if you need to adjust just a few lines of code within a method, or a chunk that is much smaller than a whole symbol.
You use other tools to find the relevant content and
@@ -104,7 +105,8 @@ prompt: |
Generally, I remind you that you rely on the regex tool with providing you the correct feedback, no need for more verification!
IMPORTANT: REMEMBER TO USE WILDCARDS WHEN APPROPRIATE! I WILL BE VERY UNHAPPY IF YOU WRITE LONG REGEXES WITHOUT USING WILDCARDS INSTEAD!
excluded_tools:
{% endif %}
excluded_tools:
- replace_lines
- insert_at_line
- delete_lines
src/serena/resources/config/modes/mode.template.yml
+3
View File
@@ -2,3 +2,6 @@
description: Description of the mode, not used in the code.
prompt: Prompt that will form part of the message sent to the model when activating this mode.
excluded_tools: []
# several tools are excluded by default and have to be explicitly included by the user
included_optional_tools: []
src/serena/resources/config/prompt_templates/system_prompt.yml
+21 -12
View File
@@ -5,45 +5,54 @@ prompts:
system_prompt: |
You are a professional coding agent concerned with one particular codebase. You have
access to semantic coding tools on which you rely heavily for all your work, as well as collection of memory
files containing general information about the codebase. You operate in a frugal and intelligent manner, always
files containing general information about the codebase. You operate in a resource-efficient and intelligent manner, always
keeping in mind to not read or generate content that is not needed for the task at hand.
When reading code in order to answer a user question or task, you should try reading only the necessary code.
Some tasks may require you to understand the architecture of large parts of the codebase, while for others,
it may be enough to read a small set of symbols or a single file.
Generally, you should avoid reading entire files unless it is absolutely necessary, instead relying on
intelligent step-by-step acquisition of information. However, if you already read a file, it does not make
intelligent step-by-step acquisition of information. {% if 'ToolMarkerSymbolicRead' in available_markers %}However, if you already read a file, it does not make
sense to further analyse it with the symbolic tools (except for the `find_referencing_symbols` tool),
as you already have the information.
as you already have the information.{% endif %}
I WILL BE SERIOUSLY UPSET IF YOU READ ENTIRE FILES WITHOUT NEED! CONSIDER INSTEAD USING THE OVERVIEW TOOL AND
SYMBOLIC TOOLS TO READ ONLY THE NECESSARY CODE FIRST!
I WILL BE SERIOUSLY UPSET IF YOU READ ENTIRE FILES WITHOUT NEED!
{% if 'ToolMarkerSymbolicRead' in available_markers %}
CONSIDER INSTEAD USING THE OVERVIEW TOOL AND SYMBOLIC TOOLS TO READ ONLY THE NECESSARY CODE FIRST!
I WILL BE EVEN MORE UPSET IF AFTER HAVING READ AN ENTIRE FILE YOU KEEP READING THE SAME CONTENT WITH THE SYMBOLIC TOOLS!
THE PURPOSE OF THE SYMBOLIC TOOLS IS TO HAVE TO READ LESS CODE, NOT READ THE SAME CONTENT MULTIPLE TIMES!
{% endif %}
You can achieve the intelligent reading of code by using the symbolic tools for getting an overview of symbols and
the relations between them, and then only reading the bodies of symbols that are necessary to answer the question
or complete the task. You can also use the standard tools like list_dir, find_file and search_for_pattern if you need to.
or complete the task.
You can use the standard tools like list_dir, find_file and search_for_pattern if you need to.
When tools allow it, you pass the `relative_path` parameter to restrict the search to a specific file or directory.
For some tools, `relative_path` can only be a file path, so make sure to properly read the tool descriptions.
If you are unsure about a symbol's name or location (to the extent that substring_matching for the symbol name is not enough), you can use the `search_for_pattern` tool, which allows fast
and flexible search for patterns in the codebase. This way you can first find candidates for symbols or files,
and then proceed with the symbolic tools.
{% if 'search_for_pattern' in available_tools %}
If you are unsure about a symbol's name or location{% if 'find_symbol' in available_tools %} (to the extent that substring_matching for the symbol name is not enough){% endif %}, you can use the `search_for_pattern` tool, which allows fast
and flexible search for patterns in the codebase.{% if 'ToolMarkerSymbolicRead' in available_markers %}This way you can first find candidates for symbols or files,
and then proceed with the symbolic tools.{% endif %}
{% endif %}
Symbols are identified by their `name_path and `relative_path`, see the description of the `find_symbols` tool for more details
{% if 'ToolMarkerSymbolicRead' in available_markers %}
Symbols are identified by their `name_path and `relative_path`, see the description of the `find_symbol` tool for more details
on how the `name_path` matches symbols.
You can get information about available symbols by using the `get_symbols_overview` tool for finding top-level symbols in a file
or directory, or by using `find_symbol` if you already know the symbol's name path. You generally try to read as little code as possible
You can get information about available symbols by using the `get_symbols_overview` tool for finding top-level symbols in a file,
or by using `find_symbol` if you already know the symbol's name path. You generally try to read as little code as possible
while still solving your task, meaning you only read the bodies when you need to, and after you have found the symbol you want to edit.
For example, if you are working with python code and already know that you need to read the body of the constructor of the class Foo, you can directly
use `find_symbol` with the name path `Foo/__init__` and `include_body=True`. If you don't know yet which methods in `Foo` you need to read or edit,
you can use `find_symbol` with the name path `Foo`, `include_body=False` and `depth=1` to get all (top-level) methods of `Foo` before proceeding
to read the desired methods with `include_body=True`
You can understand relationships between symbols by using the `find_referencing_symbols` tool.
{% endif %}
{% if 'read_memory' in available_tools %}
You generally have access to memories and it may be useful for you to read them, but also only if they help you
to answer the question or complete the task. You can infer which memories are relevant to the current task by reading
the memory names and descriptions.
{% endif %}
The context and modes of operation are described below. From them you can infer how to interact with your user
and which tasks and kinds of interactions are expected of you.
src/serena/resources/dashboard/dashboard.js
+229 -6
View File
@@ -1,5 +1,6 @@
class LogMessage {
constructor(message, toolNames) {
message = this.escapeHtml(message);
const logLevel = this.determineLogLevel(message);
const highlightedMessage = this.highlightToolNames(message, toolNames);
this.$elem = $('<div>').addClass('log-' + logLevel).html(highlightedMessage + '\n');
@@ -27,6 +28,21 @@ class LogMessage {
});
return highlightedMessage;
}
escapeHtml (convertString) {
if (typeof convertString !== 'string') return convertString;
const patterns = {
'<' : '&lt;',
'>' : '&gt;',
'&' : '&amp;',
'"' : '&quot;',
'\'' : '&#x27;',
'`' : '&#x60;'
};
return convertString.replace(/[<>&"'`]/g, match => patterns[match]);
};
}
class Dashboard {
@@ -36,6 +52,7 @@ class Dashboard {
this.toolNames = [];
this.currentMaxIdx = -1;
this.pollInterval = null;
this.failureCount = 0;
this.$logContainer = $('#log-container');
this.$errorContainer = $('#error-container');
this.$loadButton = $('#load-logs');
@@ -44,6 +61,9 @@ class Dashboard {
this.$statsSection = $('#stats-section');
this.$refreshStats = $('#refresh-stats');
this.$clearStats = $('#clear-stats');
this.$themeToggle = $('#theme-toggle');
this.$themeIcon = $('#theme-icon');
this.$themeText = $('#theme-text');
this.countChart = null;
this.tokensChart = null;
@@ -56,6 +76,10 @@ class Dashboard {
this.$toggleStats.click(this.toggleStats.bind(this));
this.$refreshStats.click(this.loadStats.bind(this));
this.$clearStats.click(this.clearStats.bind(this));
this.$themeToggle.click(this.toggleTheme.bind(this));
// initialize theme
this.initializeTheme();
// initialize the application
this.loadToolNames().then(function() {
@@ -83,6 +107,10 @@ class Dashboard {
});
}
updateTitle(activeProject) {
document.title = activeProject ? `${activeProject} – Serena Dashboard` : 'Serena Dashboard';
}
loadLogs() {
console.log("Loading logs");
let self = this;
@@ -119,6 +147,8 @@ class Dashboard {
$('#log-container').html('<div class="loading">No log messages found.</div>');
}
self.updateTitle(response.active_project);
// Start periodic polling for new logs
self.startPeriodicPolling();
},
@@ -145,6 +175,7 @@ class Dashboard {
start_idx: self.currentMaxIdx + 1
}),
success: function(response) {
self.failureCount = 0;
// Only append new messages if we have any
if (response.messages && response.messages.length > 0) {
let wasAtBottom = false;
@@ -171,9 +202,17 @@ class Dashboard {
// Update max_idx even if no new messages
self.currentMaxIdx = response.max_idx || self.currentMaxIdx;
}
// Update window title with active project
self.updateTitle(response.active_project);
},
error: function(xhr, status, error) {
console.error('Error polling for new logs:', error);
self.failureCount++;
if (self.failureCount >= 3) {
console.log('Server appears to be down, closing tab');
window.close();
}
}
});
}
@@ -278,6 +317,11 @@ class Dashboard {
// Register datalabels plugin
Chart.register(ChartDataLabels);
// Get theme-aware colors
const isDark = document.documentElement.getAttribute('data-theme') === 'dark';
const textColor = isDark ? '#ffffff' : '#000000';
const gridColor = isDark ? '#444' : '#ddd';
// Tool calls pie chart
this.countChart = new Chart(countCtx, {
type: 'pie',
@@ -290,7 +334,12 @@ class Dashboard {
},
options: {
plugins: {
legend: { display: true },
legend: {
display: true,
labels: {
color: textColor
}
},
datalabels: {
display: true,
color: 'white',
@@ -313,7 +362,12 @@ class Dashboard {
},
options: {
plugins: {
legend: { display: true },
legend: {
display: true,
labels: {
color: textColor
}
},
datalabels: {
display: true,
color: 'white',
@@ -336,7 +390,12 @@ class Dashboard {
},
options: {
plugins: {
legend: { display: true },
legend: {
display: true,
labels: {
color: textColor
}
},
datalabels: {
display: true,
color: 'white',
@@ -372,21 +431,56 @@ class Dashboard {
},
options: {
responsive: true,
plugins: {
legend: {
labels: {
color: textColor
}
}
},
scales: {
x: {
ticks: {
color: textColor
},
grid: {
color: gridColor
}
},
y: {
type: 'linear',
display: true,
position: 'left',
beginAtZero: true,
title: { display: true, text: 'Input Tokens' }
title: {
display: true,
text: 'Input Tokens',
color: textColor
},
ticks: {
color: textColor
},
grid: {
color: gridColor
}
},
y1: {
type: 'linear',
display: true,
position: 'right',
beginAtZero: true,
title: { display: true, text: 'Output Tokens' },
grid: { drawOnChartArea: false }
title: {
display: true,
text: 'Output Tokens',
color: textColor
},
ticks: {
color: textColor
},
grid: {
drawOnChartArea: false,
color: gridColor
}
}
}
}
@@ -414,6 +508,135 @@ class Dashboard {
$('#stats-summary').html(tableHtml);
}
initializeTheme() {
// Check if user has manually set a theme preference
const savedTheme = localStorage.getItem('serena-theme');
if (savedTheme) {
// User has manually set a preference, use it
this.setTheme(savedTheme);
} else {
// No manual preference, detect system color scheme
this.detectSystemTheme();
}
// Listen for system theme changes
this.setupSystemThemeListener();
}
detectSystemTheme() {
// Check if system prefers dark mode
const prefersDark = window.matchMedia('(prefers-color-scheme: dark)').matches;
const theme = prefersDark ? 'dark' : 'light';
this.setTheme(theme);
}
setupSystemThemeListener() {
// Listen for changes in system color scheme
const mediaQuery = window.matchMedia('(prefers-color-scheme: dark)');
const handleSystemThemeChange = (e) => {
// Only auto-switch if user hasn't manually set a preference
const savedTheme = localStorage.getItem('serena-theme');
if (!savedTheme) {
const newTheme = e.matches ? 'dark' : 'light';
this.setTheme(newTheme);
}
};
// Add listener for system theme changes
if (mediaQuery.addEventListener) {
mediaQuery.addEventListener('change', handleSystemThemeChange);
} else {
// Fallback for older browsers
mediaQuery.addListener(handleSystemThemeChange);
}
}
toggleTheme() {
const currentTheme = document.documentElement.getAttribute('data-theme') || 'light';
const newTheme = currentTheme === 'light' ? 'dark' : 'light';
// When user manually toggles, save their preference
localStorage.setItem('serena-theme', newTheme);
this.setTheme(newTheme);
}
setTheme(theme) {
// Set the theme on the document element
document.documentElement.setAttribute('data-theme', theme);
// Update the toggle button
if (theme === 'dark') {
this.$themeIcon.text('☀️');
this.$themeText.text('Light');
} else {
this.$themeIcon.text('🌙');
this.$themeText.text('Dark');
}
// Update the logo based on theme
this.updateLogo(theme);
// Save to localStorage
localStorage.setItem('serena-theme', theme);
// Update charts if they exist
this.updateChartsTheme();
}
updateLogo(theme) {
const logoElement = document.getElementById('serena-logo');
if (logoElement) {
if (theme === 'dark') {
logoElement.src = 'serena-logs-dark-mode.png';
} else {
logoElement.src = 'serena-logs.png';
}
}
}
updateChartsTheme() {
const isDark = document.documentElement.getAttribute('data-theme') === 'dark';
const textColor = isDark ? '#ffffff' : '#000000';
const gridColor = isDark ? '#444' : '#ddd';
// Update existing charts
if (this.countChart) {
this.countChart.options.scales.x.ticks.color = textColor;
this.countChart.options.scales.y.ticks.color = textColor;
this.countChart.options.scales.x.grid.color = gridColor;
this.countChart.options.scales.y.grid.color = gridColor;
this.countChart.update();
}
if (this.inputChart) {
this.inputChart.options.scales.x.ticks.color = textColor;
this.inputChart.options.scales.y.ticks.color = textColor;
this.inputChart.options.scales.x.grid.color = gridColor;
this.inputChart.options.scales.y.grid.color = gridColor;
this.inputChart.update();
}
if (this.outputChart) {
this.outputChart.options.scales.x.ticks.color = textColor;
this.outputChart.options.scales.y.ticks.color = textColor;
this.outputChart.options.scales.x.grid.color = gridColor;
this.outputChart.options.scales.y.grid.color = gridColor;
this.outputChart.update();
}
if (this.tokensChart) {
this.tokensChart.options.scales.x.ticks.color = textColor;
this.tokensChart.options.scales.y.ticks.color = textColor;
this.tokensChart.options.scales.y1.ticks.color = textColor;
this.tokensChart.options.scales.x.grid.color = gridColor;
this.tokensChart.options.scales.y.grid.color = gridColor;
this.tokensChart.options.scales.y1.grid.color = gridColor;
this.tokensChart.update();
}
}
shutdown() {
const self = this;
const _shutdown = function () {
src/serena/resources/dashboard/index.html
+139 -46
View File
@@ -1,5 +1,6 @@
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
@@ -12,21 +13,65 @@
<script src="https://cdn.jsdelivr.net/npm/chartjs-plugin-datalabels@2"></script>
<script src="dashboard.js"></script>
<style>
:root {
/* Light theme variables */
--bg-primary: #f5f5f5;
--bg-secondary: #ffffff;
--text-primary: #000000;
--text-secondary: #333333;
--text-muted: #666666;
--border-color: #ddd;
--btn-primary: #eaa45d;
--btn-hover: #dca662;
--btn-disabled: #6c757d;
--shadow: 0 2px 4px rgba(0, 0, 0, 0.1);
--tool-highlight: #ffff00;
--tool-highlight-text: #000000;
--log-debug: #808080;
--log-info: #000000;
--log-warning: #FF8C00;
--log-error: #FF0000;
--stats-header: #f8f9fa;
}
[data-theme="dark"] {
/* Dark theme variables */
--bg-primary: #1a1a1a;
--bg-secondary: #2d2d2d;
--text-primary: #ffffff;
--text-secondary: #e0e0e0;
--text-muted: #b0b0b0;
--border-color: #444;
--btn-primary: #eaa45d;
--btn-hover: #dca662;
--btn-disabled: #6c757d;
--shadow: 0 2px 4px rgba(0, 0, 0, 0.3);
--tool-highlight: #ffd700;
--tool-highlight-text: #000000;
--log-debug: #808080;
--log-info: #ffffff;
--log-warning: #FF8C00;
--log-error: #FF0000;
--stats-header: #3a3a3a;
}
body {
font-family: 'Consolas', 'Monaco', 'Courier New', monospace;
margin: 0;
padding: 20px;
background-color: #f5f5f5;
background-color: var(--bg-primary);
color: var(--text-primary);
transition: background-color 0.3s ease, color 0.3s ease;
}
.header {
text-align: center;
margin-bottom: 20px;
}
.log-container {
background-color: white;
border: 1px solid #ddd;
background-color: var(--bg-secondary);
border: 1px solid var(--border-color);
border-radius: 5px;
height: 600px;
overflow-y: auto;
@@ -35,11 +80,18 @@
white-space: pre-wrap;
font-size: 12px;
line-height: 1.4;
color: var(--text-primary);
transition: background-color 0.3s ease, border-color 0.3s ease, color 0.3s ease;
}
.controls {
margin-bottom: 10px;
text-align: center;
display: flex;
justify-content: center;
align-items: center;
gap: 10px;
flex-wrap: wrap;
}
.logo {
@@ -48,58 +100,84 @@
}
.btn {
background-color: #eaa45d;
background-color: var(--btn-primary);
color: white;
border: none;
padding: 8px 16px;
border-radius: 4px;
cursor: pointer;
font-size: 14px;
transition: background-color 0.3s ease;
}
.btn:hover {
background-color: #dca662;
background-color: var(--btn-hover);
}
.btn:disabled {
background-color: #6c757d;
background-color: var(--btn-disabled);
cursor: not-allowed;
}
.theme-toggle {
display: flex;
align-items: center;
gap: 5px;
background-color: var(--bg-secondary);
border: 1px solid var(--border-color);
border-radius: 4px;
padding: 6px 12px;
cursor: pointer;
transition: background-color 0.3s ease, border-color 0.3s ease;
}
.theme-toggle:hover {
background-color: var(--border-color);
}
.theme-toggle .icon {
font-size: 16px;
}
.log-debug {
color: #808080; /* Gray */
color: var(--log-debug);
}
.log-info {
color: #000000; /* Black */
color: var(--log-info);
}
.log-warning {
color: #FF8C00; /* Dark Orange */
color: var(--log-warning);
}
.log-error {
color: #FF0000; /* Red */
color: var(--log-error);
}
.log-default {
color: #000000; /* Black */
color: var(--log-info);
}
/* Tool name highlighting */
.tool-name {
background-color: #ffff00; /* Yellow background */
background-color: var(--tool-highlight);
color: var(--tool-highlight-text);
font-weight: bold;
}
.loading {
text-align: center;
color: #666;
color: var(--text-muted);
font-style: italic;
}
.error-message {
color: #FF0000;
color: var(--log-error);
text-align: center;
margin: 10px 0;
}
.charts-container {
display: flex;
flex-wrap: wrap;
@@ -108,72 +186,86 @@
max-width: 1400px;
margin: 0 auto;
}
.chart-group {
flex: 1;
min-width: 280px;
max-width: 320px;
text-align: center;
}
.chart-wide {
flex: 0 0 100%;
min-width: 100%;
margin-top: 10px;
}
.chart-group h3 {
margin: 0 0 10px 0;
color: #333;
color: var(--text-secondary);
}
.stats-summary {
margin: 0 auto;
border-collapse: collapse;
background: white;
background: var(--bg-secondary);
border-radius: 5px;
overflow: hidden;
box-shadow: 0 2px 4px rgba(0,0,0,0.1);
box-shadow: var(--shadow);
transition: background-color 0.3s ease, box-shadow 0.3s ease;
}
.stats-summary th,
.stats-summary td {
padding: 10px 20px;
text-align: left;
border-bottom: 1px solid #ddd;
border-bottom: 1px solid var(--border-color);
color: var(--text-primary);
transition: border-color 0.3s ease, color 0.3s ease;
}
.stats-summary th {
background-color: #f8f9fa;
background-color: var(--stats-header);
font-weight: bold;
transition: background-color 0.3s ease;
}
.stats-summary tr:last-child td {
border-bottom: none;
}
@media (max-width: 768px) {
.charts-container {
flex-direction: column;
}
.chart-group,
.chart-wide {
min-width: auto;
max-width: none;
}
.controls {
flex-direction: column;
gap: 5px;
}
}
</style>
</head>
<body>
<div class="header">
<img src="serena-logs.png" alt="Serena" style="max-width: 400px; height: auto;">
<img id="serena-logo" src="serena-logs.png" alt="Serena" style="max-width: 400px; height: auto;">
</div>
<div class="controls">
<button id="load-logs" class="btn">Reload Log</button>
<button id="shutdown" class="btn">Shutdown Server</button>
<button id="toggle-stats" class="btn">Show Stats</button>
<div id="theme-toggle" class="theme-toggle" title="Toggle theme">
<span class="icon" id="theme-icon">🌙</span>
<span id="theme-text">Dark</span>
</div>
</div>
<div id="error-container"></div>
@@ -184,14 +276,14 @@
<button id="refresh-stats" class="btn">Refresh Stats</button>
<button id="clear-stats" class="btn">Clear Stats</button>
</div>
<div id="stats-summary" style="margin-bottom:20px; text-align:center;"></div>
<div id="estimator-name" style="text-align:center; margin-bottom:10px;"></div>
<div id="no-stats-message" style="text-align:center; color:#666; font-style:italic; display:none;">
No tool stats collected. Have you enabled tool stats collection in the configuration?
<div id="no-stats-message" style="text-align:center; color:var(--text-muted); font-style:italic; display:none;">
No tool stats collected. Have you enabled tool stats collection in the configuration?
</div>
<div class="charts-container">
<div class="chart-group">
<h3>Tool Calls</h3>
@@ -213,9 +305,10 @@
</div>
<script>
$(document).ready(function() {
$(document).ready(function () {
const dashboard = new Dashboard();
});
</script>
</body>
</html>
</html>
src/serena/resources/dashboard/serena-logs-dark-mode.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 13 KiB

src/serena/resources/project.template.yml
+6 -3
View File
@@ -5,13 +5,17 @@
# * csharp: Requires the presence of a .sln file in the project folder.
language: python
# the encoding used by text files in the project
# For a list of possible encodings, see https://docs.python.org/3.11/library/codecs.html#standard-encodings
encoding: "utf-8"
# whether to use the project's gitignore file to ignore files
# Added on 2025-04-07
ignore_all_files_in_gitignore: true
# list of additional paths to ignore
# same syntax as gitignore, so you can use * and **
# Was previously called `ignored_dirs`, please update your config if you are using that.
# Added (renamed)on 2025-04-07
# Added (renamed) on 2025-04-07
ignored_paths: []
# whether the project is in read-only mode
@@ -19,7 +23,6 @@ ignored_paths: []
# Added on 2025-04-18
read_only: false
# list of tool names to exclude. We recommend not excluding any tools, see the readme for more details.
# Below is the complete list of tools for convenience.
# To make sure you have the latest list of tools, and to view their descriptions,
@@ -35,7 +38,7 @@ read_only: false
# * `find_referencing_symbols`: Finds symbols that reference the symbol at the given location (optionally filtered by type).
# * `find_symbol`: Performs a global (or local) search for symbols with/containing a given name/substring (optionally filtered by type).
# * `get_current_config`: Prints the current configuration of the agent, including the active and available projects, tools, contexts, and modes.
# * `get_symbols_overview`: Gets an overview of the top-level symbols defined in a given file or directory.
# * `get_symbols_overview`: Gets an overview of the top-level symbols defined in a given file.
# * `initial_instructions`: Gets the initial instructions for the current project.
# Should only be used in settings where the system prompt cannot be set,
# e.g. in clients you have no control over, like Claude Desktop.
src/serena/resources/serena_config.template.yml
+13
View File
@@ -33,6 +33,14 @@ trace_lsp_communication: False
# whether to trace the communication between Serena and the language servers.
# This is useful for debugging language server issues.
ls_specific_settings: {}
# Added on 23.08.2025
# Advanced configuration option allowing to configure language server implementation specific options. Maps the language
# (same entry as in project.yml) to the options.
# Have a look at the docstring of the constructors of the LS implementations within solidlsp (e.g., for C# or PHP) to see which options are available.
# No documentation on options means no options are available.
#
tool_timeout: 240
# timeout, in seconds, after which tool executions are terminated
@@ -48,6 +56,11 @@ jetbrains: False
# NOTE: The plugin is yet unreleased. This is for Serena developers only.
default_max_tool_answer_chars: 150000
# Used as default for tools where the apply method has a default maximal answer length.
# Even though the value of the max_answer_chars can be changed when calling the tool, it may make sense to adjust this default
# through the global configuration.
record_tool_usage_stats: False
# whether to record tool usage statistics, they will be shown in the web dashboard if recording is active.
src/serena/symbol.py
+9 -5
View File
@@ -485,14 +485,14 @@ class LanguageServerSymbolRetriever:
return symbols
def get_document_symbols(self, relative_path: str) -> list[LanguageServerSymbol]:
symbol_dicts, roots = self._lang_server.request_document_symbols(relative_path, include_body=False)
symbol_dicts, _roots = self._lang_server.request_document_symbols(relative_path, include_body=False)
symbols = [LanguageServerSymbol(s) for s in symbol_dicts]
return symbols
def find_by_location(self, location: LanguageServerSymbolLocation) -> LanguageServerSymbol | None:
if location.relative_path is None:
return None
symbol_dicts, roots = self._lang_server.request_document_symbols(location.relative_path, include_body=False)
symbol_dicts, _roots = self._lang_server.request_document_symbols(location.relative_path, include_body=False)
for symbol_dict in symbol_dicts:
symbol = LanguageServerSymbol(symbol_dict)
if symbol.location == location:
@@ -584,13 +584,17 @@ class LanguageServerSymbolRetriever:
name_path: str
kind: int
@classmethod
def from_symbol(cls, symbol: LanguageServerSymbol) -> Self:
return cls(name_path=symbol.get_name_path(), kind=int(symbol.symbol_kind))
def get_symbol_overview(self, relative_path: str) -> dict[str, list[SymbolOverviewElement]]:
path_to_symbol_infos = self._lang_server.request_overview(relative_path)
path_to_unified_symbols = self._lang_server.request_overview(relative_path)
result = {}
for file_path, symbols in path_to_symbol_infos.items():
for file_path, unified_symbols in path_to_unified_symbols.items():
# TODO: maybe include not just top-level symbols? We could filter by kind to exclude variables
# The language server methods would need to be adjusted for this.
result[file_path] = [self.SymbolOverviewElement(name_path=symbol[0], kind=int(symbol[1])) for symbol in symbols]
result[file_path] = [self.SymbolOverviewElement.from_symbol(LanguageServerSymbol(s)) for s in unified_symbols]
return result
src/serena/text_utils.py
+47 -8
View File
@@ -9,6 +9,8 @@ from typing import Any, Self
from joblib import Parallel, delayed
from serena.constants import DEFAULT_SOURCE_FILE_ENCODING
log = logging.getLogger(__name__)
@@ -236,11 +238,34 @@ def search_text(
def default_file_reader(file_path: str) -> str:
"""Reads using utf-8 encoding."""
with open(file_path, encoding="utf-8") as f:
"""Reads using the default encoding."""
with open(file_path, encoding=DEFAULT_SOURCE_FILE_ENCODING) as f:
return f.read()
def expand_braces(pattern: str) -> list[str]:
"""
Expands brace patterns in a glob string.
For example, "**/*.{js,jsx,ts,tsx}" becomes ["**/*.js", "**/*.jsx", "**/*.ts", "**/*.tsx"].
Handles multiple brace sets as well.
"""
patterns = [pattern]
while any("{" in p for p in patterns):
new_patterns = []
for p in patterns:
match = re.search(r"\{([^{}]+)\}", p)
if match:
prefix = p[: match.start()]
suffix = p[match.end() :]
options = match.group(1).split(",")
for option in options:
new_patterns.append(f"{prefix}{option}{suffix}")
else:
new_patterns.append(p)
patterns = new_patterns
return patterns
def glob_match(pattern: str, path: str) -> bool:
"""
Match a file path against a glob pattern.
@@ -251,6 +276,13 @@ def glob_match(pattern: str, path: str) -> bool:
- ? matches a single character except /
- [seq] matches any character in seq
Supports brace expansion:
- {a,b,c} expands to multiple patterns (including nesting)
Unsupported patterns:
- Bash extended glob features are unavailable in Python's fnmatch
- Extended globs like !(), ?(), +(), *(), @() are not supported
:param pattern: Glob pattern (e.g., 'src/**/*.py', '**agent.py')
:param path: File path to match against
:return: True if path matches pattern
@@ -313,14 +345,21 @@ def search_files(
"""
# Pre-filter paths (done sequentially to avoid overhead)
# Use proper glob matching instead of gitignore patterns
include_patterns = expand_braces(paths_include_glob) if paths_include_glob else None
exclude_patterns = expand_braces(paths_exclude_glob) if paths_exclude_glob else None
filtered_paths = []
for path in relative_file_paths:
if paths_include_glob and not glob_match(paths_include_glob, path):
log.debug(f"Skipping {path}: does not match include pattern {paths_include_glob}")
continue
if paths_exclude_glob and glob_match(paths_exclude_glob, path):
log.debug(f"Skipping {path}: matches exclude pattern {paths_exclude_glob}")
continue
if include_patterns:
if not any(glob_match(p, path) for p in include_patterns):
log.debug(f"Skipping {path}: does not match include pattern {paths_include_glob}")
continue
if exclude_patterns:
if any(glob_match(p, path) for p in exclude_patterns):
log.debug(f"Skipping {path}: matches exclude pattern {paths_exclude_glob}")
continue
filtered_paths.append(path)
log.info(f"Processing {len(filtered_paths)} files.")
src/serena/tools/cmd_tools.py
+22 -13
View File
@@ -2,7 +2,9 @@
Tools supporting the execution of (external) commands
"""
from serena.tools import TOOL_DEFAULT_MAX_ANSWER_LENGTH, Tool, ToolMarkerCanEdit
import os.path
from serena.tools import Tool, ToolMarkerCanEdit
from serena.util.shell import execute_shell_command
@@ -16,28 +18,35 @@ class ExecuteShellCommandTool(Tool, ToolMarkerCanEdit):
command: str,
cwd: str | None = None,
capture_stderr: bool = True,
max_answer_chars: int = TOOL_DEFAULT_MAX_ANSWER_LENGTH,
max_answer_chars: int = -1,
) -> str:
"""
Execute a shell command and return its output.
IMPORTANT: you should always consider the memory about suggested shell commands before using this tool.
If this memory was not loaded in the current conversation, you should load it using the `read_memory` tool
before using this tool.
You should have at least once looked at the suggested shell commands from the corresponding memory
created during the onboarding process before using this tool.
Never execute unsafe shell commands like `rm -rf /` or similar! Generally be very careful with deletions.
Execute a shell command and return its output. If there is a memory about suggested commands, read that first.
Never execute unsafe shell commands!
IMPORTANT: Do not use this tool to start
* long-running processes (e.g. servers) that are not intended to terminate quickly,
* processes that require user interaction.
:param command: the shell command to execute
:param cwd: the working directory to execute the command in. If None, the project root will be used.
:param capture_stderr: whether to capture and return stderr output
:param max_answer_chars: if the output is longer than this number of characters,
no content will be returned. Don't adjust unless there is really no other way to get the content
no content will be returned. -1 means using the default value, don't adjust unless there is no other way to get the content
required for the task.
:return: a JSON object containing the command's stdout and optionally stderr output
"""
_cwd = cwd or self.get_project_root()
if cwd is None:
_cwd = self.get_project_root()
else:
if os.path.isabs(cwd):
_cwd = cwd
else:
_cwd = os.path.join(self.get_project_root(), cwd)
if not os.path.isdir(_cwd):
raise FileNotFoundError(
f"Specified a relative working directory ({cwd}), but the resulting path is not a directory: {_cwd}"
)
result = execute_shell_command(command, cwd=_cwd, capture_stderr=capture_stderr)
result = result.json()
return self._limit_length(result, max_answer_chars)
src/serena/tools/config_tools.py
+6 -24
View File
@@ -1,41 +1,23 @@
import json
from serena.config.context_mode import SerenaAgentMode
from serena.tools import Tool, ToolMarkerDoesNotRequireActiveProject
from serena.tools import Tool, ToolMarkerDoesNotRequireActiveProject, ToolMarkerOptional
class ActivateProjectTool(Tool, ToolMarkerDoesNotRequireActiveProject):
"""
Activates a project by name.
Activates a project based on the project name or path.
"""
def apply(self, project: str) -> str:
"""
Activates the project with the given name.
Activates the project with the given name or path.
:param project: the name of a registered project to activate or a path to a project directory
"""
active_project = self.agent.activate_project_from_path_or_name(project)
if active_project.is_newly_created:
result_str = (
f"Created and activated a new project with name '{active_project.project_name}' at {active_project.project_root}, language: {active_project.project_config.language.value}. "
"You can activate this project later by name.\n"
f"The project's Serena configuration is in {active_project.path_to_project_yml()}. In particular, you may want to edit the project name and the initial prompt."
)
else:
result_str = f"Activated existing project with name '{active_project.project_name}' at {active_project.project_root}, language: {active_project.project_config.language.value}"
if active_project.project_config.initial_prompt:
result_str += f"\nAdditional project information:\n {active_project.project_config.initial_prompt}"
result_str += (
f"\nAvailable memories:\n {json.dumps(list(self.memories_manager.list_memories()))}"
+ "You should not read these memories directly, but rather use the `read_memory` tool to read them later if needed for the task."
)
result_str += f"\nAvailable tools:\n {json.dumps(self.agent.get_active_tool_names())}"
return result_str
return active_project.get_activation_message()
class RemoveProjectTool(Tool, ToolMarkerDoesNotRequireActiveProject):
class RemoveProjectTool(Tool, ToolMarkerDoesNotRequireActiveProject, ToolMarkerOptional):
"""
Removes a project from the Serena configuration.
"""
@@ -50,7 +32,7 @@ class RemoveProjectTool(Tool, ToolMarkerDoesNotRequireActiveProject):
return f"Successfully removed project '{project_name}' from configuration."
class SwitchModesTool(Tool):
class SwitchModesTool(Tool, ToolMarkerOptional):
"""
Activates modes by providing a list of their names
"""
src/serena/tools/file_tools.py
+57 -48
View File
@@ -14,7 +14,7 @@ from fnmatch import fnmatch
from pathlib import Path
from serena.text_utils import search_files
from serena.tools import SUCCESS_RESULT, TOOL_DEFAULT_MAX_ANSWER_LENGTH, EditedFileContext, Tool, ToolMarkerCanEdit
from serena.tools import SUCCESS_RESULT, EditedFileContext, Tool, ToolMarkerCanEdit, ToolMarkerOptional
from serena.util.file_system import scan_directory
@@ -23,13 +23,10 @@ class ReadFileTool(Tool):
Reads a file within the project directory.
"""
def apply(
self, relative_path: str, start_line: int = 0, end_line: int | None = None, max_answer_chars: int = TOOL_DEFAULT_MAX_ANSWER_LENGTH
) -> str:
def apply(self, relative_path: str, start_line: int = 0, end_line: int | None = None, max_answer_chars: int = -1) -> str:
"""
Reads the given file or a chunk of it. Generally, symbolic operations
like find_symbol or find_referencing_symbols should be preferred if you know which symbols you are looking for.
Reading the entire file is only recommended if there is no other way to get the content required for the task.
:param relative_path: the relative path to the file to read
:param start_line: the 0-based index of the first line to be retrieved.
@@ -39,14 +36,13 @@ class ReadFileTool(Tool):
required for the task.
:return: the full text of the file at the given relative path
"""
self.project.validate_relative_path(relative_path)
self.project.validate_relative_path(relative_path, require_not_ignored=True)
result = self.project.read_file(relative_path)
result_lines = result.splitlines()
if end_line is None:
result_lines = result_lines[start_line:]
else:
self.lines_read.add_lines_read(relative_path, (start_line, end_line))
result_lines = result_lines[start_line : end_line + 1]
result = "\n".join(result_lines)
@@ -60,30 +56,29 @@ class CreateTextFileTool(Tool, ToolMarkerCanEdit):
def apply(self, relative_path: str, content: str) -> str:
"""
Write a new file (or overwrite an existing file). For existing files, it is strongly recommended
to use symbolic operations like replace_symbol_body or insert_after_symbol/insert_before_symbol, if possible.
You can also use insert_at_line to insert content at a specific line for existing files if the symbolic operations
are not the right choice for what you want to do.
If ever used on an existing file, the content has to be the complete content of that file (so it
may never end with something like "The remaining content of the file is left unchanged.").
For operations that just replace a part of a file, use the replace_lines or the symbolic editing tools instead.
Write a new file or overwrite an existing file.
:param relative_path: the relative path to the file to create
:param content: the (utf-8-encoded) content to write to the file
:param content: the (appropriately encoded) content to write to the file
:return: a message indicating success or failure
"""
self.project.validate_relative_path(relative_path)
abs_path = (Path(self.get_project_root()) / relative_path).resolve()
project_root = self.get_project_root()
abs_path = (Path(project_root) / relative_path).resolve()
will_overwrite_existing = abs_path.exists()
if will_overwrite_existing:
self.project.validate_relative_path(relative_path, require_not_ignored=True)
else:
assert abs_path.is_relative_to(
self.get_project_root()
), f"Cannot create file outside of the project directory, got {relative_path=}"
abs_path.parent.mkdir(parents=True, exist_ok=True)
abs_path.write_text(content, encoding="utf-8")
abs_path.write_text(content, encoding=self.project.project_config.encoding)
answer = f"File created: {relative_path}."
if will_overwrite_existing:
answer += " Overwrote existing file."
return answer
return json.dumps(answer)
class ListDirTool(Tool):
@@ -91,25 +86,35 @@ class ListDirTool(Tool):
Lists files and directories in the given directory (optionally with recursion).
"""
def apply(self, relative_path: str, recursive: bool, max_answer_chars: int = TOOL_DEFAULT_MAX_ANSWER_LENGTH) -> str:
def apply(self, relative_path: str, recursive: bool, skip_ignored_files: bool = False, max_answer_chars: int = -1) -> str:
"""
Lists all non-gitignored files and directories in the given directory (optionally with recursion).
Lists files and directories in the given directory (optionally with recursion).
:param relative_path: the relative path to the directory to list; pass "." to scan the project root
:param recursive: whether to scan subdirectories recursively
:param skip_ignored_files: whether to skip files and directories that are ignored
:param max_answer_chars: if the output is longer than this number of characters,
no content will be returned. Don't adjust unless there is really no other way to get the content
required for the task.
no content will be returned. -1 means the default value from the config will be used.
Don't adjust unless there is really no other way to get the content required for the task.
:return: a JSON object with the names of directories and files within the given directory
"""
self.project.validate_relative_path(relative_path)
# Check if the directory exists before validation
if not self.project.relative_path_exists(relative_path):
error_info = {
"error": f"Directory not found: {relative_path}",
"project_root": self.get_project_root(),
"hint": "Check if the path is correct relative to the project root",
}
return json.dumps(error_info)
self.project.validate_relative_path(relative_path, require_not_ignored=skip_ignored_files)
dirs, files = scan_directory(
os.path.join(self.get_project_root(), relative_path),
relative_to=self.get_project_root(),
recursive=recursive,
is_ignored_dir=self.project.is_ignored_path,
is_ignored_file=self.project.is_ignored_path,
is_ignored_dir=self.project.is_ignored_path if skip_ignored_files else None,
is_ignored_file=self.project.is_ignored_path if skip_ignored_files else None,
)
result = json.dumps({"dirs": dirs, "files": files})
@@ -129,7 +134,7 @@ class FindFileTool(Tool):
:param relative_path: the relative path to the directory to search in; pass "." to scan the project root
:return: a JSON object with the list of matching files
"""
self.project.validate_relative_path(relative_path)
self.project.validate_relative_path(relative_path, require_not_ignored=True)
dir_to_scan = os.path.join(self.get_project_root(), relative_path)
@@ -140,7 +145,7 @@ class FindFileTool(Tool):
filename = os.path.basename(abs_path)
return not fnmatch(filename, file_mask)
dirs, files = scan_directory(
_dirs, files = scan_directory(
path=dir_to_scan,
recursive=True,
is_ignored_dir=self.project.is_ignored_path,
@@ -173,19 +178,21 @@ class ReplaceRegexTool(Tool, ToolMarkerCanEdit):
Always try to use wildcards to avoid specifying the exact content of the code to be replaced,
especially if it spans several lines.
IMPORTANT: REMEMBER TO USE WILDCARDS WHEN APPROPRIATE! I WILL BE VERY UNHAPPY IF YOU WRITE LONG REGEXES WITHOUT USING WILDCARDS INSTEAD!
IMPORTANT: REMEMBER TO USE WILDCARDS WHEN APPROPRIATE! I WILL BE VERY UNHAPPY IF YOU WRITE UNNECESSARILY LONG REGEXES WITHOUT USING WILDCARDS!
:param relative_path: the relative path to the file
:param regex: a Python-style regular expression, matches of which will be replaced.
Dot matches all characters, multi-line matching is enabled.
:param repl: the string to replace the matched content with, which may contain
backreferences like \1, \2, etc.
IMPORTANT: Make sure to escape special characters appropriately!
Use "\n" to insert a newline, but use "\\n" to insert the string "\n" within a string literal.
:param allow_multiple_occurrences: if True, the regex may match multiple occurrences in the file
and all of them will be replaced.
If this is set to False and the regex matches multiple occurrences, an error will be returned
(and you may retry with a revised, more specific regex).
"""
self.project.validate_relative_path(relative_path)
self.project.validate_relative_path(relative_path, require_not_ignored=True)
with EditedFileContext(relative_path, self.agent) as context:
original_content = context.get_original_content()
updated_content, n = re.subn(regex, repl, original_content, flags=re.DOTALL | re.MULTILINE)
@@ -200,7 +207,7 @@ class ReplaceRegexTool(Tool, ToolMarkerCanEdit):
return SUCCESS_RESULT
class DeleteLinesTool(Tool, ToolMarkerCanEdit):
class DeleteLinesTool(Tool, ToolMarkerCanEdit, ToolMarkerOptional):
"""
Deletes a range of lines within a file.
"""
@@ -220,15 +227,12 @@ class DeleteLinesTool(Tool, ToolMarkerCanEdit):
:param start_line: the 0-based index of the first line to be deleted
:param end_line: the 0-based index of the last line to be deleted
"""
if not self.lines_read.were_lines_read(relative_path, (start_line, end_line)):
read_lines_tool = self.agent.get_tool(ReadFileTool)
return f"Error: Must call `{read_lines_tool.get_name_from_cls()}` first to read exactly the affected lines."
code_editor = self.create_code_editor()
code_editor.delete_lines(relative_path, start_line, end_line)
return SUCCESS_RESULT
class ReplaceLinesTool(Tool, ToolMarkerCanEdit):
class ReplaceLinesTool(Tool, ToolMarkerCanEdit, ToolMarkerOptional):
"""
Replaces a range of lines within a file with new content.
"""
@@ -259,7 +263,7 @@ class ReplaceLinesTool(Tool, ToolMarkerCanEdit):
return SUCCESS_RESULT
class InsertAtLineTool(Tool, ToolMarkerCanEdit):
class InsertAtLineTool(Tool, ToolMarkerCanEdit, ToolMarkerOptional):
"""
Inserts content at a given line in a file.
"""
@@ -297,11 +301,11 @@ class SearchForPatternTool(Tool):
substring_pattern: str,
context_lines_before: int = 0,
context_lines_after: int = 0,
paths_include_glob: str | None = None,
paths_exclude_glob: str | None = None,
paths_include_glob: str = "",
paths_exclude_glob: str = "",
relative_path: str = "",
restrict_search_to_code_files: bool = False,
max_answer_chars: int = TOOL_DEFAULT_MAX_ANSWER_LENGTH,
max_answer_chars: int = -1,
) -> str:
"""
Offers a flexible search for arbitrary patterns in the codebase, including the
@@ -335,14 +339,18 @@ class SearchForPatternTool(Tool):
:param context_lines_after: Number of lines of context to include after each match
:param paths_include_glob: optional glob pattern specifying files to include in the search.
Matches against relative file paths from the project root (e.g., "*.py", "src/**/*.ts").
Only matches files, not directories.
Supports standard glob patterns (*, ?, [seq], **, etc.) and brace expansion {a,b,c}.
Only matches files, not directories. If left empty, all non-ignored files will be included.
:param paths_exclude_glob: optional glob pattern specifying files to exclude from the search.
Matches against relative file paths from the project root (e.g., "*test*", "**/*_generated.py").
Takes precedence over paths_include_glob. Only matches files, not directories.
Supports standard glob patterns (*, ?, [seq], **, etc.) and brace expansion {a,b,c}.
Takes precedence over paths_include_glob. Only matches files, not directories. If left empty, no files are excluded.
:param relative_path: only subpaths of this path (relative to the repo root) will be analyzed. If a path to a single
file is passed, only that will be searched. The path must exist, otherwise a `FileNotFoundError` is raised.
:param max_answer_chars: if the output is longer than this number of characters,
no content will be returned. Don't adjust unless there is really no other way to get the content
no content will be returned.
-1 means the default value from the config will be used.
Don't adjust unless there is really no other way to get the content
required for the task. Instead, if the output is too long, you should
make a stricter query.
:param restrict_search_to_code_files: whether to restrict the search to only those files where
@@ -351,7 +359,7 @@ class SearchForPatternTool(Tool):
For example, for finding classes or methods from a name pattern.
Setting to False is a better choice if you also want to search in non-code files, like in html or yaml files,
which is why it is the default.
:return: A JSON object mapping file paths to lists of matched consecutive lines (with context, if requested).
:return: A mapping of file paths to lists of matched consecutive lines.
"""
abs_path = os.path.join(self.get_project_root(), relative_path)
if not os.path.exists(abs_path):
@@ -363,14 +371,14 @@ class SearchForPatternTool(Tool):
relative_path=relative_path,
context_lines_before=context_lines_before,
context_lines_after=context_lines_after,
paths_include_glob=paths_include_glob,
paths_exclude_glob=paths_exclude_glob,
paths_include_glob=paths_include_glob.strip(),
paths_exclude_glob=paths_exclude_glob.strip(),
)
else:
if os.path.isfile(abs_path):
rel_paths_to_search = [relative_path]
else:
dirs, rel_paths_to_search = scan_directory(
_dirs, rel_paths_to_search = scan_directory(
path=abs_path,
recursive=True,
is_ignored_dir=self.project.is_ignored_path,
@@ -382,6 +390,7 @@ class SearchForPatternTool(Tool):
matches = search_files(
rel_paths_to_search,
substring_pattern,
file_reader=self.project.read_file,
root_path=self.get_project_root(),
paths_include_glob=paths_include_glob,
paths_exclude_glob=paths_exclude_glob,
src/serena/tools/jetbrains_plugin_client.py
+31 -6
View File
@@ -8,6 +8,7 @@ from pathlib import Path
from typing import Any, Optional, Self, TypeVar
import requests
from requests import Response
from sensai.util.string import ToStringMixin
from serena.project import Project
@@ -78,6 +79,7 @@ class JetBrainsPluginClient(ToStringMixin):
def _make_request(self, method: str, endpoint: str, data: Optional[dict] = None) -> dict[str, Any]:
url = f"{self.base_url}{endpoint}"
response: Response | None = None
try:
if method.upper() == "GET":
response = self.session.get(url, timeout=self.timeout)
@@ -100,8 +102,10 @@ class JetBrainsPluginClient(ToStringMixin):
raise ConnectionError(f"Failed to connect to Serena service at {url}: {e}")
except requests.exceptions.Timeout as e:
raise ConnectionError(f"Request to {url} timed out: {e}")
except requests.exceptions.HTTPError:
raise APIError(f"API request failed with status {response.status_code}: {response.text}")
except requests.exceptions.HTTPError as e:
if response is not None:
raise APIError(f"API request failed with status {response.status_code}: {response.text}")
raise APIError(f"API request failed with HTTP error: {e}")
except requests.exceptions.RequestException as e:
raise SerenaClientError(f"Request failed: {e}")
@@ -132,7 +136,7 @@ class JetBrainsPluginClient(ToStringMixin):
self, name_path: str, relative_path: str | None = None, include_body: bool = False, depth: int = 0, include_location: bool = False
) -> dict[str, Any]:
"""
Find symbols by name.
Finds symbols by name.
:param name_path: the name path to match
:param relative_path: the relative path to which to restrict the search
@@ -152,7 +156,7 @@ class JetBrainsPluginClient(ToStringMixin):
def find_references(self, name_path: str, relative_path: str) -> dict[str, Any]:
"""
Find references to a symbol.
Finds references to a symbol.
:param name_path: the name path of the symbol
:param relative_path: the relative path
@@ -168,10 +172,31 @@ class JetBrainsPluginClient(ToStringMixin):
request_data = {"relativePath": relative_path}
return self._make_request("POST", "/getSymbolsOverview", request_data)
def rename_symbol(
self, name_path: str, relative_path: str, new_name: str, rename_in_comments: bool, rename_in_text_occurrences: bool
) -> None:
"""
Renames a symbol.
:param name_path: the name path of the symbol
:param relative_path: the relative path
:param new_name: the new name for the symbol
:param rename_in_comments: whether to rename in comments
:param rename_in_text_occurrences: whether to rename in text occurrences
"""
request_data = {
"namePath": name_path,
"relativePath": relative_path,
"newName": new_name,
"renameInComments": rename_in_comments,
"renameInTextOccurrences": rename_in_text_occurrences,
}
self._make_request("POST", "/renameSymbol", request_data)
def is_service_available(self) -> bool:
try:
response = self.heartbeat()
return response.get("status") == "OK"
self.project_root()
return True
except (ConnectionError, APIError):
return False
src/serena/tools/jetbrains_tools.py
+11 -8
View File
@@ -1,10 +1,10 @@
import json
from serena.tools import TOOL_DEFAULT_MAX_ANSWER_LENGTH, Tool, ToolMarkerOptional
from serena.tools import Tool, ToolMarkerOptional, ToolMarkerSymbolicRead
from serena.tools.jetbrains_plugin_client import JetBrainsPluginClient
class JetBrainsFindSymbolTool(Tool, ToolMarkerOptional):
class JetBrainsFindSymbolTool(Tool, ToolMarkerSymbolicRead, ToolMarkerOptional):
"""
Performs a global (or local) search for symbols with/containing a given name/substring (optionally filtered by type).
"""
@@ -15,7 +15,7 @@ class JetBrainsFindSymbolTool(Tool, ToolMarkerOptional):
depth: int = 0,
relative_path: str | None = None,
include_body: bool = False,
max_answer_chars: int = TOOL_DEFAULT_MAX_ANSWER_LENGTH,
max_answer_chars: int = -1,
) -> str:
"""
Retrieves information on all symbols/code entities (classes, methods, etc.) based on the given `name_path`,
@@ -57,6 +57,7 @@ class JetBrainsFindSymbolTool(Tool, ToolMarkerOptional):
speed up the search as well as reduce the number of results.
:param include_body: If True, include the symbol's source code. Use judiciously.
:param max_answer_chars: max characters for the JSON result. If exceeded, no content is returned.
-1 means the default value from the config will be used.
:return: JSON string: a list of symbols (with locations) matching the name.
"""
with JetBrainsPluginClient.from_project(self.project) as client:
@@ -70,7 +71,7 @@ class JetBrainsFindSymbolTool(Tool, ToolMarkerOptional):
return self._limit_length(result, max_answer_chars)
class JetBrainsFindReferencingSymbolsTool(Tool, ToolMarkerOptional):
class JetBrainsFindReferencingSymbolsTool(Tool, ToolMarkerSymbolicRead, ToolMarkerOptional):
"""
Finds symbols that reference the given symbol
"""
@@ -79,7 +80,7 @@ class JetBrainsFindReferencingSymbolsTool(Tool, ToolMarkerOptional):
self,
name_path: str,
relative_path: str,
max_answer_chars: int = TOOL_DEFAULT_MAX_ANSWER_LENGTH,
max_answer_chars: int = -1,
) -> str:
"""
Finds symbols that reference the symbol at the given `name_path`.
@@ -88,7 +89,8 @@ class JetBrainsFindReferencingSymbolsTool(Tool, ToolMarkerOptional):
:param name_path: name path of the symbol for which to find references; matching logic as described in find symbol tool.
:param relative_path: the relative path to the file containing the symbol for which to find references.
Note that here you can't pass a directory but must pass a file.
:param max_answer_chars: max characters for the JSON result. If exceeded, no content is returned.
:param max_answer_chars: max characters for the JSON result. If exceeded, no content is returned. -1 means the
default value from the config will be used.
:return: a list of JSON objects with the symbols referencing the requested symbol
"""
with JetBrainsPluginClient.from_project(self.project) as client:
@@ -100,7 +102,7 @@ class JetBrainsFindReferencingSymbolsTool(Tool, ToolMarkerOptional):
return self._limit_length(result, max_answer_chars)
class JetBrainsGetSymbolsOverviewTool(Tool, ToolMarkerOptional):
class JetBrainsGetSymbolsOverviewTool(Tool, ToolMarkerSymbolicRead, ToolMarkerOptional):
"""
Retrieves an overview of the top-level symbols within a specified file
"""
@@ -108,7 +110,7 @@ class JetBrainsGetSymbolsOverviewTool(Tool, ToolMarkerOptional):
def apply(
self,
relative_path: str,
max_answer_chars: int = TOOL_DEFAULT_MAX_ANSWER_LENGTH,
max_answer_chars: int = -1,
) -> str:
"""
Gets an overview of the top-level symbols in the given file.
@@ -119,6 +121,7 @@ class JetBrainsGetSymbolsOverviewTool(Tool, ToolMarkerOptional):
:param relative_path: the relative path to the file to get the overview of
:param max_answer_chars: max characters for the JSON result. If exceeded, no content is returned.
-1 means the default value from the config will be used.
:return: a JSON object containing the symbols
"""
with JetBrainsPluginClient.from_project(self.project) as client:
src/serena/tools/memory_tools.py
+8 -12
View File
@@ -1,6 +1,6 @@
import json
from serena.tools import TOOL_DEFAULT_MAX_ANSWER_LENGTH, Tool
from serena.tools import Tool
class WriteMemoryTool(Tool):
@@ -8,18 +8,14 @@ class WriteMemoryTool(Tool):
Writes a named memory (for future reference) to Serena's project-specific memory store.
"""
def apply(self, memory_name: str, content: str, max_answer_chars: int = TOOL_DEFAULT_MAX_ANSWER_LENGTH) -> str:
def apply(self, memory_name: str, content: str, max_answer_chars: int = -1) -> str:
"""
Write some information about this project that can be useful for future tasks to a memory.
Use markdown formatting for the content.
The information should be short and to the point.
The memory name should be meaningful, such that from the name you can infer what the information is about.
It is better to have multiple small memories than to have a single large one because
memories will be read one by one and we only ever want to read relevant memories.
This tool is either called during the onboarding process or when you have identified
something worth remembering about the project from the past conversation.
Write some information (utf-8-encoded) about this project that can be useful for future tasks to a memory in md format.
The memory name should be meaningful.
"""
# NOTE: utf-8 encoding is configured in the MemoriesManager
if max_answer_chars == -1:
max_answer_chars = self.agent.serena_config.default_max_tool_answer_chars
if len(content) > max_answer_chars:
raise ValueError(
f"Content for {memory_name} is too long. Max length is {max_answer_chars} characters. " + "Please make the content shorter."
@@ -33,7 +29,7 @@ class ReadMemoryTool(Tool):
Reads the memory with the given name from Serena's project-specific memory store.
"""
def apply(self, memory_file_name: str, max_answer_chars: int = TOOL_DEFAULT_MAX_ANSWER_LENGTH) -> str:
def apply(self, memory_file_name: str, max_answer_chars: int = -1) -> str:
"""
Read the content of a memory file. This tool should only be used if the information
is relevant to the current task. You can infer whether the information
src/serena/tools/symbol_tools.py
+87 -46
View File
@@ -4,11 +4,18 @@ Language server-related tools
import dataclasses
import json
import os
from collections.abc import Sequence
from copy import copy
from typing import Any
from serena.tools import SUCCESS_RESULT, TOOL_DEFAULT_MAX_ANSWER_LENGTH, Tool, ToolMarkerCanEdit
from serena.tools import (
SUCCESS_RESULT,
Tool,
ToolMarkerSymbolicEdit,
ToolMarkerSymbolicRead,
)
from serena.tools.tools_base import ToolMarkerOptional
from solidlsp.ls_types import SymbolKind
@@ -28,63 +35,64 @@ def _sanitize_symbol_dict(symbol_dict: dict[str, Any]) -> dict[str, Any]:
return symbol_dict
class RestartLanguageServerTool(Tool):
class RestartLanguageServerTool(Tool, ToolMarkerOptional):
"""Restarts the language server, may be necessary when edits not through Serena happen."""
def apply(self) -> str:
"""Use this tool only on explicit user request or after confirmation.
It may be necessary to restart the language server if the user performs edits
not through Serena, so the language server state becomes outdated and further editing attempts lead to errors.
If such editing errors happen, you should suggest using this tool.
It may be necessary to restart the language server if it hangs.
"""
self.agent.reset_language_server()
return SUCCESS_RESULT
class GetSymbolsOverviewTool(Tool):
class GetSymbolsOverviewTool(Tool, ToolMarkerSymbolicRead):
"""
Gets an overview of the top-level symbols defined in a given file or directory.
Gets an overview of the top-level symbols defined in a given file.
"""
def apply(self, relative_path: str, max_answer_chars: int = TOOL_DEFAULT_MAX_ANSWER_LENGTH) -> str:
def apply(self, relative_path: str, max_answer_chars: int = -1) -> str:
"""
Gets an overview of the given file or directory.
For each analyzed file, we list the top-level symbols in the file (name_path, kind).
Use this tool to get a high-level understanding of the code symbols.
Calling this is often a good idea before more targeted reading, searching or editing operations on the code symbols.
Before requesting a symbol overview, it is usually a good idea to narrow down the scope of the overview
by first understanding the basic directory structure of the repository that you can get from memories
or by using the `list_dir` and `find_file` tools (or similar).
Use this tool to get a high-level understanding of the code symbols in a file.
This should be the first tool to call when you want to understand a new file, unless you already know
what you are looking for.
:param relative_path: the relative path to the file or directory to get the overview of
:param relative_path: the relative path to the file to get the overview of
:param max_answer_chars: if the overview is longer than this number of characters,
no content will be returned. Don't adjust unless there is really no other way to get the content
required for the task. If the overview is too long, you should use a smaller directory instead,
(e.g. a subdirectory).
:return: a JSON object mapping relative paths of all contained files to info about top-level symbols in the file (name_path, kind).
no content will be returned. -1 means the default value from the config will be used.
Don't adjust unless there is really no other way to get the content required for the task.
:return: a JSON object containing info about top-level symbols in the file
"""
symbol_retriever = self.create_language_server_symbol_retriever()
result = symbol_retriever.get_symbol_overview(relative_path)
result_json_str = json.dumps({k: [dataclasses.asdict(i) for i in l] for k, l in result.items()})
file_path = os.path.join(self.project.project_root, relative_path)
# The symbol overview is capable of working with both files and directories,
# but we want to ensure that the user provides a file path.
if not os.path.exists(file_path):
raise FileNotFoundError(f"File or directory {relative_path} does not exist in the project.")
if os.path.isdir(file_path):
raise ValueError(f"Expected a file path, but got a directory path: {relative_path}. ")
result = symbol_retriever.get_symbol_overview(relative_path)[relative_path]
result_json_str = json.dumps([dataclasses.asdict(i) for i in result])
return self._limit_length(result_json_str, max_answer_chars)
class FindSymbolTool(Tool):
class FindSymbolTool(Tool, ToolMarkerSymbolicRead):
"""
Performs a global (or local) search for symbols with/containing a given name/substring (optionally filtered by type).
"""
# noinspection PyDefaultArgument
def apply(
self,
name_path: str,
depth: int = 0,
relative_path: str | None = None,
relative_path: str = "",
include_body: bool = False,
include_kinds: list[int] | None = None,
exclude_kinds: list[int] | None = None,
include_kinds: list[int] = [], # noqa: B006
exclude_kinds: list[int] = [], # noqa: B006
substring_matching: bool = False,
max_answer_chars: int = TOOL_DEFAULT_MAX_ANSWER_LENGTH,
max_answer_chars: int = -1,
) -> str:
"""
Retrieves information on all symbols/code entities (classes, methods, etc.) based on the given `name_path`,
@@ -128,11 +136,14 @@ class FindSymbolTool(Tool):
:param include_kinds: Optional. List of LSP symbol kind integers to include. (e.g., 5 for Class, 12 for Function).
Valid kinds: 1=file, 2=module, 3=namespace, 4=package, 5=class, 6=method, 7=property, 8=field, 9=constructor, 10=enum,
11=interface, 12=function, 13=variable, 14=constant, 15=string, 16=number, 17=boolean, 18=array, 19=object,
20=key, 21=null, 22=enum member, 23=struct, 24=event, 25=operator, 26=type parameter
20=key, 21=null, 22=enum member, 23=struct, 24=event, 25=operator, 26=type parameter.
If not provided, all kinds are included.
:param exclude_kinds: Optional. List of LSP symbol kind integers to exclude. Takes precedence over `include_kinds`.
If not provided, no kinds are excluded.
:param substring_matching: If True, use substring matching for the last segment of `name`.
:param max_answer_chars: Max characters for the JSON result. If exceeded, no content is returned.
:return: JSON string: a list of symbols (with locations) matching the name.
-1 means the default value from the config will be used.
:return: a list of symbols (with locations) matching the name.
"""
parsed_include_kinds: Sequence[SymbolKind] | None = [SymbolKind(k) for k in include_kinds] if include_kinds else None
parsed_exclude_kinds: Sequence[SymbolKind] | None = [SymbolKind(k) for k in exclude_kinds] if exclude_kinds else None
@@ -150,24 +161,23 @@ class FindSymbolTool(Tool):
return self._limit_length(result, max_answer_chars)
class FindReferencingSymbolsTool(Tool):
class FindReferencingSymbolsTool(Tool, ToolMarkerSymbolicRead):
"""
Finds symbols that reference the symbol at the given location (optionally filtered by type).
"""
# noinspection PyDefaultArgument
def apply(
self,
name_path: str,
relative_path: str,
include_kinds: list[int] | None = None,
exclude_kinds: list[int] | None = None,
max_answer_chars: int = TOOL_DEFAULT_MAX_ANSWER_LENGTH,
include_kinds: list[int] = [], # noqa: B006
exclude_kinds: list[int] = [], # noqa: B006
max_answer_chars: int = -1,
) -> str:
"""
Finds symbols that reference the symbol at the given `name_path`. The result will contain metadata about the referencing symbols
as well as a short code snippet around the reference (unless `include_body` is True, then the short snippet will be omitted).
Note that among other kinds of references, this function can be used to find (direct) subclasses of a class,
as subclasses are referencing symbols that have the kind class.
Finds references to the symbol at the given `name_path`. The result will contain metadata about the referencing symbols
as well as a short code snippet around the reference.
:param name_path: for finding the symbol to find references for, same logic as in the `find_symbol` tool.
:param relative_path: the relative path to the file containing the symbol for which to find references.
@@ -204,7 +214,7 @@ class FindReferencingSymbolsTool(Tool):
return self._limit_length(result, max_answer_chars)
class ReplaceSymbolBodyTool(Tool, ToolMarkerCanEdit):
class ReplaceSymbolBodyTool(Tool, ToolMarkerSymbolicEdit):
"""
Replaces the full definition of a symbol.
"""
@@ -218,10 +228,15 @@ class ReplaceSymbolBodyTool(Tool, ToolMarkerCanEdit):
r"""
Replaces the body of the symbol with the given `name_path`.
The tool shall be used to replace symbol bodies that have been previously retrieved
(e.g. via `find_symbol`).
IMPORTANT: Do not use this tool if you do not know what exactly constitutes the body of the symbol.
:param name_path: for finding the symbol to replace, same logic as in the `find_symbol` tool.
:param relative_path: the relative path to the file containing the symbol
:param body: the new symbol body. Important: Begin directly with the symbol definition and provide no
leading indentation for the first line (but do indent the rest of the body according to the context).
:param body: the new symbol body. The symbol body is the definition of a symbol
in the programming language, including e.g. the signature line for functions.
IMPORTANT: The body does NOT include any preceding docstrings/comments or imports, in particular.
"""
code_editor = self.create_code_editor()
code_editor.replace_body(
@@ -232,7 +247,7 @@ class ReplaceSymbolBodyTool(Tool, ToolMarkerCanEdit):
return SUCCESS_RESULT
class InsertAfterSymbolTool(Tool, ToolMarkerCanEdit):
class InsertAfterSymbolTool(Tool, ToolMarkerSymbolicEdit):
"""
Inserts content after the end of the definition of a given symbol.
"""
@@ -257,7 +272,7 @@ class InsertAfterSymbolTool(Tool, ToolMarkerCanEdit):
return SUCCESS_RESULT
class InsertBeforeSymbolTool(Tool, ToolMarkerCanEdit):
class InsertBeforeSymbolTool(Tool, ToolMarkerSymbolicEdit):
"""
Inserts content before the beginning of the definition of a given symbol.
"""
@@ -269,9 +284,9 @@ class InsertBeforeSymbolTool(Tool, ToolMarkerCanEdit):
body: str,
) -> str:
"""
Inserts the given body/content before the beginning of the definition of the given symbol (via the symbol's location).
A typical use case is to insert a new class, function, method, field or variable assignment.
It also can be used to insert a new import statement before the first symbol in the file.
Inserts the given content before the beginning of the definition of the given symbol (via the symbol's location).
A typical use case is to insert a new class, function, method, field or variable assignment; or
a new import statement before the first symbol in the file.
:param name_path: name path of the symbol before which to insert content (definitions in the `find_symbol` tool apply)
:param relative_path: the relative path to the file containing the symbol
@@ -280,3 +295,29 @@ class InsertBeforeSymbolTool(Tool, ToolMarkerCanEdit):
code_editor = self.create_code_editor()
code_editor.insert_before_symbol(name_path, relative_file_path=relative_path, body=body)
return SUCCESS_RESULT
class RenameSymbolTool(Tool, ToolMarkerSymbolicEdit):
"""
Renames a symbol throughout the codebase using language server refactoring capabilities.
"""
def apply(
self,
name_path: str,
relative_path: str,
new_name: str,
) -> str:
"""
Renames the symbol with the given `name_path` to `new_name` throughout the entire codebase.
Note: for languages with method overloading, like Java, name_path may have to include a method's
signature to uniquely identify a method.
:param name_path: name path of the symbol to rename (definitions in the `find_symbol` tool apply)
:param relative_path: the relative path to the file containing the symbol to rename
:param new_name: the new name for the symbol
:return: result summary indicating success or failure
"""
code_editor = self.create_code_editor()
status_message = code_editor.rename_symbol(name_path, relative_file_path=relative_path, new_name=new_name)
return status_message
src/serena/tools/tools_base.py
+63 -20
View File
@@ -10,7 +10,7 @@ from mcp.server.fastmcp.utilities.func_metadata import FuncMetadata, func_metada
from sensai.util import logging
from sensai.util.string import dict_string
from serena.project import Project
from serena.project import MemoriesManager, Project
from serena.prompt_factory import PromptFactory
from serena.symbol import LanguageServerSymbolRetriever
from serena.util.class_decorators import singleton
@@ -18,7 +18,7 @@ from serena.util.inspection import iter_subclasses
from solidlsp.ls_exceptions import SolidLSPException
if TYPE_CHECKING:
from serena.agent import LinesRead, MemoriesManager, SerenaAgent
from serena.agent import SerenaAgent
from serena.code_editor import CodeEditor
log = logging.getLogger(__name__)
@@ -42,8 +42,7 @@ class Component(ABC):
@property
def memories_manager(self) -> "MemoriesManager":
assert self.agent.memories_manager is not None
return self.agent.memories_manager
return self.project.memories_manager
def create_language_server_symbol_retriever(self) -> LanguageServerSymbolRetriever:
if not self.agent.is_using_language_server():
@@ -64,31 +63,41 @@ class Component(ABC):
else:
return JetBrainsCodeEditor(project=self.project, agent=self.agent)
@property
def lines_read(self) -> "LinesRead":
assert self.agent.lines_read is not None
return self.agent.lines_read
class ToolMarker:
"""
Base class for tool markers.
"""
TOOL_DEFAULT_MAX_ANSWER_LENGTH = int(2e5)
class ToolMarkerCanEdit:
class ToolMarkerCanEdit(ToolMarker):
"""
Marker class for all tools that can perform editing operations on files.
"""
class ToolMarkerDoesNotRequireActiveProject:
class ToolMarkerDoesNotRequireActiveProject(ToolMarker):
pass
class ToolMarkerOptional:
class ToolMarkerOptional(ToolMarker):
"""
Marker class for optional tools that are disabled by default.
"""
class ToolMarkerSymbolicRead(ToolMarker):
"""
Marker class for tools that perform symbol read operations.
"""
class ToolMarkerSymbolicEdit(ToolMarkerCanEdit):
"""
Marker class for tools that perform symbolic edit operations.
"""
class ApplyMethodProtocol(Protocol):
"""Callable protocol for the apply method of a tool."""
@@ -196,8 +205,11 @@ class Tool(Component):
params[param] = value
log.info(f"{self.get_name_from_cls()}: {dict_string(params)}")
@staticmethod
def _limit_length(result: str, max_answer_chars: int) -> str:
def _limit_length(self, result: str, max_answer_chars: int) -> str:
if max_answer_chars == -1:
max_answer_chars = self.agent.serena_config.default_max_tool_answer_chars
if max_answer_chars <= 0:
raise ValueError(f"Must be positive or the default (-1), got: {max_answer_chars=}")
if (n_chars := len(result)) > max_answer_chars:
result = (
f"The answer is too long ({n_chars} characters). "
@@ -229,7 +241,7 @@ class Tool(Component):
if not isinstance(self, ToolMarkerDoesNotRequireActiveProject):
if self.agent._active_project is None:
return (
"Error: No active project. Ask to user to select a project from this list: "
"Error: No active project. Ask the user to provide the project path or to select a project from this list of known projects: "
+ f"{self.agent.serena_config.project_names}"
)
if self.agent.is_using_language_server() and not self.agent.is_language_server_running():
@@ -345,18 +357,49 @@ class ToolRegistry:
def get_all_tool_classes(self) -> list[type[Tool]]:
return list(t.tool_class for t in self._tool_dict.values())
def get_tool_classes_default_enabled(self) -> list[type[Tool]]:
"""
:return: the list of tool classes that are enabled by default (i.e. non-optional tools).
"""
return [t.tool_class for t in self._tool_dict.values() if not t.is_optional]
def get_tool_classes_optional(self) -> list[type[Tool]]:
"""
:return: the list of tool classes that are optional (i.e. disabled by default).
"""
return [t.tool_class for t in self._tool_dict.values() if t.is_optional]
def get_tool_names_default_enabled(self) -> list[str]:
"""
:return: the list of tool names that are enabled by default (i.e. non-optional tools).
"""
return [t.tool_name for t in self._tool_dict.values() if not t.is_optional]
def print_tool_overview(self, tools: Iterable[type[Tool] | Tool] | None = None) -> None:
def get_tool_names_optional(self) -> list[str]:
"""
Print a summary of the tools. If no tools are passed, a summary of all tools is printed.
:return: the list of tool names that are optional (i.e. disabled by default).
"""
return [t.tool_name for t in self._tool_dict.values() if t.is_optional]
def get_tool_names(self) -> list[str]:
"""
:return: the list of all tool names.
"""
return list(self._tool_dict.keys())
def print_tool_overview(
self, tools: Iterable[type[Tool] | Tool] | None = None, include_optional: bool = False, only_optional: bool = False
) -> None:
"""
Print a summary of the tools. If no tools are passed, a summary of the selection of tools (all, default or only optional) is printed.
"""
if tools is None:
tools = [tool.tool_class for tool in self._tool_dict.values() if not tool.is_optional]
if only_optional:
tools = self.get_tool_classes_optional()
elif include_optional:
tools = self.get_all_tool_classes()
else:
tools = self.get_tool_classes_default_enabled()
tool_dict: dict[str, type[Tool] | Tool] = {}
for tool_class in tools:
src/serena/tools/workflow_tools.py
+3 -3
View File
@@ -5,7 +5,7 @@ Tools supporting the general workflow of the agent
import json
import platform
from serena.tools import Tool, ToolMarkerDoesNotRequireActiveProject
from serena.tools import Tool, ToolMarkerDoesNotRequireActiveProject, ToolMarkerOptional
class CheckOnboardingPerformedTool(Tool):
@@ -96,7 +96,7 @@ class ThinkAboutWhetherYouAreDoneTool(Tool):
return self.prompt_factory.create_think_about_whether_you_are_done()
class SummarizeChangesTool(Tool):
class SummarizeChangesTool(Tool, ToolMarkerOptional):
"""
Provides instructions for summarizing the changes made to the codebase.
"""
@@ -122,7 +122,7 @@ class PrepareForNewConversationTool(Tool):
return self.prompt_factory.create_prepare_for_new_conversation()
class InitialInstructionsTool(Tool, ToolMarkerDoesNotRequireActiveProject):
class InitialInstructionsTool(Tool, ToolMarkerDoesNotRequireActiveProject, ToolMarkerOptional):
"""
Gets the initial instructions for the current project.
Should only be used in settings where the system prompt cannot be set,
src/serena/util/file_system.py
+70 -38
View File
@@ -1,13 +1,13 @@
import glob
import logging
import os
from collections.abc import Callable
from collections.abc import Callable, Iterator
from dataclasses import dataclass, field
from pathlib import Path
from typing import NamedTuple
import pathspec
from pathspec import PathSpec
from sensai.util.logging import LogTime
log = logging.getLogger(__name__)
@@ -23,8 +23,8 @@ def scan_directory(
path: str,
recursive: bool = False,
relative_to: str | None = None,
is_ignored_dir: Callable[[str], bool] = lambda x: False,
is_ignored_file: Callable[[str], bool] = lambda x: False,
is_ignored_dir: Callable[[str], bool] | None = None,
is_ignored_file: Callable[[str], bool] | None = None,
) -> ScanResult:
"""
:param path: the path to scan
@@ -34,37 +34,52 @@ def scan_directory(
:param is_ignored_file: a function with which to determine whether the given file (abs. path) shall be ignored
:return: the list of directories and files
"""
if is_ignored_file is None:
is_ignored_file = lambda x: False
if is_ignored_dir is None:
is_ignored_dir = lambda x: False
files = []
directories = []
abs_path = os.path.abspath(path)
rel_base = os.path.abspath(relative_to) if relative_to else None
with os.scandir(abs_path) as entries:
for entry in entries:
entry_path = entry.path
try:
with os.scandir(abs_path) as entries:
for entry in entries:
try:
entry_path = entry.path
if rel_base:
result_path = os.path.relpath(entry_path, rel_base)
else:
result_path = entry_path
if rel_base:
result_path = os.path.relpath(entry_path, rel_base)
else:
result_path = entry_path
if entry.is_file():
if not is_ignored_file(entry_path):
files.append(result_path)
elif entry.is_dir():
if not is_ignored_dir(entry_path):
directories.append(result_path)
if recursive:
sub_result = scan_directory(
entry_path,
recursive=True,
relative_to=relative_to,
is_ignored_dir=is_ignored_dir,
is_ignored_file=is_ignored_file,
)
files.extend(sub_result.files)
directories.extend(sub_result.directories)
if entry.is_file():
if not is_ignored_file(entry_path):
files.append(result_path)
elif entry.is_dir():
if not is_ignored_dir(entry_path):
directories.append(result_path)
if recursive:
sub_result = scan_directory(
entry_path,
recursive=True,
relative_to=relative_to,
is_ignored_dir=is_ignored_dir,
is_ignored_file=is_ignored_file,
)
files.extend(sub_result.files)
directories.extend(sub_result.directories)
except PermissionError as ex:
# Skip files/directories that cannot be accessed due to permission issues
log.debug(f"Skipping entry due to permission error: {entry.path}", exc_info=ex)
continue
except PermissionError as ex:
# Skip the entire directory if it cannot be accessed
log.debug(f"Skipping directory due to permission error: {abs_path}", exc_info=ex)
return ScanResult([], [])
return ScanResult(directories, files)
@@ -128,20 +143,36 @@ class GitignoreParser:
def _load_gitignore_files(self) -> None:
"""Load all gitignore files from the repository."""
gitignore_files = self._find_gitignore_files()
with LogTime("Loading of .gitignore files", logger=log):
for gitignore_path in self._iter_gitignore_files():
log.info("Processing .gitignore file: %s", gitignore_path)
spec = self._create_ignore_spec(gitignore_path)
if spec.patterns: # Only add non-empty specs
self.ignore_specs.append(spec)
for gitignore_file in gitignore_files:
spec = self._create_ignore_spec(gitignore_file)
if spec.patterns: # Only add non-empty specs
self.ignore_specs.append(spec)
def _find_gitignore_files(self) -> list[str]:
def _iter_gitignore_files(self, follow_symlinks: bool = False) -> Iterator[str]:
"""
Find all .gitignore files in the repository.
Iteratively discover .gitignore files in a top-down fashion, starting from the repository root.
Directory paths are skipped if they match any already loaded ignore patterns.
:return: List of absolute paths to .gitignore files
:return: an iterator yielding paths to .gitignore files (top-down)
"""
return glob.glob(self.repo_root + "/.gitignore") + glob.glob(self.repo_root + "/**/.gitignore")
queue: list[str] = [self.repo_root]
def scan(abs_path: str | None) -> Iterator[str]:
for entry in os.scandir(abs_path):
if entry.is_dir(follow_symlinks=follow_symlinks):
queue.append(entry.path)
elif entry.is_file(follow_symlinks=follow_symlinks) and entry.name == ".gitignore":
yield entry.path
while queue:
next_abs_path = queue.pop(0)
if next_abs_path != self.repo_root:
rel_path = os.path.relpath(next_abs_path, self.repo_root)
if self.should_ignore(rel_path):
continue
yield from scan(next_abs_path)
def _create_ignore_spec(self, gitignore_file_path: str) -> GitignoreSpec:
"""
@@ -214,7 +245,8 @@ class GitignoreParser:
# Non-anchored patterns can match anywhere below the gitignore directory
# We need to preserve this behavior
if line.startswith("**/"):
adjusted_pattern = line
# Even if pattern starts with **, it should still be scoped to the subdirectory
adjusted_pattern = os.path.join(rel_dir, line)
else:
# Add the directory prefix but also allow matching in subdirectories
adjusted_pattern = os.path.join(rel_dir, "**", line)
src/serena/util/general.py
+4 -2
View File
@@ -4,6 +4,8 @@ from typing import Literal, overload
from ruamel.yaml import YAML
from ruamel.yaml.comments import CommentedMap
from serena.constants import SERENA_FILE_ENCODING
def _create_YAML(preserve_comments: bool = False) -> YAML:
"""
@@ -20,7 +22,7 @@ def load_yaml(path: str, preserve_comments: Literal[False]) -> dict: ...
@overload
def load_yaml(path: str, preserve_comments: Literal[True]) -> CommentedMap: ...
def load_yaml(path: str, preserve_comments: bool = False) -> dict | CommentedMap:
with open(path, encoding="utf-8") as f:
with open(path, encoding=SERENA_FILE_ENCODING) as f:
yaml = _create_YAML(preserve_comments)
return yaml.load(f)
@@ -28,5 +30,5 @@ def load_yaml(path: str, preserve_comments: bool = False) -> dict | CommentedMap
def save_yaml(path: str, data: dict | CommentedMap, preserve_comments: bool = False) -> None:
yaml = _create_YAML(preserve_comments)
os.makedirs(os.path.dirname(path), exist_ok=True)
with open(path, "w", encoding="utf-8") as f:
with open(path, "w", encoding=SERENA_FILE_ENCODING) as f:
yaml.dump(data, f)
src/serena/util/shell.py
+5 -13
View File
@@ -1,9 +1,10 @@
import os
import platform
import subprocess
from pydantic import BaseModel
from solidlsp.util.subprocess_util import subprocess_kwargs
class ShellCommandResult(BaseModel):
stdout: str
@@ -24,18 +25,17 @@ def execute_shell_command(command: str, cwd: str | None = None, capture_stderr:
if cwd is None:
cwd = os.getcwd()
is_windows = platform.system() == "Windows"
process = subprocess.Popen(
command,
shell=not is_windows,
shell=True,
stdin=subprocess.DEVNULL,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE if capture_stderr else None,
creationflags=subprocess.CREATE_NO_WINDOW if is_windows else 0, # type: ignore
text=True,
encoding="utf-8",
errors="replace",
cwd=cwd,
**subprocess_kwargs(),
)
stdout, stderr = process.communicate()
@@ -43,15 +43,7 @@ def execute_shell_command(command: str, cwd: str | None = None, capture_stderr:
def subprocess_check_output(args: list[str], encoding: str = "utf-8", strip: bool = True, timeout: float | None = None) -> str:
kwargs = {
"stdin": subprocess.DEVNULL,
"stderr": subprocess.PIPE,
"timeout": timeout,
"env": os.environ.copy(),
}
if platform.system() == "Windows":
kwargs["creationflags"] = subprocess.CREATE_NO_WINDOW # type: ignore
output = subprocess.check_output(args, **kwargs).decode(encoding) # type: ignore
output = subprocess.check_output(args, stdin=subprocess.DEVNULL, stderr=subprocess.PIPE, timeout=timeout, env=os.environ.copy(), **subprocess_kwargs()).decode(encoding) # type: ignore
if strip:
output = output.strip()
return output
src/solidlsp/language_servers/al_language_server.py
+969
View File
@@ -0,0 +1,969 @@
"""AL Language Server implementation for Microsoft Dynamics 365 Business Central."""
import logging
import os
import pathlib
import platform
import stat
import time
import zipfile
from pathlib import Path
import requests
from overrides import override
from solidlsp.language_servers.common import quote_windows_path
from solidlsp.ls import SolidLanguageServer
from solidlsp.ls_config import LanguageServerConfig
from solidlsp.ls_logger import LanguageServerLogger
from solidlsp.lsp_protocol_handler.lsp_types import Definition, DefinitionParams, LocationLink
from solidlsp.lsp_protocol_handler.server import ProcessLaunchInfo
from solidlsp.settings import SolidLSPSettings
class ALLanguageServer(SolidLanguageServer):
"""
Language server implementation for AL (Microsoft Dynamics 365 Business Central).
This implementation uses the AL Language Server from the VS Code AL extension
(ms-dynamics-smb.al). The extension must be installed or available locally.
Key Features:
- Automatic download of AL extension from VS Code marketplace if not present
- Platform-specific executable detection (Windows/Linux/macOS)
- Special initialization sequence required by AL Language Server
- Custom AL-specific LSP commands (al/gotodefinition, al/setActiveWorkspace)
- File opening requirement before symbol retrieval
"""
def __init__(
self, config: LanguageServerConfig, logger: LanguageServerLogger, repository_root_path: str, solidlsp_settings: SolidLSPSettings
):
"""
Initialize the AL Language Server.
Args:
config: Language server configuration
logger: Logger instance for debugging
repository_root_path: Root path of the AL project (must contain app.json)
solidlsp_settings: Solid LSP settings
Note:
The initialization process will automatically:
1. Check for AL extension in the resources directory
2. Download it from VS Code marketplace if not found
3. Extract and configure the platform-specific executable
"""
# Setup runtime dependencies and get the language server command
# This will download the AL extension if needed
cmd = self._setup_runtime_dependencies(logger, config, solidlsp_settings)
self._project_load_check_supported: bool = True
"""Whether the AL server supports the project load status check request.
Some AL server versions don't support the 'al/hasProjectClosureLoadedRequest'
custom LSP request. This flag starts as True and is set to False if the
request fails, preventing repeated unsuccessful attempts.
"""
super().__init__(
config,
logger,
repository_root_path,
ProcessLaunchInfo(cmd=cmd, cwd=repository_root_path),
"al", # Language ID for LSP
solidlsp_settings,
)
@classmethod
def _download_al_extension(cls, logger: LanguageServerLogger, url: str, target_dir: str) -> bool:
"""
Download and extract the AL extension from VS Code marketplace.
The VS Code marketplace packages extensions as .vsix files (which are ZIP archives).
This method downloads the VSIX file and extracts it to get the language server binaries.
Args:
logger: Logger for tracking download progress
url: VS Code marketplace URL for the AL extension
target_dir: Directory where the extension will be extracted
Returns:
True if successful, False otherwise
Note:
The download includes progress tracking and proper user-agent headers
to ensure compatibility with the VS Code marketplace.
"""
try:
logger.log(f"Downloading AL extension from {url}", logging.INFO)
# Create target directory for the extension
os.makedirs(target_dir, exist_ok=True)
# Download with proper headers to mimic VS Code marketplace client
# These headers are required for the marketplace to serve the VSIX file
headers = {
"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36",
"Accept": "application/octet-stream, application/vsix, */*",
}
response = requests.get(url, headers=headers, stream=True, timeout=300)
response.raise_for_status()
# Save to temporary VSIX file (will be deleted after extraction)
temp_file = os.path.join(target_dir, "al_extension_temp.vsix")
total_size = int(response.headers.get("content-length", 0))
logger.log(f"Downloading {total_size / 1024 / 1024:.1f} MB...", logging.INFO)
with open(temp_file, "wb") as f:
downloaded = 0
for chunk in response.iter_content(chunk_size=8192):
if chunk:
f.write(chunk)
downloaded += len(chunk)
if total_size > 0 and downloaded % (10 * 1024 * 1024) == 0: # Log progress every 10MB
progress = (downloaded / total_size) * 100
logger.log(f"Download progress: {progress:.1f}%", logging.INFO)
logger.log("Download complete, extracting...", logging.INFO)
# Extract VSIX file (VSIX files are just ZIP archives with a different extension)
# This will extract the extension folder containing the language server binaries
with zipfile.ZipFile(temp_file, "r") as zip_ref:
zip_ref.extractall(target_dir)
# Clean up temp file
os.remove(temp_file)
logger.log("AL extension extracted successfully", logging.INFO)
return True
except Exception as e:
logger.log(f"Error downloading/extracting AL extension: {e}", logging.ERROR)
return False
@classmethod
def _setup_runtime_dependencies(
cls, logger: LanguageServerLogger, config: LanguageServerConfig, solidlsp_settings: SolidLSPSettings
) -> str:
"""
Setup runtime dependencies for AL Language Server and return the command to start the server.
This method handles the complete setup process:
1. Checks for existing AL extension installations
2. Downloads from VS Code marketplace if not found
3. Configures executable permissions on Unix systems
4. Returns the properly formatted command string
The AL Language Server executable is located in different paths based on the platform:
- Windows: bin/win32/Microsoft.Dynamics.Nav.EditorServices.Host.exe
- Linux: bin/linux/Microsoft.Dynamics.Nav.EditorServices.Host
- macOS: bin/darwin/Microsoft.Dynamics.Nav.EditorServices.Host
"""
system = platform.system()
# Find existing extension or download if needed
extension_path = cls._find_al_extension(logger, solidlsp_settings)
if extension_path is None:
logger.log("AL extension not found on disk, attempting to download...", logging.INFO)
extension_path = cls._download_and_install_al_extension(logger, solidlsp_settings)
if extension_path is None:
raise RuntimeError(
"Failed to locate or download AL Language Server. Please either:\n"
"1. Set AL_EXTENSION_PATH environment variable to the AL extension directory\n"
"2. Install the AL extension in VS Code (ms-dynamics-smb.al)\n"
"3. Ensure internet connection for automatic download"
)
# Build executable path based on platform
executable_path = cls._get_executable_path(extension_path, system)
if not os.path.exists(executable_path):
raise RuntimeError(f"AL Language Server executable not found at: {executable_path}")
# Prepare and return the executable command
return cls._prepare_executable(executable_path, system, logger)
@classmethod
def _find_al_extension(cls, logger: LanguageServerLogger, solidlsp_settings: SolidLSPSettings) -> str | None:
"""
Find AL extension in various locations.
Search order:
1. Environment variable (AL_EXTENSION_PATH)
2. Default download location (~/.serena/ls_resources/al-extension)
3. VS Code installed extensions
Returns:
Path to AL extension directory or None if not found
"""
# Check environment variable
env_path = os.environ.get("AL_EXTENSION_PATH")
if env_path and os.path.exists(env_path):
logger.log(f"Found AL extension via AL_EXTENSION_PATH: {env_path}", logging.DEBUG)
return env_path
elif env_path:
logger.log(f"AL_EXTENSION_PATH set but directory not found: {env_path}", logging.WARNING)
# Check default download location
default_path = os.path.join(cls.ls_resources_dir(solidlsp_settings), "al-extension", "extension")
if os.path.exists(default_path):
logger.log(f"Found AL extension in default location: {default_path}", logging.DEBUG)
return default_path
# Search VS Code extensions
vscode_path = cls._find_al_extension_in_vscode(logger)
if vscode_path:
logger.log(f"Found AL extension in VS Code: {vscode_path}", logging.DEBUG)
return vscode_path
logger.log("AL extension not found in any known location", logging.DEBUG)
return None
@classmethod
def _download_and_install_al_extension(cls, logger: LanguageServerLogger, solidlsp_settings: SolidLSPSettings) -> str | None:
"""
Download and install AL extension from VS Code marketplace.
Returns:
Path to installed extension or None if download failed
"""
al_extension_dir = os.path.join(cls.ls_resources_dir(solidlsp_settings), "al-extension")
# AL extension version - using latest stable version
AL_VERSION = "latest"
url = f"https://marketplace.visualstudio.com/_apis/public/gallery/publishers/ms-dynamics-smb/vsextensions/al/{AL_VERSION}/vspackage"
logger.log(f"Downloading AL extension from: {url}", logging.INFO)
if cls._download_al_extension(logger, url, al_extension_dir):
extension_path = os.path.join(al_extension_dir, "extension")
if os.path.exists(extension_path):
logger.log("AL extension downloaded and installed successfully", logging.INFO)
return extension_path
else:
logger.log(f"Download completed but extension not found at: {extension_path}", logging.ERROR)
else:
logger.log("Failed to download AL extension from marketplace", logging.ERROR)
return None
@classmethod
def _get_executable_path(cls, extension_path: str, system: str) -> str:
"""
Build platform-specific executable path.
Args:
extension_path: Path to AL extension directory
system: Operating system name
Returns:
Full path to executable
"""
if system == "Windows":
return os.path.join(extension_path, "bin", "win32", "Microsoft.Dynamics.Nav.EditorServices.Host.exe")
elif system == "Linux":
return os.path.join(extension_path, "bin", "linux", "Microsoft.Dynamics.Nav.EditorServices.Host")
elif system == "Darwin":
return os.path.join(extension_path, "bin", "darwin", "Microsoft.Dynamics.Nav.EditorServices.Host")
else:
raise RuntimeError(f"Unsupported platform: {system}")
@classmethod
def _prepare_executable(cls, executable_path: str, system: str, logger: LanguageServerLogger) -> str:
"""
Prepare the executable by setting permissions and handling path quoting.
Args:
executable_path: Path to the executable
system: Operating system name
logger: Logger instance
Returns:
Properly formatted command string
"""
# Make sure executable has proper permissions on Unix-like systems
if system in ["Linux", "Darwin"]:
st = os.stat(executable_path)
os.chmod(executable_path, st.st_mode | stat.S_IEXEC)
logger.log(f"Set execute permission on: {executable_path}", logging.DEBUG)
logger.log(f"Using AL Language Server executable: {executable_path}", logging.INFO)
# The AL Language Server uses stdio for LSP communication by default
# Use the utility function to handle Windows path quoting
return quote_windows_path(executable_path)
@classmethod
def _get_language_server_command_fallback(cls, logger: LanguageServerLogger) -> str:
"""
Get the command to start the AL language server.
Returns:
Command string to launch the AL language server
Raises:
RuntimeError: If AL extension cannot be found
"""
# Check if AL extension path is configured via environment variable
al_extension_path = os.environ.get("AL_EXTENSION_PATH")
if not al_extension_path:
# Try to find the extension in the current working directory
# (for development/testing when extension is in the serena repo)
cwd_path = Path.cwd()
potential_extension = None
# Look for ms-dynamics-smb.al-* directories
for item in cwd_path.iterdir():
if item.is_dir() and item.name.startswith("ms-dynamics-smb.al-"):
potential_extension = item
break
if potential_extension:
al_extension_path = str(potential_extension)
logger.log(f"Found AL extension in current directory: {al_extension_path}", logging.DEBUG)
else:
# Try to find in common VS Code extension locations
al_extension_path = cls._find_al_extension_in_vscode(logger)
if not al_extension_path:
raise RuntimeError(
"AL Language Server not found. Please either:\n"
"1. Set AL_EXTENSION_PATH environment variable to the VS Code AL extension directory\n"
"2. Install the AL extension in VS Code (ms-dynamics-smb.al)\n"
"3. Place the extension directory in the current working directory"
)
# Determine platform-specific executable
system = platform.system()
if system == "Windows":
executable = os.path.join(al_extension_path, "bin", "win32", "Microsoft.Dynamics.Nav.EditorServices.Host.exe")
elif system == "Linux":
executable = os.path.join(al_extension_path, "bin", "linux", "Microsoft.Dynamics.Nav.EditorServices.Host")
elif system == "Darwin":
executable = os.path.join(al_extension_path, "bin", "darwin", "Microsoft.Dynamics.Nav.EditorServices.Host")
else:
raise RuntimeError(f"Unsupported platform: {system}")
# Verify executable exists
if not os.path.exists(executable):
raise RuntimeError(
f"AL Language Server executable not found at: {executable}\nPlease ensure the AL extension is properly installed."
)
# Make sure executable has proper permissions on Unix-like systems
if system in ["Linux", "Darwin"]:
st = os.stat(executable)
os.chmod(executable, st.st_mode | stat.S_IEXEC)
logger.log(f"Using AL Language Server executable: {executable}", logging.INFO)
# The AL Language Server uses stdio for LSP communication (no --stdio flag needed)
# Use the utility function to handle Windows path quoting
return quote_windows_path(executable)
@classmethod
def _find_al_extension_in_vscode(cls, logger: LanguageServerLogger) -> str | None:
"""
Try to find AL extension in common VS Code extension locations.
Returns:
Path to AL extension directory or None if not found
"""
home = Path.home()
possible_paths = []
# Common VS Code extension paths
if platform.system() == "Windows":
possible_paths.extend(
[
home / ".vscode" / "extensions",
home / ".vscode-insiders" / "extensions",
Path(os.environ.get("APPDATA", "")) / "Code" / "User" / "extensions",
Path(os.environ.get("APPDATA", "")) / "Code - Insiders" / "User" / "extensions",
]
)
else:
possible_paths.extend(
[
home / ".vscode" / "extensions",
home / ".vscode-server" / "extensions",
home / ".vscode-insiders" / "extensions",
]
)
for base_path in possible_paths:
if base_path.exists():
logger.log(f"Searching for AL extension in: {base_path}", logging.DEBUG)
# Look for AL extension directories
for item in base_path.iterdir():
if item.is_dir() and item.name.startswith("ms-dynamics-smb.al-"):
logger.log(f"Found AL extension at: {item}", logging.DEBUG)
return str(item)
return None
@staticmethod
def _get_initialize_params(repository_absolute_path: str) -> dict:
"""
Returns the initialize params for the AL Language Server.
"""
# Ensure we have an absolute path for URI generation
repository_path = pathlib.Path(repository_absolute_path).resolve()
root_uri = repository_path.as_uri()
# AL requires extensive capabilities based on VS Code trace
initialize_params = {
"processId": os.getpid(),
"rootPath": str(repository_path),
"rootUri": root_uri,
"capabilities": {
"workspace": {
"applyEdit": True,
"workspaceEdit": {
"documentChanges": True,
"resourceOperations": ["create", "rename", "delete"],
"failureHandling": "textOnlyTransactional",
"normalizesLineEndings": True,
},
"configuration": True,
"didChangeWatchedFiles": {"dynamicRegistration": True},
"symbol": {"dynamicRegistration": True, "symbolKind": {"valueSet": list(range(1, 27))}},
"executeCommand": {"dynamicRegistration": True},
"didChangeConfiguration": {"dynamicRegistration": True},
"workspaceFolders": True,
},
"textDocument": {
"synchronization": {"dynamicRegistration": True, "willSave": True, "willSaveWaitUntil": True, "didSave": True},
"completion": {
"dynamicRegistration": True,
"contextSupport": True,
"completionItem": {
"snippetSupport": True,
"commitCharactersSupport": True,
"documentationFormat": ["markdown", "plaintext"],
"deprecatedSupport": True,
"preselectSupport": True,
},
},
"hover": {"dynamicRegistration": True, "contentFormat": ["markdown", "plaintext"]},
"definition": {"dynamicRegistration": True, "linkSupport": True},
"references": {"dynamicRegistration": True},
"documentHighlight": {"dynamicRegistration": True},
"documentSymbol": {
"dynamicRegistration": True,
"symbolKind": {"valueSet": list(range(1, 27))},
"hierarchicalDocumentSymbolSupport": True,
},
"codeAction": {"dynamicRegistration": True},
"formatting": {"dynamicRegistration": True},
"rangeFormatting": {"dynamicRegistration": True},
"rename": {"dynamicRegistration": True, "prepareSupport": True},
},
"window": {
"showMessage": {"messageActionItem": {"additionalPropertiesSupport": True}},
"showDocument": {"support": True},
"workDoneProgress": True,
},
},
"trace": "verbose",
"workspaceFolders": [{"uri": root_uri, "name": repository_path.name}],
}
return initialize_params
@override
def _start_server(self):
"""
Starts the AL Language Server process and initializes it.
This method sets up custom notification handlers for AL-specific messages
before starting the server. The AL server sends various notifications
during initialization and project loading that need to be handled.
"""
# Set up event handlers
def do_nothing(params):
return
def window_log_message(msg):
self.logger.log(f"AL LSP: window/logMessage: {msg}", logging.INFO)
def publish_diagnostics(params):
# AL server publishes diagnostics during initialization
uri = params.get("uri", "")
diagnostics = params.get("diagnostics", [])
self.logger.log(f"AL LSP: Diagnostics for {uri}: {len(diagnostics)} issues", logging.DEBUG)
def handle_al_notifications(params):
# AL server sends custom notifications during project loading
self.logger.log("AL LSP: Notification received", logging.DEBUG)
# Register handlers for AL-specific notifications
# These notifications are sent by the AL server during initialization and operation
self.server.on_notification("window/logMessage", window_log_message) # Server log messages
self.server.on_notification("textDocument/publishDiagnostics", publish_diagnostics) # Compilation diagnostics
self.server.on_notification("$/progress", do_nothing) # Progress notifications during loading
self.server.on_notification("al/refreshExplorerObjects", handle_al_notifications) # AL-specific object updates
# Start the server process
self.logger.log("Starting AL Language Server process", logging.INFO)
self.server.start()
# Send initialize request
initialize_params = self._get_initialize_params(self.repository_root_path)
self.logger.log(
"Sending initialize request from LSP client to AL LSP server and awaiting response",
logging.INFO,
)
# Send initialize and wait for response
resp = self.server.send_request("initialize", initialize_params)
if resp is None:
raise RuntimeError("AL Language Server initialization failed - no response")
self.logger.log("AL Language Server initialized successfully", logging.INFO)
# Send initialized notification
self.server.send_notification("initialized", {})
self.logger.log("Sent initialized notification", logging.INFO)
@override
def start(self) -> "ALLanguageServer":
"""
Start the AL Language Server with special initialization.
"""
# Call parent start method
super().start()
# AL-specific post-initialization
self._post_initialize_al_workspace()
# Note: set_active_workspace() can be called manually if needed for multi-workspace scenarios
# We don't call it automatically to avoid issues during single-workspace initialization
return self
def _post_initialize_al_workspace(self) -> None:
"""
Post-initialization setup for AL Language Server.
The AL server requires additional setup after initialization:
1. Send workspace configuration - provides AL settings and paths
2. Open app.json to trigger project loading - AL uses app.json to identify project structure
3. Optionally wait for project to be loaded if supported
This special initialization sequence is unique to AL and necessary for proper
symbol resolution and navigation features.
"""
# No sleep needed - server is already initialized
# Send workspace configuration first
# This tells AL about assembly paths, package caches, and code analysis settings
try:
self.server.send_notification(
"workspace/didChangeConfiguration",
{
"settings": {
"workspacePath": self.repository_root_path,
"alResourceConfigurationSettings": {
"assemblyProbingPaths": ["./.netpackages"],
"codeAnalyzers": [],
"enableCodeAnalysis": False,
"backgroundCodeAnalysis": "Project",
"packageCachePaths": ["./.alpackages"],
"ruleSetPath": None,
"enableCodeActions": True,
"incrementalBuild": False,
"outputAnalyzerStatistics": True,
"enableExternalRulesets": True,
},
"setActiveWorkspace": True,
"expectedProjectReferenceDefinitions": [],
"activeWorkspaceClosure": [self.repository_root_path],
}
},
)
self.logger.log("Sent workspace configuration", logging.DEBUG)
except Exception as e:
self.logger.log(f"Failed to send workspace config: {e}", logging.WARNING)
# Check if app.json exists and open it
# app.json is the AL project manifest file (similar to package.json for Node.js)
# Opening it triggers AL to load the project and index all AL files
app_json_path = Path(self.repository_root_path) / "app.json"
if app_json_path.exists():
try:
with open(app_json_path, encoding="utf-8") as f:
app_json_content = f.read()
# Use forward slashes for URI
app_json_uri = app_json_path.as_uri()
# Send textDocument/didOpen for app.json
self.server.send_notification(
"textDocument/didOpen",
{"textDocument": {"uri": app_json_uri, "languageId": "json", "version": 1, "text": app_json_content}},
)
self.logger.log(f"Opened app.json: {app_json_uri}", logging.DEBUG)
except Exception as e:
self.logger.log(f"Failed to open app.json: {e}", logging.WARNING)
# Try to set active workspace (AL-specific custom LSP request)
# This is optional and may not be supported by all AL server versions
workspace_uri = Path(self.repository_root_path).resolve().as_uri()
try:
result = self.server.send_request(
"al/setActiveWorkspace",
{
"currentWorkspaceFolderPath": {"uri": workspace_uri, "name": Path(self.repository_root_path).name, "index": 0},
"settings": {
"workspacePath": self.repository_root_path,
"setActiveWorkspace": True,
},
},
timeout=2, # Quick timeout since this is optional
)
self.logger.log(f"Set active workspace result: {result}", logging.DEBUG)
except Exception as e:
# This is a custom AL request, not critical if it fails
self.logger.log(f"Failed to set active workspace (non-critical): {e}", logging.DEBUG)
# Check if project supports load status check (optional)
# Many AL server versions don't support this, so we use a short timeout
# and continue regardless of the result
self._wait_for_project_load(timeout=3)
@override
def is_ignored_dirname(self, dirname: str) -> bool:
"""
Define AL-specific directories to ignore during file scanning.
These directories contain generated files, dependencies, or cache data
that should not be analyzed for symbols.
Args:
dirname: Directory name to check
Returns:
True if directory should be ignored
"""
al_ignore_dirs = {
".alpackages", # AL package cache - downloaded dependencies
".alcache", # AL compiler cache - intermediate compilation files
".altemplates", # AL templates - code generation templates
".snapshots", # Test snapshots - test result snapshots
"out", # Compiled output - generated .app files
".vscode", # VS Code settings - editor configuration
"Reference", # Reference assemblies - .NET dependencies
".netpackages", # .NET packages - NuGet packages for AL
"bin", # Binary output - compiled binaries
"obj", # Object files - intermediate build artifacts
}
# Check parent class ignore list first, then AL-specific
return super().is_ignored_dirname(dirname) or dirname in al_ignore_dirs
@override
def request_full_symbol_tree(self, within_relative_path: str | None = None, include_body: bool = False) -> list[dict]:
"""
Override to handle AL's requirement of opening files before requesting symbols.
The AL Language Server requires files to be explicitly opened via textDocument/didOpen
before it can provide meaningful symbols. Without this, it only returns directory symbols.
This is different from most language servers which can provide symbols for unopened files.
This method:
1. Scans the repository for all AL files (.al and .dal extensions)
2. Opens each file with the AL server
3. Requests symbols for each file
4. Combines all symbols into a hierarchical tree structure
5. Closes the files to free resources
Args:
within_relative_path: Restrict search to this file or directory path
include_body: Whether to include symbol body content
Returns:
Full symbol tree with all AL symbols from opened files organized by directory
"""
self.logger.log("AL: Starting request_full_symbol_tree with file opening", logging.DEBUG)
# Determine the root path for scanning
if within_relative_path is not None:
within_abs_path = os.path.join(self.repository_root_path, within_relative_path)
if not os.path.exists(within_abs_path):
raise FileNotFoundError(f"File or directory not found: {within_abs_path}")
if os.path.isfile(within_abs_path):
# Single file case - use parent class implementation
_, root_nodes = self.request_document_symbols(within_relative_path, include_body=include_body)
return root_nodes
# Directory case - scan within this directory
scan_root = Path(within_abs_path)
else:
# Scan entire repository
scan_root = Path(self.repository_root_path)
# For AL, we always need to open files to get symbols
al_files = []
# Walk through the repository to find all AL files
for root, dirs, files in os.walk(scan_root):
# Skip ignored directories
dirs[:] = [d for d in dirs if not self.is_ignored_dirname(d)]
# Find AL files
for file in files:
if file.endswith((".al", ".dal")):
file_path = Path(root) / file
# Use forward slashes for consistent paths
try:
relative_path = str(file_path.relative_to(self.repository_root_path)).replace("\\", "/")
al_files.append((file_path, relative_path))
except ValueError:
# File is outside repository root, skip it
continue
self.logger.log(f"AL: Found {len(al_files)} AL files", logging.DEBUG)
if not al_files:
self.logger.log("AL: No AL files found in repository", logging.WARNING)
return []
# Collect all symbols from all files
all_file_symbols = []
for file_path, relative_path in al_files:
try:
# Use our overridden request_document_symbols which handles opening
self.logger.log(f"AL: Getting symbols for {relative_path}", logging.DEBUG)
all_syms, root_syms = self.request_document_symbols(relative_path, include_body=include_body)
if root_syms:
# Create a file-level symbol containing the document symbols
file_symbol = {
"name": file_path.stem, # Just the filename without extension
"kind": 1, # File
"children": root_syms,
"location": {
"uri": file_path.as_uri(),
"relativePath": relative_path,
"absolutePath": str(file_path),
"range": {"start": {"line": 0, "character": 0}, "end": {"line": 0, "character": 0}},
},
}
all_file_symbols.append(file_symbol)
self.logger.log(f"AL: Added {len(root_syms)} symbols from {relative_path}", logging.DEBUG)
elif all_syms:
# If we only got all_syms but not root, use all_syms
file_symbol = {
"name": file_path.stem,
"kind": 1, # File
"children": all_syms,
"location": {
"uri": file_path.as_uri(),
"relativePath": relative_path,
"absolutePath": str(file_path),
"range": {"start": {"line": 0, "character": 0}, "end": {"line": 0, "character": 0}},
},
}
all_file_symbols.append(file_symbol)
self.logger.log(f"AL: Added {len(all_syms)} symbols from {relative_path}", logging.DEBUG)
except Exception as e:
self.logger.log(f"AL: Failed to get symbols for {relative_path}: {e}", logging.WARNING)
if all_file_symbols:
self.logger.log(f"AL: Returning symbols from {len(all_file_symbols)} files", logging.DEBUG)
# Group files by directory
directory_structure = {}
for file_symbol in all_file_symbols:
rel_path = file_symbol["location"]["relativePath"]
path_parts = rel_path.split("/")
if len(path_parts) > 1:
# File is in a subdirectory
dir_path = "/".join(path_parts[:-1])
if dir_path not in directory_structure:
directory_structure[dir_path] = []
directory_structure[dir_path].append(file_symbol)
else:
# File is in root
if "." not in directory_structure:
directory_structure["."] = []
directory_structure["."].append(file_symbol)
# Build hierarchical structure
result = []
repo_path = Path(self.repository_root_path)
for dir_path, file_symbols in directory_structure.items():
if dir_path == ".":
# Root level files
result.extend(file_symbols)
else:
# Create directory symbol
dir_symbol = {
"name": Path(dir_path).name,
"kind": 4, # Package/Directory
"children": file_symbols,
"location": {
"relativePath": dir_path,
"absolutePath": str(repo_path / dir_path),
"range": {"start": {"line": 0, "character": 0}, "end": {"line": 0, "character": 0}},
},
}
result.append(dir_symbol)
return result
else:
self.logger.log("AL: No symbols found in any files", logging.WARNING)
return []
# ===== Phase 1: Custom AL Command Implementations =====
@override
def _send_definition_request(self, definition_params: DefinitionParams) -> Definition | list[LocationLink] | None:
"""
Override to use AL's custom gotodefinition command.
AL Language Server uses 'al/gotodefinition' instead of the standard
'textDocument/definition' request. This custom command provides better
navigation for AL-specific constructs like table extensions, page extensions,
and codeunit references.
If the custom command fails, we fall back to the standard LSP method.
"""
# Convert standard params to AL format (same structure, different method)
al_params = {"textDocument": definition_params["textDocument"], "position": definition_params["position"]}
try:
# Use custom AL command instead of standard LSP
response = self.server.send_request("al/gotodefinition", al_params)
self.logger.log(f"AL gotodefinition response: {response}", logging.DEBUG)
return response
except Exception as e:
self.logger.log(f"Failed to use al/gotodefinition, falling back to standard: {e}", logging.WARNING)
# Fallback to standard LSP method if custom command fails
return super()._send_definition_request(definition_params)
def check_project_loaded(self) -> bool:
"""
Check if AL project closure is fully loaded.
Uses AL's custom 'al/hasProjectClosureLoadedRequest' to determine if
the project and all its dependencies have been fully loaded and indexed.
This is important because AL operations may fail or return incomplete
results if the project is still loading.
Returns:
bool: True if project is loaded, False otherwise
"""
if not hasattr(self, "server") or not self.server_started:
self.logger.log("Cannot check project load - server not started", logging.DEBUG)
return False
# Check if we've already determined this request isn't supported
if not self._project_load_check_supported:
return True # Assume loaded if check isn't supported
try:
# Use a very short timeout since this is just a status check
response = self.server.send_request("al/hasProjectClosureLoadedRequest", {}, timeout=1)
# Response can be boolean directly, dict with 'loaded' field, or None
if isinstance(response, bool):
return response
elif isinstance(response, dict):
return response.get("loaded", False)
elif response is None:
# None typically means the project is still loading
self.logger.log("Project load check returned None", logging.DEBUG)
return False
else:
self.logger.log(f"Unexpected response type for project load check: {type(response)}", logging.DEBUG)
return False
except Exception as e:
# Mark as unsupported to avoid repeated failed attempts
self._project_load_check_supported = False
self.logger.log(f"Project load check not supported by this AL server version: {e}", logging.DEBUG)
# Assume loaded if we can't check
return True
def _wait_for_project_load(self, timeout: int = 3) -> bool:
"""
Wait for project to be fully loaded.
Polls the AL server to check if the project is loaded.
This is optional as not all AL server versions support this check.
We use a short timeout and continue regardless of the result.
Args:
timeout: Maximum time to wait in seconds (default 3s)
Returns:
bool: True if project loaded within timeout, False otherwise
"""
start_time = time.time()
self.logger.log(f"Checking AL project load status (timeout: {timeout}s)...", logging.DEBUG)
while time.time() - start_time < timeout:
if self.check_project_loaded():
elapsed = time.time() - start_time
self.logger.log(f"AL project fully loaded after {elapsed:.1f}s", logging.INFO)
return True
time.sleep(0.5)
self.logger.log(f"Project load check timed out after {timeout}s (non-critical)", logging.DEBUG)
return False
def set_active_workspace(self, workspace_uri: str | None = None) -> None:
"""
Set the active AL workspace.
This is important when multiple workspaces exist to ensure operations
target the correct workspace. The AL server can handle multiple projects
simultaneously, but only one can be "active" at a time for operations
like symbol search and navigation.
This uses the custom 'al/setActiveWorkspace' LSP command.
Args:
workspace_uri: URI of workspace to set as active, or None to use repository root
"""
if not hasattr(self, "server") or not self.server_started:
self.logger.log("Cannot set active workspace - server not started", logging.DEBUG)
return
if workspace_uri is None:
workspace_uri = Path(self.repository_root_path).resolve().as_uri()
params = {"workspaceUri": workspace_uri}
try:
self.server.send_request("al/setActiveWorkspace", params)
self.logger.log(f"Set active workspace to: {workspace_uri}", logging.INFO)
except Exception as e:
self.logger.log(f"Failed to set active workspace: {e}", logging.WARNING)
# Non-critical error, continue operation
src/solidlsp/language_servers/bash_language_server.py
+230
View File
@@ -0,0 +1,230 @@
"""
Provides Bash specific instantiation of the LanguageServer class using bash-language-server.
Contains various configurations and settings specific to Bash scripting.
"""
import logging
import os
import pathlib
import shutil
import threading
from solidlsp import ls_types
from solidlsp.language_servers.common import RuntimeDependency, RuntimeDependencyCollection
from solidlsp.ls import SolidLanguageServer
from solidlsp.ls_config import LanguageServerConfig
from solidlsp.ls_logger import LanguageServerLogger
from solidlsp.lsp_protocol_handler.lsp_types import InitializeParams
from solidlsp.lsp_protocol_handler.server import ProcessLaunchInfo
from solidlsp.settings import SolidLSPSettings
class BashLanguageServer(SolidLanguageServer):
"""
Provides Bash specific instantiation of the LanguageServer class using bash-language-server.
Contains various configurations and settings specific to Bash scripting.
"""
def __init__(
self, config: LanguageServerConfig, logger: LanguageServerLogger, repository_root_path: str, solidlsp_settings: SolidLSPSettings
):
"""
Creates a BashLanguageServer instance. This class is not meant to be instantiated directly.
Use LanguageServer.create() instead.
"""
bash_lsp_executable_path = self._setup_runtime_dependencies(logger, config, solidlsp_settings)
super().__init__(
config,
logger,
repository_root_path,
ProcessLaunchInfo(cmd=bash_lsp_executable_path, cwd=repository_root_path),
"bash",
solidlsp_settings,
)
self.server_ready = threading.Event()
self.initialize_searcher_command_available = threading.Event()
@classmethod
def _setup_runtime_dependencies(
cls, logger: LanguageServerLogger, config: LanguageServerConfig, solidlsp_settings: SolidLSPSettings
) -> str:
"""
Setup runtime dependencies for Bash Language Server and return the command to start the server.
"""
# Verify both node and npm are installed
is_node_installed = shutil.which("node") is not None
assert is_node_installed, "node is not installed or isn't in PATH. Please install NodeJS and try again."
is_npm_installed = shutil.which("npm") is not None
assert is_npm_installed, "npm is not installed or isn't in PATH. Please install npm and try again."
deps = RuntimeDependencyCollection(
[
RuntimeDependency(
id="bash-language-server",
description="bash-language-server package",
command="npm install --prefix ./ bash-language-server@5.6.0",
platform_id="any",
),
]
)
# Install bash-language-server if not already installed
bash_ls_dir = os.path.join(cls.ls_resources_dir(solidlsp_settings), "bash-lsp")
bash_executable_path = os.path.join(bash_ls_dir, "node_modules", ".bin", "bash-language-server")
# Handle Windows executable extension
if os.name == "nt":
bash_executable_path += ".cmd"
if not os.path.exists(bash_executable_path):
logger.log(f"Bash Language Server executable not found at {bash_executable_path}. Installing...", logging.INFO)
deps.install(logger, bash_ls_dir)
logger.log("Bash language server dependencies installed successfully", logging.INFO)
if not os.path.exists(bash_executable_path):
raise FileNotFoundError(
f"bash-language-server executable not found at {bash_executable_path}, something went wrong with the installation."
)
return f"{bash_executable_path} start"
@staticmethod
def _get_initialize_params(repository_absolute_path: str) -> InitializeParams:
"""
Returns the initialize params for the Bash Language Server.
"""
root_uri = pathlib.Path(repository_absolute_path).as_uri()
initialize_params = {
"locale": "en",
"capabilities": {
"textDocument": {
"synchronization": {"didSave": True, "dynamicRegistration": True},
"completion": {"dynamicRegistration": True, "completionItem": {"snippetSupport": True}},
"definition": {"dynamicRegistration": True},
"references": {"dynamicRegistration": True},
"documentSymbol": {
"dynamicRegistration": True,
"hierarchicalDocumentSymbolSupport": True,
"symbolKind": {"valueSet": list(range(1, 27))},
},
"hover": {"dynamicRegistration": True, "contentFormat": ["markdown", "plaintext"]},
"signatureHelp": {"dynamicRegistration": True},
"codeAction": {"dynamicRegistration": True},
},
"workspace": {
"workspaceFolders": True,
"didChangeConfiguration": {"dynamicRegistration": True},
"symbol": {"dynamicRegistration": True},
},
},
"processId": os.getpid(),
"rootPath": repository_absolute_path,
"rootUri": root_uri,
"workspaceFolders": [
{
"uri": root_uri,
"name": os.path.basename(repository_absolute_path),
}
],
}
return initialize_params
def _start_server(self):
"""
Starts the Bash Language Server, waits for the server to be ready and yields the LanguageServer instance.
"""
def register_capability_handler(params):
assert "registrations" in params
for registration in params["registrations"]:
if registration["method"] == "workspace/executeCommand":
self.initialize_searcher_command_available.set()
return
def execute_client_command_handler(params):
return []
def do_nothing(params):
return
def window_log_message(msg):
self.logger.log(f"LSP: window/logMessage: {msg}", logging.INFO)
# Check for bash-language-server ready signals
message_text = msg.get("message", "")
if "Analyzing" in message_text or "analysis complete" in message_text.lower():
self.logger.log("Bash language server analysis signals detected", logging.INFO)
self.server_ready.set()
self.completions_available.set()
self.server.on_request("client/registerCapability", register_capability_handler)
self.server.on_notification("window/logMessage", window_log_message)
self.server.on_request("workspace/executeClientCommand", execute_client_command_handler)
self.server.on_notification("$/progress", do_nothing)
self.server.on_notification("textDocument/publishDiagnostics", do_nothing)
self.logger.log("Starting Bash server process", logging.INFO)
self.server.start()
initialize_params = self._get_initialize_params(self.repository_root_path)
self.logger.log(
"Sending initialize request from LSP client to LSP server and awaiting response",
logging.INFO,
)
init_response = self.server.send.initialize(initialize_params)
self.logger.log(f"Received initialize response from bash server: {init_response}", logging.DEBUG)
# Enhanced capability checks for bash-language-server 5.6.0
assert init_response["capabilities"]["textDocumentSync"] in [1, 2] # Full or Incremental
assert "completionProvider" in init_response["capabilities"]
# Verify document symbol support is available
if "documentSymbolProvider" in init_response["capabilities"]:
self.logger.log("Bash server supports document symbols", logging.INFO)
else:
self.logger.log("Warning: Bash server does not report document symbol support", logging.WARNING)
self.server.notify.initialized({})
# Wait for server readiness with timeout
self.logger.log("Waiting for Bash language server to be ready...", logging.INFO)
if not self.server_ready.wait(timeout=3.0):
# Fallback: assume server is ready after timeout
self.logger.log("Timeout waiting for bash server ready signal, proceeding anyway", logging.WARNING)
self.server_ready.set()
self.completions_available.set()
else:
self.logger.log("Bash server initialization complete", logging.INFO)
def request_document_symbols(
self, relative_file_path: str, include_body: bool = False
) -> tuple[list[ls_types.UnifiedSymbolInformation], list[ls_types.UnifiedSymbolInformation]]:
"""
Request document symbols from bash-language-server via LSP.
Uses the standard LSP documentSymbol request which provides reliable function detection
for all bash function syntaxes including:
- function name() { ... } (with function keyword)
- name() { ... } (traditional syntax)
- Functions with various indentation levels
- Functions with comments before/after/inside
Args:
relative_file_path: Path to the bash file relative to repository root
include_body: Whether to include function bodies in symbol information
Returns:
Tuple of (all_symbols, root_symbols) detected by the LSP server
"""
self.logger.log(f"Requesting document symbols via LSP for {relative_file_path}", logging.DEBUG)
# Use the standard LSP approach - bash-language-server handles all function syntaxes correctly
all_symbols, root_symbols = super().request_document_symbols(relative_file_path, include_body)
# Log detection results for debugging
functions = [s for s in all_symbols if s.get("kind") == 12]
self.logger.log(
f"LSP function detection for {relative_file_path}: Found {len(functions)} functions",
logging.INFO,
)
return all_symbols, root_symbols
src/solidlsp/language_servers/clangd_language_server.py
+10 -3
View File
@@ -5,7 +5,6 @@ Provides C/C++ specific instantiation of the LanguageServer class. Contains vari
import logging
import os
import pathlib
import stat
import threading
from typing import Any, cast
@@ -71,6 +70,14 @@ class ClangdLanguageServer(SolidLanguageServer):
archive_type="zip",
binary_name="clangd_19.1.2/bin/clangd.exe",
),
RuntimeDependency(
id="Clangd",
description="Clangd for macOS (x64)",
url="https://github.com/clangd/clangd/releases/download/19.1.2/clangd-mac-19.1.2.zip",
platform_id="osx-x64",
archive_type="zip",
binary_name="clangd_19.1.2/bin/clangd",
),
RuntimeDependency(
id="Clangd",
description="Clangd for macOS (Arm64)",
@@ -83,7 +90,7 @@ class ClangdLanguageServer(SolidLanguageServer):
)
clangd_ls_dir = os.path.join(cls.ls_resources_dir(solidlsp_settings), "clangd")
dep = deps.single_for_current_platform()
dep = deps.get_single_dep_for_current_platform()
clangd_executable_path = deps.binary_path(clangd_ls_dir)
if not os.path.exists(clangd_executable_path):
logger.log(
@@ -96,7 +103,7 @@ class ClangdLanguageServer(SolidLanguageServer):
f"Clangd executable not found at {clangd_executable_path}.\n"
"Make sure you have installed clangd. See https://clangd.llvm.org/installation"
)
os.chmod(clangd_executable_path, stat.S_IEXEC)
os.chmod(clangd_executable_path, 0o755)
return clangd_executable_path
src/solidlsp/language_servers/clojure_lsp.py
+9 -3
View File
@@ -6,7 +6,6 @@ import logging
import os
import pathlib
import shutil
import stat
import subprocess
import threading
from typing import cast
@@ -56,6 +55,13 @@ class ClojureLSP(SolidLanguageServer):
archive_type="zip",
binary_name="clojure-lsp",
),
RuntimeDependency(
id="clojure-lsp",
url=f"{clojure_lsp_releases}/clojure-lsp-native-macos-amd64.zip",
platform_id="osx-x64",
archive_type="zip",
binary_name="clojure-lsp",
),
RuntimeDependency(
id="clojure-lsp",
url=f"{clojure_lsp_releases}/clojure-lsp-native-linux-aarch64.zip",
@@ -107,7 +113,7 @@ class ClojureLSP(SolidLanguageServer):
"""Setup runtime dependencies for clojure-lsp and return the command to start the server."""
verify_clojure_cli()
deps = ClojureLSP.runtime_dependencies
dependency = deps.single_for_current_platform()
dependency = deps.get_single_dep_for_current_platform()
clojurelsp_ls_dir = cls.ls_resources_dir(solidlsp_settings)
clojurelsp_executable_path = deps.binary_path(clojurelsp_ls_dir)
@@ -119,7 +125,7 @@ class ClojureLSP(SolidLanguageServer):
deps.install(logger, clojurelsp_ls_dir)
if not os.path.exists(clojurelsp_executable_path):
raise FileNotFoundError(f"Download failed? Could not find clojure-lsp executable at {clojurelsp_executable_path}")
os.chmod(clojurelsp_executable_path, stat.S_IEXEC)
os.chmod(clojurelsp_executable_path, 0o755)
return clojurelsp_executable_path
@staticmethod
src/solidlsp/language_servers/common.py
+77 -21
View File
@@ -2,12 +2,15 @@ from __future__ import annotations
import logging
import os
import platform
import subprocess
from collections.abc import Sequence
from dataclasses import dataclass
from collections.abc import Iterable, Mapping, Sequence
from dataclasses import dataclass, replace
from typing import Any, cast
from solidlsp.ls_logger import LanguageServerLogger
from solidlsp.ls_utils import FileUtils, PlatformUtils
from solidlsp.util.subprocess_util import subprocess_kwargs
log = logging.getLogger(__name__)
@@ -21,7 +24,7 @@ class RuntimeDependency:
url: str | None = None
archive_type: str | None = None
binary_name: str | None = None
command: str | None = None
command: str | list[str] | None = None
package_name: str | None = None
package_version: str | None = None
extract_path: str | None = None
@@ -31,23 +34,47 @@ class RuntimeDependency:
class RuntimeDependencyCollection:
"""Utility to handle installation of runtime dependencies."""
def __init__(self, dependencies: Sequence[RuntimeDependency]):
self._dependencies = list(dependencies)
def __init__(self, dependencies: Sequence[RuntimeDependency], overrides: Iterable[Mapping[str, Any]] = ()) -> None:
"""Initialize the collection with a list of dependencies and optional overrides.
def for_platform(self, platform_id: str) -> list[RuntimeDependency]:
return [d for d in self._dependencies if d.platform_id in (platform_id, "any", "platform-agnostic", None)]
:param dependencies: List of base RuntimeDependency instances. The combination of 'id' and 'platform_id' must be unique.
:param overrides: List of dictionaries which represent overrides or additions to the base dependencies.
Each entry must contain at least the 'id' key, and optionally 'platform_id' to uniquely identify the dependency to override.
"""
self._id_and_platform_id_to_dep: dict[tuple[str, str | None], RuntimeDependency] = {}
for dep in dependencies:
dep_key = (dep.id, dep.platform_id)
if dep_key in self._id_and_platform_id_to_dep:
raise ValueError(f"Duplicate runtime dependency with id '{dep.id}' and platform_id '{dep.platform_id}':\n{dep}")
self._id_and_platform_id_to_dep[dep_key] = dep
def for_current_platform(self) -> list[RuntimeDependency]:
return self.for_platform(PlatformUtils.get_platform_id().value)
for dep_values_override in overrides:
override_key = cast(tuple[str, str | None], (dep_values_override["id"], dep_values_override.get("platform_id")))
base_dep = self._id_and_platform_id_to_dep.get(override_key)
if base_dep is None:
new_runtime_dep = RuntimeDependency(**dep_values_override)
self._id_and_platform_id_to_dep[override_key] = new_runtime_dep
else:
self._id_and_platform_id_to_dep[override_key] = replace(base_dep, **dep_values_override)
def single_for_current_platform(self) -> RuntimeDependency:
deps = self.for_current_platform()
def get_dependencies_for_platform(self, platform_id: str) -> list[RuntimeDependency]:
return [d for d in self._id_and_platform_id_to_dep.values() if d.platform_id in (platform_id, "any", "platform-agnostic", None)]
def get_dependencies_for_current_platform(self) -> list[RuntimeDependency]:
return self.get_dependencies_for_platform(PlatformUtils.get_platform_id().value)
def get_single_dep_for_current_platform(self, dependency_id: str | None = None) -> RuntimeDependency:
deps = self.get_dependencies_for_current_platform()
if dependency_id is not None:
deps = [d for d in deps if d.id == dependency_id]
if len(deps) != 1:
raise RuntimeError(f"Expected exactly one runtime dependency for {PlatformUtils.get_platform_id().value}, found {len(deps)}")
raise RuntimeError(
f"Expected exactly one runtime dependency for platform-{PlatformUtils.get_platform_id().value} and {dependency_id=}, found {len(deps)}"
)
return deps[0]
def binary_path(self, target_dir: str) -> str:
dep = self.single_for_current_platform()
dep = self.get_single_dep_for_current_platform()
if not dep.binary_name:
return target_dir
return os.path.join(target_dir, dep.binary_name)
@@ -59,7 +86,7 @@ class RuntimeDependencyCollection:
"""
os.makedirs(target_dir, exist_ok=True)
results: dict[str, str] = {}
for dep in self.for_current_platform():
for dep in self.get_dependencies_for_current_platform():
if dep.url:
self._install_from_url(dep, logger, target_dir)
if dep.command:
@@ -71,15 +98,21 @@ class RuntimeDependencyCollection:
return results
@staticmethod
def _run_command(command: str, cwd: str) -> None:
kwargs = {}
if PlatformUtils.get_platform_id().is_windows():
kwargs["creationflags"] = subprocess.CREATE_NO_WINDOW # type: ignore
else:
def _run_command(command: str | list[str], cwd: str) -> None:
kwargs = subprocess_kwargs()
if not PlatformUtils.get_platform_id().is_windows():
import pwd
kwargs["user"] = pwd.getpwuid(os.getuid()).pw_name # type: ignore
log.info("Running command '%s' in '%s'", command, cwd)
is_windows = platform.system() == "Windows"
if not isinstance(command, str) and not is_windows:
# Since we are using the shell, we need to convert the command list to a single string
# on Linux/macOS
command = " ".join(command)
log.info("Running command %s in '%s'", f"'{command}'" if isinstance(command, str) else command, cwd)
completed_process = subprocess.run(
command,
shell=True,
@@ -102,8 +135,31 @@ class RuntimeDependencyCollection:
if not dep.url:
raise ValueError(f"Dependency {dep.id} has no URL")
if dep.archive_type == "gz" and dep.binary_name:
if dep.archive_type in ("gz", "binary") and dep.binary_name:
dest = os.path.join(target_dir, dep.binary_name)
FileUtils.download_and_extract_archive(logger, dep.url, dest, dep.archive_type)
else:
FileUtils.download_and_extract_archive(logger, dep.url, target_dir, dep.archive_type or "zip")
def quote_windows_path(path: str) -> str:
"""
Quote a path for Windows command execution if needed.
On Windows, paths need to be quoted for proper command execution.
The function checks if the path is already quoted to avoid double-quoting.
On other platforms, the path is returned unchanged.
Args:
path: The file path to potentially quote
Returns:
The quoted path on Windows (if not already quoted), unchanged path on other platforms
"""
if platform.system() == "Windows":
# Check if already quoted to avoid double-quoting
if path.startswith('"') and path.endswith('"'):
return path
return f'"{path}"'
return path
src/solidlsp/language_servers/csharp_language_server.py
+83 -54
View File
@@ -7,7 +7,6 @@ import logging
import os
import platform
import shutil
import stat
import subprocess
import tarfile
import threading
@@ -27,11 +26,11 @@ from solidlsp.ls_utils import PathUtils
from solidlsp.lsp_protocol_handler.lsp_types import InitializeParams, InitializeResult
from solidlsp.lsp_protocol_handler.server import ProcessLaunchInfo
from solidlsp.settings import SolidLSPSettings
from solidlsp.util.zip import SafeZipExtractor
from .common import RuntimeDependency
from .common import RuntimeDependency, RuntimeDependencyCollection
# Runtime dependencies configuration
RUNTIME_DEPENDENCIES = [
_RUNTIME_DEPENDENCIES = [
RuntimeDependency(
id="CSharpLanguageServer",
description="Microsoft.CodeAnalysis.LanguageServer for Windows (x64)",
@@ -108,6 +107,14 @@ RUNTIME_DEPENDENCIES = [
archive_type="tar.gz",
binary_name="dotnet",
),
RuntimeDependency(
id="DotNetRuntime",
description=".NET 9 Runtime for Linux (ARM64)",
url="https://builds.dotnet.microsoft.com/dotnet/Runtime/9.0.6/dotnet-runtime-9.0.6-linux-arm64.tar.gz",
platform_id="linux-arm64",
archive_type="tar.gz",
binary_name="dotnet",
),
RuntimeDependency(
id="DotNetRuntime",
description=".NET 9 Runtime for macOS (x64)",
@@ -173,8 +180,22 @@ def find_solution_or_project_file(root_dir: str) -> str | None:
class CSharpLanguageServer(SolidLanguageServer):
"""
Provides C# specific instantiation of the LanguageServer class using Microsoft.CodeAnalysis.LanguageServer.
This is the official Roslyn-based language server from Microsoft.
Provides C# specific instantiation of the LanguageServer class using `Microsoft.CodeAnalysis.LanguageServer`,
the official Roslyn-based language server from Microsoft.
You can pass a list of runtime dependency overrides in ls_specific_settings["csharp"]. This is a list of
dicts, each containing at least the "id" key, and optionally "platform_id" to uniquely identify the dependency to override.
For example, to override the URL of the .NET runtime on windows-x64, add the entry:
```
{
"id": "DotNetRuntime",
"platform_id": "win-x64",
"url": "https://example.com/custom-dotnet-runtime.zip"
}
```
See the `_RUNTIME_DEPENDENCIES` variable above for the available dependency ids and platform_ids.
"""
def __init__(
@@ -227,52 +248,43 @@ class CSharpLanguageServer(SolidLanguageServer):
Ensure .NET runtime and Microsoft.CodeAnalysis.LanguageServer are available.
Returns a tuple of (dotnet_path, language_server_dll_path).
"""
runtime_id = CSharpLanguageServer._get_runtime_id()
lang_server_dep, dotnet_runtime_dep = CSharpLanguageServer._get_runtime_dependencies(runtime_id)
language_enum = cls.get_language_enum_instance()
ls_specific_settings = solidlsp_settings.ls_specific_settings or {}
language_specific_config = ls_specific_settings.get(language_enum, {})
runtime_dependency_overrides = cast(list[dict[str, Any]], language_specific_config.get("runtime_dependencies", []))
logger.log("Resolving runtime dependencies", logging.DEBUG)
if language_specific_config:
logger.log(
f"Language-specific config for {language_enum}: {language_specific_config}",
logging.DEBUG,
)
runtime_dependencies = RuntimeDependencyCollection(
_RUNTIME_DEPENDENCIES,
overrides=runtime_dependency_overrides,
)
logger.log(
f"Available runtime dependencies: {runtime_dependencies.get_dependencies_for_current_platform}",
logging.DEBUG,
)
# Find the dependencies for our platform
lang_server_dep = runtime_dependencies.get_single_dep_for_current_platform("CSharpLanguageServer")
dotnet_runtime_dep = runtime_dependencies.get_single_dep_for_current_platform("DotNetRuntime")
dotnet_path = CSharpLanguageServer._ensure_dotnet_runtime(logger, dotnet_runtime_dep, solidlsp_settings)
server_dll_path = CSharpLanguageServer._ensure_language_server(logger, lang_server_dep, solidlsp_settings)
return dotnet_path, server_dll_path
@staticmethod
def _get_runtime_id() -> str:
"""Determine the runtime ID based on the platform."""
system = platform.system().lower()
machine = platform.machine().lower()
if system == "windows":
return "win-x64" if machine in ["amd64", "x86_64"] else "win-arm64"
elif system == "darwin":
return "osx-x64" if machine in ["x86_64"] else "osx-arm64"
elif system == "linux":
return "linux-x64" if machine in ["x86_64", "amd64"] else "linux-arm64"
else:
raise SolidLSPException(f"Unsupported platform: {system} {machine}")
@staticmethod
def _get_runtime_dependencies(runtime_id: str) -> tuple[RuntimeDependency, RuntimeDependency]:
"""Get the language server and .NET runtime dependencies for the platform."""
lang_server_dep = None
dotnet_runtime_dep = None
for dep in RUNTIME_DEPENDENCIES:
if dep.id == "CSharpLanguageServer" and dep.platform_id == runtime_id:
lang_server_dep = dep
elif dep.id == "DotNetRuntime" and dep.platform_id == runtime_id:
dotnet_runtime_dep = dep
if not lang_server_dep:
raise SolidLSPException(f"No C# language server dependency found for platform {runtime_id}")
if not dotnet_runtime_dep:
raise SolidLSPException(f"No .NET runtime dependency found for platform {runtime_id}")
return lang_server_dep, dotnet_runtime_dep
@classmethod
def _ensure_dotnet_runtime(
cls, logger: LanguageServerLogger, runtime_dep: RuntimeDependency, solidlsp_settings: SolidLSPSettings
cls, logger: LanguageServerLogger, dotnet_runtime_dep: RuntimeDependency, solidlsp_settings: SolidLSPSettings
) -> str:
"""Ensure .NET runtime is available and return the dotnet executable path."""
# TODO: use RuntimeDependency util methods instead of custom validation/download logic
# Check if dotnet is already available on the system
system_dotnet = shutil.which("dotnet")
if system_dotnet:
@@ -286,7 +298,7 @@ class CSharpLanguageServer(SolidLanguageServer):
pass
# Download .NET 9 runtime using config
return cls._ensure_dotnet_runtime_from_config(logger, runtime_dep, solidlsp_settings)
return cls._ensure_dotnet_runtime_from_config(logger, dotnet_runtime_dep, solidlsp_settings)
@classmethod
def _ensure_language_server(
@@ -318,7 +330,7 @@ class CSharpLanguageServer(SolidLanguageServer):
# Make executable on Unix systems
if platform.system().lower() != "windows":
server_dll.chmod(server_dll.stat().st_mode | stat.S_IEXEC)
server_dll.chmod(0o755)
logger.log(f"Successfully installed Microsoft.CodeAnalysis.LanguageServer to {server_dll}", logging.INFO)
return str(server_dll)
@@ -391,8 +403,9 @@ class CSharpLanguageServer(SolidLanguageServer):
package_extract_dir = temp_dir / f"{package_name}.{package_version}"
package_extract_dir.mkdir(exist_ok=True)
with zipfile.ZipFile(nupkg_file, "r") as zip_ref:
zip_ref.extractall(package_extract_dir)
# Use SafeZipExtractor to handle long paths and skip errors
extractor = SafeZipExtractor(archive_path=nupkg_file, extract_dir=package_extract_dir, verbose=False)
extractor.extract_all()
# Clean up the nupkg file
nupkg_file.unlink()
@@ -407,12 +420,14 @@ class CSharpLanguageServer(SolidLanguageServer):
@classmethod
def _ensure_dotnet_runtime_from_config(
cls, logger: LanguageServerLogger, runtime_dep: RuntimeDependency, solidlsp_settings: SolidLSPSettings
cls, logger: LanguageServerLogger, dotnet_runtime_dep: RuntimeDependency, solidlsp_settings: SolidLSPSettings
) -> str:
"""
Ensure .NET 9 runtime is available using runtime dependency configuration.
Returns the path to the dotnet executable.
"""
# TODO: use RuntimeDependency util methods instead of custom download logic
# Check if dotnet is already available on the system
system_dotnet = shutil.which("dotnet")
if system_dotnet:
@@ -427,8 +442,7 @@ class CSharpLanguageServer(SolidLanguageServer):
# Download .NET 9 runtime using config
dotnet_dir = Path(cls.ls_resources_dir(solidlsp_settings)) / "dotnet-runtime-9.0"
assert runtime_dep.binary_name is not None
dotnet_exe = dotnet_dir / runtime_dep.binary_name
dotnet_exe = dotnet_dir / dotnet_runtime_dep.binary_name
if dotnet_exe.exists():
logger.log(f"Using cached .NET runtime from {dotnet_exe}", logging.INFO)
@@ -438,9 +452,16 @@ class CSharpLanguageServer(SolidLanguageServer):
logger.log("Downloading .NET 9 runtime...", logging.INFO)
dotnet_dir.mkdir(parents=True, exist_ok=True)
url = runtime_dep.url
assert url is not None
archive_type = runtime_dep.archive_type
custom_dotnet_runtime_url = solidlsp_settings.ls_specific_settings.get(cls.get_language_enum_instance(), {}).get(
"dotnet_runtime_url"
)
if custom_dotnet_runtime_url is not None:
logger.log(f"Using custom .NET runtime url: {custom_dotnet_runtime_url}", logging.INFO)
url = custom_dotnet_runtime_url
else:
url = dotnet_runtime_dep.url
archive_type = dotnet_runtime_dep.archive_type
# Download the runtime
download_path = dotnet_dir / f"dotnet-runtime.{archive_type}"
@@ -462,7 +483,7 @@ class CSharpLanguageServer(SolidLanguageServer):
# Make dotnet executable on Unix
if platform.system().lower() != "windows":
dotnet_exe.chmod(dotnet_exe.stat().st_mode | stat.S_IEXEC)
dotnet_exe.chmod(0o755)
logger.log(f"Successfully installed .NET 9 runtime to {dotnet_exe}", logging.INFO)
return str(dotnet_exe)
@@ -629,6 +650,9 @@ class CSharpLanguageServer(SolidLanguageServer):
# Just acknowledge the request - we don't need to track these for now
return
def handle_project_needs_restore(params):
return
# Set up notification handlers
self.server.on_notification("window/logMessage", window_log_message)
self.server.on_notification("$/progress", handle_progress)
@@ -636,6 +660,7 @@ class CSharpLanguageServer(SolidLanguageServer):
self.server.on_request("workspace/configuration", handle_workspace_configuration)
self.server.on_request("window/workDoneProgress/create", handle_work_done_progress_create)
self.server.on_request("client/registerCapability", handle_register_capability)
self.server.on_request("workspace/_roslyn_projectNeedsRestore", handle_project_needs_restore)
self.logger.log("Starting Microsoft.CodeAnalysis.LanguageServer process", logging.INFO)
@@ -736,3 +761,7 @@ class CSharpLanguageServer(SolidLanguageServer):
project_uris = [PathUtils.path_to_uri(project_file) for project_file in project_files]
self.server.notify.send_notification("project/open", {"projects": project_uris})
self.logger.log(f"Opened project files: {project_files}", logging.DEBUG)
@override
def _get_wait_time_for_cross_file_referencing(self) -> float:
return 2
src/solidlsp/language_servers/dart_language_server.py
+1 -2
View File
@@ -1,7 +1,6 @@
import logging
import os
import pathlib
import stat
from typing import cast
from solidlsp.ls import SolidLanguageServer
@@ -89,7 +88,7 @@ class DartLanguageServer(SolidLanguageServer):
deps.install(logger, dart_ls_dir)
assert os.path.exists(dart_executable_path)
os.chmod(dart_executable_path, stat.S_IEXEC)
os.chmod(dart_executable_path, 0o755)
return f"{dart_executable_path} language-server --client-id multilspy.dart --client-version 1.2"
src/solidlsp/language_servers/eclipse_jdtls.py
+4 -5
View File
@@ -7,7 +7,6 @@ import logging
import os
import pathlib
import shutil
import stat
import threading
import uuid
from pathlib import PurePath
@@ -169,7 +168,7 @@ class EclipseJDTLS(SolidLanguageServer):
runtime_dependencies = {
"gradle": {
"platform-agnostic": {
"url": "https://services.gradle.org/distributions/gradle-7.3.3-bin.zip",
"url": "https://services.gradle.org/distributions/gradle-8.14.2-bin.zip",
"archiveType": "zip",
"relative_extraction_path": ".",
}
@@ -246,7 +245,7 @@ class EclipseJDTLS(SolidLanguageServer):
gradle_path = str(
PurePath(
cls.ls_resources_dir(solidlsp_settings),
"gradle-7.3.3",
"gradle-8.14.2",
)
)
@@ -280,7 +279,7 @@ class EclipseJDTLS(SolidLanguageServer):
):
FileUtils.download_and_extract_archive(logger, dependency["url"], vscode_java_path, dependency["archiveType"])
os.chmod(jre_path, stat.S_IEXEC)
os.chmod(jre_path, 0o755)
assert os.path.exists(vscode_java_path)
assert os.path.exists(jre_home_path)
@@ -533,7 +532,7 @@ class EclipseJDTLS(SolidLanguageServer):
},
"gradle": {
"enabled": True,
"wrapper": {"enabled": True},
"wrapper": {"enabled": False},
"version": None,
"home": "abs(static/gradle-7.3.3)",
"java": {"home": "abs(static/launch_jres/21.0.7-linux-x86_64)"},
src/solidlsp/language_servers/elixir_tools/elixir_tools.py
+14 -3
View File
@@ -109,6 +109,7 @@ class ElixirTools(SolidLanguageServer):
valid_platforms = [
PlatformId.LINUX_x64,
PlatformId.LINUX_arm64,
PlatformId.OSX_x64,
PlatformId.OSX_arm64,
]
@@ -116,20 +117,30 @@ class ElixirTools(SolidLanguageServer):
next_ls_dir = os.path.join(cls.ls_resources_dir(solidlsp_settings), "next-ls")
NEXTLS_VERSION = "v0.23.4"
# Define runtime dependencies inline
runtime_deps = {
PlatformId.LINUX_x64: RuntimeDependency(
id="next_ls_linux_amd64",
platform_id="linux-x64",
url="https://github.com/elixir-tools/next-ls/releases/download/v0.23.3/next_ls_linux_amd64",
url=f"https://github.com/elixir-tools/next-ls/releases/download/{NEXTLS_VERSION}/next_ls_linux_amd64",
archive_type="binary",
binary_name="next_ls_linux_amd64",
extract_path="next_ls",
),
PlatformId.LINUX_arm64: RuntimeDependency(
id="next_ls_linux_arm64",
platform_id="linux-arm64",
url=f"https://github.com/elixir-tools/next-ls/releases/download/{NEXTLS_VERSION}/next_ls_linux_arm64",
archive_type="binary",
binary_name="next_ls_linux_arm64",
extract_path="next_ls",
),
PlatformId.OSX_x64: RuntimeDependency(
id="next_ls_darwin_amd64",
platform_id="osx-x64",
url="https://github.com/elixir-tools/next-ls/releases/download/v0.23.3/next_ls_darwin_amd64",
url=f"https://github.com/elixir-tools/next-ls/releases/download/{NEXTLS_VERSION}/next_ls_darwin_amd64",
archive_type="binary",
binary_name="next_ls_darwin_amd64",
extract_path="next_ls",
@@ -137,7 +148,7 @@ class ElixirTools(SolidLanguageServer):
PlatformId.OSX_arm64: RuntimeDependency(
id="next_ls_darwin_arm64",
platform_id="osx-arm64",
url="https://github.com/elixir-tools/next-ls/releases/download/v0.23.3/next_ls_darwin_arm64",
url=f"https://github.com/elixir-tools/next-ls/releases/download/{NEXTLS_VERSION}/next_ls_darwin_arm64",
archive_type="binary",
binary_name="next_ls_darwin_arm64",
extract_path="next_ls",
src/solidlsp/language_servers/elm_language_server.py
+189
View File
@@ -0,0 +1,189 @@
"""
Provides Elm specific instantiation of the LanguageServer class. Contains various configurations and settings specific to Elm.
"""
import logging
import os
import pathlib
import shutil
import threading
from overrides import override
from sensai.util.logging import LogTime
from solidlsp.ls import SolidLanguageServer
from solidlsp.ls_config import LanguageServerConfig
from solidlsp.ls_logger import LanguageServerLogger
from solidlsp.lsp_protocol_handler.lsp_types import InitializeParams
from solidlsp.lsp_protocol_handler.server import ProcessLaunchInfo
from solidlsp.settings import SolidLSPSettings
from .common import RuntimeDependency, RuntimeDependencyCollection
class ElmLanguageServer(SolidLanguageServer):
"""
Provides Elm specific instantiation of the LanguageServer class. Contains various configurations and settings specific to Elm.
"""
def __init__(
self, config: LanguageServerConfig, logger: LanguageServerLogger, repository_root_path: str, solidlsp_settings: SolidLSPSettings
):
"""
Creates an ElmLanguageServer instance. This class is not meant to be instantiated directly. Use LanguageServer.create() instead.
"""
elm_lsp_executable_path = self._setup_runtime_dependencies(logger, config, solidlsp_settings)
super().__init__(
config,
logger,
repository_root_path,
ProcessLaunchInfo(cmd=elm_lsp_executable_path, cwd=repository_root_path),
"elm",
solidlsp_settings,
)
self.server_ready = threading.Event()
@override
def is_ignored_dirname(self, dirname: str) -> bool:
return super().is_ignored_dirname(dirname) or dirname in [
"elm-stuff",
"node_modules",
"dist",
"build",
]
@classmethod
def _setup_runtime_dependencies(
cls, logger: LanguageServerLogger, config: LanguageServerConfig, solidlsp_settings: SolidLSPSettings
) -> list[str]:
"""
Setup runtime dependencies for Elm Language Server and return the command to start the server.
"""
# Check if elm-language-server is already installed globally
system_elm_ls = shutil.which("elm-language-server")
if system_elm_ls:
logger.log(f"Found system-installed elm-language-server at {system_elm_ls}", logging.INFO)
return [system_elm_ls, "--stdio"]
# Verify node and npm are installed
is_node_installed = shutil.which("node") is not None
assert is_node_installed, "node is not installed or isn't in PATH. Please install NodeJS and try again."
is_npm_installed = shutil.which("npm") is not None
assert is_npm_installed, "npm is not installed or isn't in PATH. Please install npm and try again."
deps = RuntimeDependencyCollection(
[
RuntimeDependency(
id="elm-language-server",
description="@elm-tooling/elm-language-server package",
command=["npm", "install", "--prefix", "./", "@elm-tooling/elm-language-server@2.8.0"],
platform_id="any",
),
]
)
# Install elm-language-server if not already installed
elm_ls_dir = os.path.join(cls.ls_resources_dir(solidlsp_settings), "elm-lsp")
elm_ls_executable_path = os.path.join(elm_ls_dir, "node_modules", ".bin", "elm-language-server")
if not os.path.exists(elm_ls_executable_path):
logger.log(f"Elm Language Server executable not found at {elm_ls_executable_path}. Installing...", logging.INFO)
with LogTime("Installation of Elm language server dependencies", logger=logger.logger):
deps.install(logger, elm_ls_dir)
if not os.path.exists(elm_ls_executable_path):
raise FileNotFoundError(
f"elm-language-server executable not found at {elm_ls_executable_path}, something went wrong with the installation."
)
return [elm_ls_executable_path, "--stdio"]
@staticmethod
def _get_initialize_params(repository_absolute_path: str) -> InitializeParams:
"""
Returns the initialize params for the Elm Language Server.
"""
root_uri = pathlib.Path(repository_absolute_path).as_uri()
initialize_params = {
"locale": "en",
"capabilities": {
"textDocument": {
"synchronization": {"didSave": True, "dynamicRegistration": True},
"completion": {"dynamicRegistration": True, "completionItem": {"snippetSupport": True}},
"definition": {"dynamicRegistration": True},
"references": {"dynamicRegistration": True},
"documentSymbol": {
"dynamicRegistration": True,
"hierarchicalDocumentSymbolSupport": True,
"symbolKind": {"valueSet": list(range(1, 27))},
},
"hover": {"dynamicRegistration": True, "contentFormat": ["markdown", "plaintext"]},
"codeAction": {"dynamicRegistration": True},
"rename": {"dynamicRegistration": True},
},
"workspace": {
"workspaceFolders": True,
"didChangeConfiguration": {"dynamicRegistration": True},
"symbol": {"dynamicRegistration": True},
},
},
"initializationOptions": {
"elmPath": "elm",
"elmFormatPath": "elm-format",
"elmTestPath": "elm-test",
"skipInstallPackageConfirmation": True,
"onlyUpdateDiagnosticsOnSave": False,
},
"processId": os.getpid(),
"rootPath": repository_absolute_path,
"rootUri": root_uri,
"workspaceFolders": [
{
"uri": root_uri,
"name": os.path.basename(repository_absolute_path),
}
],
}
return initialize_params
def _start_server(self):
"""
Starts the Elm Language Server, waits for the server to be ready and yields the LanguageServer instance.
"""
def do_nothing(params):
return
def window_log_message(msg):
self.logger.log(f"LSP: window/logMessage: {msg}", logging.INFO)
self.server.on_notification("window/logMessage", window_log_message)
self.server.on_notification("$/progress", do_nothing)
self.server.on_notification("textDocument/publishDiagnostics", do_nothing)
self.logger.log("Starting Elm server process", logging.INFO)
self.server.start()
initialize_params = self._get_initialize_params(self.repository_root_path)
self.logger.log(
"Sending initialize request from LSP client to LSP server and awaiting response",
logging.INFO,
)
init_response = self.server.send.initialize(initialize_params)
# Elm-specific capability checks
assert "textDocumentSync" in init_response["capabilities"]
assert "completionProvider" in init_response["capabilities"]
assert "definitionProvider" in init_response["capabilities"]
assert "referencesProvider" in init_response["capabilities"]
assert "documentSymbolProvider" in init_response["capabilities"]
self.server.notify.initialized({})
self.logger.log("Elm server initialized successfully, waiting for workspace scan...", logging.INFO)
self.server_ready.set()
self.completions_available.set()
self.logger.log("Elm server ready", logging.INFO)
@override
def _get_wait_time_for_cross_file_referencing(self) -> float:
return 1.0
src/solidlsp/language_servers/erlang_language_server.py
+230
View File
@@ -0,0 +1,230 @@
"""Erlang Language Server implementation using Erlang LS."""
import logging
import os
import shutil
import subprocess
import threading
import time
from overrides import override
from solidlsp.ls import SolidLanguageServer
from solidlsp.ls_config import LanguageServerConfig
from solidlsp.ls_logger import LanguageServerLogger
from solidlsp.lsp_protocol_handler.server import ProcessLaunchInfo
from solidlsp.settings import SolidLSPSettings
class ErlangLanguageServer(SolidLanguageServer):
"""Language server for Erlang using Erlang LS."""
def __init__(
self,
config: LanguageServerConfig,
logger: LanguageServerLogger,
repository_root_path: str,
solidlsp_settings: SolidLSPSettings,
):
"""
Creates an ErlangLanguageServer instance. This class is not meant to be instantiated directly.
Use LanguageServer.create() instead.
"""
self.erlang_ls_path = shutil.which("erlang_ls")
if not self.erlang_ls_path:
raise RuntimeError("Erlang LS not found. Install from: https://github.com/erlang-ls/erlang_ls")
if not self._check_erlang_installation():
raise RuntimeError("Erlang/OTP not found. Install from: https://www.erlang.org/downloads")
super().__init__(
config,
logger,
repository_root_path,
ProcessLaunchInfo(cmd=[self.erlang_ls_path, "--transport", "stdio"], cwd=repository_root_path),
"erlang",
solidlsp_settings,
)
# Add server readiness tracking like Elixir
self.server_ready = threading.Event()
# Set generous timeout for Erlang LS initialization
self.set_request_timeout(120.0)
def _check_erlang_installation(self) -> bool:
"""Check if Erlang/OTP is available."""
try:
result = subprocess.run(["erl", "-version"], check=False, capture_output=True, text=True, timeout=10)
return result.returncode == 0
except (subprocess.SubprocessError, FileNotFoundError):
return False
@classmethod
def _get_erlang_version(cls):
"""Get the installed Erlang/OTP version or None if not found."""
try:
result = subprocess.run(["erl", "-version"], check=False, capture_output=True, text=True, timeout=10)
if result.returncode == 0:
return result.stderr.strip() # erl -version outputs to stderr
except (subprocess.SubprocessError, FileNotFoundError):
return None
return None
@classmethod
def _check_rebar3_available(cls) -> bool:
"""Check if rebar3 build tool is available."""
try:
result = subprocess.run(["rebar3", "version"], check=False, capture_output=True, text=True, timeout=10)
return result.returncode == 0
except (subprocess.SubprocessError, FileNotFoundError):
return False
def _start_server(self):
"""Start Erlang LS server process with proper initialization waiting."""
def register_capability_handler(params):
return
def window_log_message(msg):
"""Handle window/logMessage notifications from Erlang LS"""
message_text = msg.get("message", "")
self.logger.log(f"LSP: window/logMessage: {message_text}", logging.INFO)
# Look for Erlang LS readiness signals
# Common patterns: "Started Erlang LS", "initialized", "ready"
readiness_signals = [
"Started Erlang LS",
"server started",
"initialized",
"ready to serve requests",
"compilation finished",
"indexing complete",
]
message_lower = message_text.lower()
for signal in readiness_signals:
if signal.lower() in message_lower:
self.logger.log(f"Erlang LS readiness signal detected: {message_text}", logging.INFO)
self.server_ready.set()
break
def do_nothing(params):
return
def check_server_ready(params):
"""Handle $/progress notifications from Erlang LS as fallback."""
value = params.get("value", {})
# Check for initialization completion progress
if value.get("kind") == "end":
message = value.get("message", "")
if any(word in message.lower() for word in ["initialized", "ready", "complete"]):
self.logger.log("Erlang LS initialization progress completed", logging.INFO)
# Set as fallback if no window/logMessage was received
if not self.server_ready.is_set():
self.server_ready.set()
# Set up notification handlers
self.server.on_request("client/registerCapability", register_capability_handler)
self.server.on_notification("window/logMessage", window_log_message)
self.server.on_notification("$/progress", check_server_ready)
self.server.on_notification("window/workDoneProgress/create", do_nothing)
self.server.on_notification("$/workDoneProgress", do_nothing)
self.server.on_notification("textDocument/publishDiagnostics", do_nothing)
self.logger.log("Starting Erlang LS server process", logging.INFO)
self.server.start()
# Send initialize request
initialize_params = {
"processId": None,
"rootPath": self.repository_root_path,
"rootUri": f"file://{self.repository_root_path}",
"capabilities": {
"textDocument": {
"synchronization": {"didSave": True},
"completion": {"dynamicRegistration": True},
"definition": {"dynamicRegistration": True},
"references": {"dynamicRegistration": True},
"documentSymbol": {"dynamicRegistration": True},
"hover": {"dynamicRegistration": True},
}
},
}
self.logger.log("Sending initialize request to Erlang LS", logging.INFO)
init_response = self.server.send.initialize(initialize_params)
# Verify server capabilities
if "capabilities" in init_response:
self.logger.log(f"Erlang LS capabilities: {list(init_response['capabilities'].keys())}", logging.INFO)
self.server.notify.initialized({})
self.completions_available.set()
# Wait for Erlang LS to be ready - adjust timeout based on environment
is_ci = os.getenv("CI") == "true" or os.getenv("GITHUB_ACTIONS") == "true"
is_macos = os.uname().sysname == "Darwin" if hasattr(os, "uname") else False
# macOS in CI can be particularly slow for language server startup
if is_ci and is_macos:
ready_timeout = 240.0 # 4 minutes for macOS CI
env_desc = "macOS CI"
elif is_ci:
ready_timeout = 180.0 # 3 minutes for other CI
env_desc = "CI"
else:
ready_timeout = 60.0 # 1 minute for local
env_desc = "local"
self.logger.log(f"Waiting up to {ready_timeout} seconds for Erlang LS readiness ({env_desc} environment)...", logging.INFO)
if self.server_ready.wait(timeout=ready_timeout):
self.logger.log("Erlang LS is ready and available for requests", logging.INFO)
# Add settling period for indexing - adjust based on environment
settling_time = 15.0 if is_ci else 5.0
self.logger.log(f"Allowing {settling_time} seconds for Erlang LS indexing to complete...", logging.INFO)
time.sleep(settling_time)
self.logger.log("Erlang LS settling period complete", logging.INFO)
else:
# Set ready anyway and continue - Erlang LS might not send explicit ready messages
self.logger.log(
f"Erlang LS readiness timeout reached after {ready_timeout}s, proceeding anyway (common in CI)", logging.WARNING
)
self.server_ready.set()
# Still give some time for basic initialization even without explicit readiness signal
basic_settling_time = 20.0 if is_ci else 10.0
self.logger.log(f"Allowing {basic_settling_time} seconds for basic Erlang LS initialization...", logging.INFO)
time.sleep(basic_settling_time)
self.logger.log("Basic Erlang LS initialization period complete", logging.INFO)
@override
def is_ignored_dirname(self, dirname: str) -> bool:
# For Erlang projects, we should ignore:
# - _build: rebar3 build artifacts
# - deps: dependencies
# - ebin: compiled beam files
# - .rebar3: rebar3 cache
# - logs: log files
# - node_modules: if the project has JavaScript components
return super().is_ignored_dirname(dirname) or dirname in [
"_build",
"deps",
"ebin",
".rebar3",
"logs",
"node_modules",
"_checkouts",
"cover",
]
def is_ignored_filename(self, filename: str) -> bool:
"""Check if a filename should be ignored."""
# Ignore compiled BEAM files
if filename.endswith(".beam"):
return True
# Don't ignore Erlang source files, header files, or configuration files
return False
src/solidlsp/language_servers/intelephense.py
+24 -9
View File
@@ -26,6 +26,10 @@ from .common import RuntimeDependency, RuntimeDependencyCollection
class Intelephense(SolidLanguageServer):
"""
Provides PHP specific instantiation of the LanguageServer class using Intelephense.
You can pass the following entries in ls_specific_settings["php"]:
- maxMemory
- maxFileSize
"""
@override
@@ -37,9 +41,7 @@ class Intelephense(SolidLanguageServer):
return super().is_ignored_dirname(dirname) or dirname in ["node_modules", "vendor", "cache"]
@classmethod
def _setup_runtime_dependencies(
cls, logger: LanguageServerLogger, config: LanguageServerConfig, solidlsp_settings: SolidLSPSettings
) -> str:
def _setup_runtime_dependencies(cls, logger: LanguageServerLogger, solidlsp_settings: SolidLSPSettings) -> list[str]:
"""
Setup runtime dependencies for Intelephense and return the command to start the server.
"""
@@ -82,13 +84,13 @@ class Intelephense(SolidLanguageServer):
intelephense_executable_path
), f"intelephense executable not found at {intelephense_executable_path}, something went wrong."
return f"{intelephense_executable_path} --stdio"
return [intelephense_executable_path, "--stdio"]
def __init__(
self, config: LanguageServerConfig, logger: LanguageServerLogger, repository_root_path: str, solidlsp_settings: SolidLSPSettings
):
# Setup runtime dependencies before initializing
intelephense_cmd = self._setup_runtime_dependencies(logger, config, solidlsp_settings)
intelephense_cmd = self._setup_runtime_dependencies(logger, solidlsp_settings)
super().__init__(
config,
@@ -100,10 +102,9 @@ class Intelephense(SolidLanguageServer):
)
self.request_id = 0
@staticmethod
def _get_initialize_params(repository_absolute_path: str) -> InitializeParams:
def _get_initialize_params(self, repository_absolute_path: str) -> InitializeParams:
"""
Returns the initialize params for the TypeScript Language Server.
Returns the initialization params for the Intelephense Language Server.
"""
root_uri = pathlib.Path(repository_absolute_path).as_uri()
initialize_params = {
@@ -125,8 +126,22 @@ class Intelephense(SolidLanguageServer):
}
],
}
initialization_options = {}
# Add license key if provided via environment variable
license_key = os.environ.get("INTELEPHENSE_LICENSE_KEY")
if license_key:
initialization_options["licenceKey"] = license_key
return cast(InitializeParams, initialize_params)
custom_intelephense_settings = self._solidlsp_settings.ls_specific_settings.get(self.get_language_enum_instance(), {})
max_memory = custom_intelephense_settings.get("maxMemory")
max_file_size = custom_intelephense_settings.get("maxFileSize")
if max_memory is not None:
initialization_options["intelephense.maxMemory"] = max_memory
if max_file_size is not None:
initialization_options["intelephense.files.maxSize"] = max_file_size
initialize_params["initializationOptions"] = initialization_options
return initialize_params
def _start_server(self) -> None:
"""Start Intelephense server process"""
src/solidlsp/language_servers/julia_server.py
+181
View File
@@ -0,0 +1,181 @@
import logging
import os
import pathlib
import platform
import shutil
import subprocess
from overrides import override
from solidlsp.ls import SolidLanguageServer
from solidlsp.ls_config import LanguageServerConfig
from solidlsp.ls_logger import LanguageServerLogger
from solidlsp.lsp_protocol_handler.lsp_types import InitializeParams
from solidlsp.lsp_protocol_handler.server import ProcessLaunchInfo
from solidlsp.settings import SolidLSPSettings
class JuliaLanguageServer(SolidLanguageServer):
"""
Language server implementation for Julia using LanguageServer.jl.
"""
def __init__(
self,
config: LanguageServerConfig,
logger: LanguageServerLogger,
repository_root_path: str,
solidlsp_settings: SolidLSPSettings,
):
julia_executable = self._setup_runtime_dependency(logger) # PASS LOGGER
julia_code = "using LanguageServer; runserver()"
if platform.system() == "Windows":
# On Windows, pass as list (Serena handles shell=True differently)
julia_ls_cmd = [julia_executable, "--startup-file=no", "--history-file=no", "-e", julia_code, repository_root_path]
else:
# On Linux/macOS, build shell-escaped string
import shlex
julia_ls_cmd = (
f"{shlex.quote(julia_executable)} "
f"--startup-file=no "
f"--history-file=no "
f"-e {shlex.quote(julia_code)} "
f"{shlex.quote(repository_root_path)}"
)
logger.log(f"[JULIA DEBUG] Command: {julia_ls_cmd}", logging.INFO)
super().__init__(
config,
logger,
repository_root_path,
ProcessLaunchInfo(cmd=julia_ls_cmd, cwd=repository_root_path),
"julia",
solidlsp_settings,
)
@staticmethod
def _setup_runtime_dependency(logger: LanguageServerLogger) -> str:
"""
Check if the Julia runtime is available and return its full path.
Raises RuntimeError with a helpful message if the dependency is missing.
"""
# First check if julia is in PATH
julia_path = shutil.which("julia")
# If not found in PATH, check common installation locations
if julia_path is None:
common_locations = [
os.path.expanduser("~/.juliaup/bin/julia"),
os.path.expanduser("~/.julia/bin/julia"),
"/usr/local/bin/julia",
"/usr/bin/julia",
]
for location in common_locations:
if os.path.isfile(location) and os.access(location, os.X_OK):
julia_path = location
break
if julia_path is None:
raise RuntimeError(
"Julia is not installed or not in your PATH. "
"Please install Julia from https://julialang.org/downloads/ and ensure it is accessible. "
f"Checked locations: {common_locations}"
)
# Check if LanguageServer.jl is installed
check_cmd = [julia_path, "-e", "using LanguageServer"]
try:
result = subprocess.run(check_cmd, check=False, capture_output=True, text=True, timeout=10)
if result.returncode != 0:
# LanguageServer.jl not found, install it
JuliaLanguageServer._install_language_server(julia_path, logger)
except subprocess.TimeoutExpired:
# Assume it needs installation
JuliaLanguageServer._install_language_server(julia_path, logger)
return julia_path
@staticmethod
def _install_language_server(julia_path: str, logger: LanguageServerLogger) -> None:
"""Install LanguageServer.jl package."""
logger.log("LanguageServer.jl not found. Installing... (this may take a minute)", logging.INFO)
install_cmd = [julia_path, "-e", 'using Pkg; Pkg.add("LanguageServer")']
try:
result = subprocess.run(install_cmd, check=False, capture_output=True, text=True, timeout=300) # 5 minutes for installation
if result.returncode == 0:
logger.log("LanguageServer.jl installed successfully!", logging.INFO)
else:
raise RuntimeError(f"Failed to install LanguageServer.jl: {result.stderr}")
except subprocess.TimeoutExpired:
raise RuntimeError(
"LanguageServer.jl installation timed out. Please install manually: julia -e 'using Pkg; Pkg.add(\"LanguageServer\")'"
)
@override
def is_ignored_dirname(self, dirname: str) -> bool:
"""Define language-specific directories to ignore for Julia projects."""
return super().is_ignored_dirname(dirname) or dirname in [".julia", "build", "dist"]
def _get_initialize_params(self, repository_absolute_path: str) -> InitializeParams:
"""
Returns the initialize params for the Julia Language Server.
"""
root_uri = pathlib.Path(repository_absolute_path).as_uri()
initialize_params: InitializeParams = { # type: ignore
"processId": os.getpid(),
"rootPath": repository_absolute_path,
"rootUri": root_uri,
"capabilities": {
"workspace": {"workspaceFolders": True},
"textDocument": {
"definition": {"dynamicRegistration": True},
"references": {"dynamicRegistration": True},
"documentSymbol": {"dynamicRegistration": True},
},
},
"workspaceFolders": [
{
"uri": root_uri,
"name": os.path.basename(repository_absolute_path),
}
],
}
return initialize_params
def _start_server(self):
"""Start the LanguageServer.jl server process."""
def do_nothing(params):
return
def window_log_message(msg):
self.logger.log(f"LSP: window/logMessage: {msg}", logging.INFO)
self.server.on_notification("window/logMessage", window_log_message)
self.server.on_notification("$/progress", do_nothing)
self.server.on_notification("textDocument/publishDiagnostics", do_nothing)
self.logger.log("Starting LanguageServer.jl server process", logging.INFO)
self.server.start()
initialize_params = self._get_initialize_params(self.repository_root_path)
self.logger.log(
"Sending initialize request to Julia Language Server",
logging.INFO,
)
init_response = self.server.send.initialize(initialize_params)
assert "definitionProvider" in init_response["capabilities"]
assert "referencesProvider" in init_response["capabilities"]
assert "documentSymbolProvider" in init_response["capabilities"]
self.server.notify.initialized({})
self.completions_available.set()
self.logger.log("Julia Language Server is initialized and ready.", logging.INFO)
src/solidlsp/language_servers/kotlin_language_server.py
+5 -5
View File
@@ -44,7 +44,7 @@ class KotlinLanguageServer(SolidLanguageServer):
self.runtime_dependency_paths = runtime_dependency_paths
# Create command to execute the Kotlin Language Server script
cmd = f'"{self.runtime_dependency_paths.kotlin_executable_path}"'
cmd = [self.runtime_dependency_paths.kotlin_executable_path, "--stdio"]
# Set environment variables including JAVA_HOME
proc_env = {"JAVA_HOME": self.runtime_dependency_paths.java_home_path}
@@ -77,7 +77,7 @@ class KotlinLanguageServer(SolidLanguageServer):
"runtimeDependency": {
"id": "KotlinLsp",
"description": "Kotlin Language Server",
"url": "https://github.com/fwcd/kotlin-language-server/releases/download/1.3.13/server.zip",
"url": "https://download-cdn.jetbrains.com/kotlin-lsp/0.253.10629/kotlin-0.253.10629.zip",
"archiveType": "zip",
},
"java": {
@@ -139,13 +139,13 @@ class KotlinLanguageServer(SolidLanguageServer):
assert os.path.exists(java_path), f"Java executable not found at {java_path}"
# Setup Kotlin Language Server paths
kotlin_ls_dir = os.path.join(static_dir, "server")
kotlin_ls_dir = static_dir
# Get platform-specific executable script path
if platform_id.value.startswith("win-"):
kotlin_script = os.path.join(kotlin_ls_dir, "bin", "kotlin-language-server.bat")
kotlin_script = os.path.join(kotlin_ls_dir, "kotlin-lsp.cmd")
else:
kotlin_script = os.path.join(kotlin_ls_dir, "bin", "kotlin-language-server")
kotlin_script = os.path.join(kotlin_ls_dir, "kotlin-lsp.sh")
# Download and extract Kotlin Language Server if script doesn't exist
if not os.path.exists(kotlin_script):
src/solidlsp/language_servers/lua_ls.py
+297
View File
@@ -0,0 +1,297 @@
"""
Provides Lua specific instantiation of the LanguageServer class using lua-language-server.
"""
import logging
import os
import pathlib
import platform
import shutil
import tarfile
import threading
import zipfile
from pathlib import Path
import requests
from overrides import override
from solidlsp.ls import SolidLanguageServer
from solidlsp.ls_config import LanguageServerConfig
from solidlsp.ls_logger import LanguageServerLogger
from solidlsp.lsp_protocol_handler.lsp_types import InitializeParams
from solidlsp.lsp_protocol_handler.server import ProcessLaunchInfo
from solidlsp.settings import SolidLSPSettings
class LuaLanguageServer(SolidLanguageServer):
"""
Provides Lua specific instantiation of the LanguageServer class using lua-language-server.
"""
@override
def is_ignored_dirname(self, dirname: str) -> bool:
# For Lua projects, we should ignore:
# - .luarocks: package manager cache
# - lua_modules: local dependencies
# - node_modules: if the project has JavaScript components
return super().is_ignored_dirname(dirname) or dirname in [".luarocks", "lua_modules", "node_modules", "build", "dist", ".cache"]
@staticmethod
def _get_lua_ls_path():
"""Get the path to lua-language-server executable."""
# First check if it's in PATH
lua_ls = shutil.which("lua-language-server")
if lua_ls:
return lua_ls
# Check common installation locations
home = Path.home()
possible_paths = [
home / ".local" / "bin" / "lua-language-server",
home / ".serena" / "language_servers" / "lua" / "bin" / "lua-language-server",
Path("/usr/local/bin/lua-language-server"),
Path("/opt/lua-language-server/bin/lua-language-server"),
]
# Add Windows-specific paths
if platform.system() == "Windows":
possible_paths.extend(
[
home / "AppData" / "Local" / "lua-language-server" / "bin" / "lua-language-server.exe",
home / ".serena" / "language_servers" / "lua" / "bin" / "lua-language-server.exe",
]
)
for path in possible_paths:
if path.exists():
return str(path)
return None
@staticmethod
def _download_lua_ls():
"""Download and install lua-language-server if not present."""
system = platform.system()
machine = platform.machine().lower()
lua_ls_version = "3.15.0"
# Map platform and architecture to download URL
if system == "Linux":
if machine in ["x86_64", "amd64"]:
download_name = f"lua-language-server-{lua_ls_version}-linux-x64.tar.gz"
elif machine in ["aarch64", "arm64"]:
download_name = f"lua-language-server-{lua_ls_version}-linux-arm64.tar.gz"
else:
raise RuntimeError(f"Unsupported Linux architecture: {machine}")
elif system == "Darwin":
if machine in ["x86_64", "amd64"]:
download_name = f"lua-language-server-{lua_ls_version}-darwin-x64.tar.gz"
elif machine in ["arm64", "aarch64"]:
download_name = f"lua-language-server-{lua_ls_version}-darwin-arm64.tar.gz"
else:
raise RuntimeError(f"Unsupported macOS architecture: {machine}")
elif system == "Windows":
if machine in ["amd64", "x86_64"]:
download_name = f"lua-language-server-{lua_ls_version}-win32-x64.zip"
else:
raise RuntimeError(f"Unsupported Windows architecture: {machine}")
else:
raise RuntimeError(f"Unsupported operating system: {system}")
download_url = f"https://github.com/LuaLS/lua-language-server/releases/download/{lua_ls_version}/{download_name}"
# Create installation directory
install_dir = Path.home() / ".serena" / "language_servers" / "lua"
install_dir.mkdir(parents=True, exist_ok=True)
# Download the file
print(f"Downloading lua-language-server from {download_url}...")
response = requests.get(download_url, stream=True)
response.raise_for_status()
# Save and extract
download_path = install_dir / download_name
with open(download_path, "wb") as f:
for chunk in response.iter_content(chunk_size=8192):
f.write(chunk)
print(f"Extracting lua-language-server to {install_dir}...")
if download_name.endswith(".tar.gz"):
with tarfile.open(download_path, "r:gz") as tar:
tar.extractall(install_dir)
elif download_name.endswith(".zip"):
with zipfile.ZipFile(download_path, "r") as zip_ref:
zip_ref.extractall(install_dir)
# Clean up download file
download_path.unlink()
# Make executable on Unix systems
if system != "Windows":
lua_ls_path = install_dir / "bin" / "lua-language-server"
if lua_ls_path.exists():
lua_ls_path.chmod(0o755)
return str(lua_ls_path)
else:
lua_ls_path = install_dir / "bin" / "lua-language-server.exe"
if lua_ls_path.exists():
return str(lua_ls_path)
raise RuntimeError("Failed to find lua-language-server executable after extraction")
@staticmethod
def _setup_runtime_dependency():
"""
Check if required Lua runtime dependencies are available.
Downloads lua-language-server if not present.
"""
lua_ls_path = LuaLanguageServer._get_lua_ls_path()
if not lua_ls_path:
print("lua-language-server not found. Downloading...")
lua_ls_path = LuaLanguageServer._download_lua_ls()
print(f"lua-language-server installed at: {lua_ls_path}")
return lua_ls_path
def __init__(
self, config: LanguageServerConfig, logger: LanguageServerLogger, repository_root_path: str, solidlsp_settings: SolidLSPSettings
):
lua_ls_path = self._setup_runtime_dependency()
super().__init__(
config,
logger,
repository_root_path,
ProcessLaunchInfo(cmd=lua_ls_path, cwd=repository_root_path),
"lua",
solidlsp_settings,
)
self.server_ready = threading.Event()
self.request_id = 0
@staticmethod
def _get_initialize_params(repository_absolute_path: str) -> InitializeParams:
"""
Returns the initialize params for the Lua Language Server.
"""
root_uri = pathlib.Path(repository_absolute_path).as_uri()
initialize_params = {
"locale": "en",
"capabilities": {
"textDocument": {
"synchronization": {"didSave": True, "dynamicRegistration": True},
"definition": {"dynamicRegistration": True},
"references": {"dynamicRegistration": True},
"documentSymbol": {
"dynamicRegistration": True,
"hierarchicalDocumentSymbolSupport": True,
"symbolKind": {"valueSet": list(range(1, 27))},
},
"completion": {
"dynamicRegistration": True,
"completionItem": {
"snippetSupport": True,
"commitCharactersSupport": True,
"documentationFormat": ["markdown", "plaintext"],
"deprecatedSupport": True,
"preselectSupport": True,
},
},
"hover": {
"dynamicRegistration": True,
"contentFormat": ["markdown", "plaintext"],
},
"signatureHelp": {
"dynamicRegistration": True,
"signatureInformation": {
"documentationFormat": ["markdown", "plaintext"],
"parameterInformation": {"labelOffsetSupport": True},
},
},
},
"workspace": {
"workspaceFolders": True,
"didChangeConfiguration": {"dynamicRegistration": True},
"configuration": True,
"symbol": {
"dynamicRegistration": True,
"symbolKind": {"valueSet": list(range(1, 27))},
},
},
},
"processId": os.getpid(),
"rootPath": repository_absolute_path,
"rootUri": root_uri,
"workspaceFolders": [
{
"uri": root_uri,
"name": os.path.basename(repository_absolute_path),
}
],
"initializationOptions": {
# Lua Language Server specific options
"runtime": {
"version": "Lua 5.4",
"path": ["?.lua", "?/init.lua"],
},
"diagnostics": {
"enable": True,
"globals": ["vim", "describe", "it", "before_each", "after_each"], # Common globals
},
"workspace": {
"library": [], # Can be extended with project-specific libraries
"checkThirdParty": False,
"userThirdParty": [],
},
"telemetry": {
"enable": False,
},
"completion": {
"enable": True,
"callSnippet": "Both",
"keywordSnippet": "Both",
},
},
}
return initialize_params
def _start_server(self):
"""Start Lua Language Server process"""
def register_capability_handler(params):
return
def window_log_message(msg):
self.logger.log(f"LSP: window/logMessage: {msg}", logging.INFO)
def do_nothing(params):
return
self.server.on_request("client/registerCapability", register_capability_handler)
self.server.on_notification("window/logMessage", window_log_message)
self.server.on_notification("$/progress", do_nothing)
self.server.on_notification("textDocument/publishDiagnostics", do_nothing)
self.logger.log("Starting Lua Language Server process", logging.INFO)
self.server.start()
initialize_params = self._get_initialize_params(self.repository_root_path)
self.logger.log(
"Sending initialize request from LSP client to LSP server and awaiting response",
logging.INFO,
)
init_response = self.server.send.initialize(initialize_params)
# Verify server capabilities
assert "textDocumentSync" in init_response["capabilities"]
assert "definitionProvider" in init_response["capabilities"]
assert "documentSymbolProvider" in init_response["capabilities"]
assert "referencesProvider" in init_response["capabilities"]
self.server.notify.initialized({})
self.completions_available.set()
# Lua Language Server is typically ready immediately after initialization
self.server_ready.set()
self.server_ready.wait()
src/solidlsp/language_servers/marksman.py
+193
View File
@@ -0,0 +1,193 @@
"""
Provides Markdown specific instantiation of the LanguageServer class using marksman.
Contains various configurations and settings specific to Markdown.
"""
import logging
import os
import pathlib
import threading
from overrides import override
from solidlsp.ls import SolidLanguageServer
from solidlsp.ls_config import LanguageServerConfig
from solidlsp.ls_logger import LanguageServerLogger
from solidlsp.lsp_protocol_handler.lsp_types import InitializeParams
from solidlsp.lsp_protocol_handler.server import ProcessLaunchInfo
from solidlsp.settings import SolidLSPSettings
from .common import RuntimeDependency, RuntimeDependencyCollection
class Marksman(SolidLanguageServer):
"""
Provides Markdown specific instantiation of the LanguageServer class using marksman.
"""
marksman_releases = "https://github.com/artempyanykh/marksman/releases/download/2024-12-18"
runtime_dependencies = RuntimeDependencyCollection(
[
RuntimeDependency(
id="marksman",
url=f"{marksman_releases}/marksman-linux-x64",
platform_id="linux-x64",
archive_type="binary",
binary_name="marksman",
),
RuntimeDependency(
id="marksman",
url=f"{marksman_releases}/marksman-linux-arm64",
platform_id="linux-arm64",
archive_type="binary",
binary_name="marksman",
),
RuntimeDependency(
id="marksman",
url=f"{marksman_releases}/marksman-macos",
platform_id="osx-x64",
archive_type="binary",
binary_name="marksman",
),
RuntimeDependency(
id="marksman",
url=f"{marksman_releases}/marksman-macos",
platform_id="osx-arm64",
archive_type="binary",
binary_name="marksman",
),
RuntimeDependency(
id="marksman",
url=f"{marksman_releases}/marksman.exe",
platform_id="win-x64",
archive_type="binary",
binary_name="marksman.exe",
),
]
)
@classmethod
def _setup_runtime_dependencies(
cls, logger: LanguageServerLogger, config: LanguageServerConfig, solidlsp_settings: SolidLSPSettings
) -> str:
"""Setup runtime dependencies for marksman and return the command to start the server."""
deps = cls.runtime_dependencies
dependency = deps.get_single_dep_for_current_platform()
marksman_ls_dir = cls.ls_resources_dir(solidlsp_settings)
marksman_executable_path = deps.binary_path(marksman_ls_dir)
if not os.path.exists(marksman_executable_path):
logger.log(
f"Downloading marksman from {dependency.url} to {marksman_ls_dir}",
logging.INFO,
)
deps.install(logger, marksman_ls_dir)
if not os.path.exists(marksman_executable_path):
raise FileNotFoundError(f"Download failed? Could not find marksman executable at {marksman_executable_path}")
os.chmod(marksman_executable_path, 0o755)
return marksman_executable_path
def __init__(
self, config: LanguageServerConfig, logger: LanguageServerLogger, repository_root_path: str, solidlsp_settings: SolidLSPSettings
):
"""
Creates a Marksman instance. This class is not meant to be instantiated directly.
Use LanguageServer.create() instead.
"""
marksman_executable_path = self._setup_runtime_dependencies(logger, config, solidlsp_settings)
super().__init__(
config,
logger,
repository_root_path,
ProcessLaunchInfo(cmd=f"{marksman_executable_path} server", cwd=repository_root_path),
"markdown",
solidlsp_settings,
)
self.server_ready = threading.Event()
@override
def is_ignored_dirname(self, dirname: str) -> bool:
return super().is_ignored_dirname(dirname) or dirname in ["node_modules", ".obsidian", ".vitepress", ".vuepress"]
@staticmethod
def _get_initialize_params(repository_absolute_path: str) -> InitializeParams:
"""
Returns the initialize params for the Marksman Language Server.
"""
root_uri = pathlib.Path(repository_absolute_path).as_uri()
initialize_params: InitializeParams = { # type: ignore
"processId": os.getpid(),
"locale": "en",
"rootPath": repository_absolute_path,
"rootUri": root_uri,
"capabilities": {
"textDocument": {
"synchronization": {"didSave": True, "dynamicRegistration": True},
"completion": {"dynamicRegistration": True, "completionItem": {"snippetSupport": True}},
"definition": {"dynamicRegistration": True},
"references": {"dynamicRegistration": True},
"documentSymbol": {
"dynamicRegistration": True,
"hierarchicalDocumentSymbolSupport": True,
"symbolKind": {"valueSet": list(range(1, 27))},
},
"hover": {"dynamicRegistration": True, "contentFormat": ["markdown", "plaintext"]},
"codeAction": {"dynamicRegistration": True},
},
"workspace": {
"workspaceFolders": True,
"didChangeConfiguration": {"dynamicRegistration": True},
"symbol": {"dynamicRegistration": True},
},
},
"workspaceFolders": [
{
"uri": root_uri,
"name": os.path.basename(repository_absolute_path),
}
],
}
return initialize_params
def _start_server(self):
"""
Starts the Marksman Language Server and waits for it to be ready.
"""
def register_capability_handler(_params):
return
def window_log_message(msg):
self.logger.log(f"LSP: window/logMessage: {msg}", logging.INFO)
def do_nothing(_params):
return
self.server.on_request("client/registerCapability", register_capability_handler)
self.server.on_notification("window/logMessage", window_log_message)
self.server.on_notification("$/progress", do_nothing)
self.server.on_notification("textDocument/publishDiagnostics", do_nothing)
self.logger.log("Starting marksman server process", logging.INFO)
self.server.start()
initialize_params = self._get_initialize_params(self.repository_root_path)
self.logger.log(
"Sending initialize request from LSP client to marksman server and awaiting response",
logging.INFO,
)
init_response = self.server.send.initialize(initialize_params)
self.logger.log(f"Received initialize response from marksman server: {init_response}", logging.DEBUG)
# Verify server capabilities
assert "textDocumentSync" in init_response["capabilities"]
assert "completionProvider" in init_response["capabilities"]
assert "definitionProvider" in init_response["capabilities"]
self.server.notify.initialized({})
# marksman is typically ready immediately after initialization
self.logger.log("Marksman server initialization complete", logging.INFO)
self.server_ready.set()
self.completions_available.set()
src/solidlsp/language_servers/nixd_ls.py
+396
View File
@@ -0,0 +1,396 @@
"""
Provides Nix specific instantiation of the LanguageServer class using nixd (Nix Language Server).
Note: Windows is not supported as Nix itself doesn't support Windows natively.
"""
import logging
import os
import pathlib
import platform
import shutil
import subprocess
import threading
from pathlib import Path
from overrides import override
from solidlsp import ls_types
from solidlsp.ls import SolidLanguageServer
from solidlsp.ls_config import LanguageServerConfig
from solidlsp.ls_logger import LanguageServerLogger
from solidlsp.lsp_protocol_handler.lsp_types import InitializeParams
from solidlsp.lsp_protocol_handler.server import ProcessLaunchInfo
from solidlsp.settings import SolidLSPSettings
class NixLanguageServer(SolidLanguageServer):
"""
Provides Nix specific instantiation of the LanguageServer class using nixd.
"""
def _extend_nix_symbol_range_to_include_semicolon(
self, symbol: ls_types.UnifiedSymbolInformation, file_content: str
) -> ls_types.UnifiedSymbolInformation:
"""
Extend symbol range to include trailing semicolon for Nix attribute symbols.
nixd provides ranges that exclude semicolons (expression-level), but serena needs
statement-level ranges that include semicolons for proper replacement.
"""
range_info = symbol["range"]
end_line = range_info["end"]["line"]
end_char = range_info["end"]["character"]
# Split file content into lines
lines = file_content.split("\n")
if end_line >= len(lines):
return symbol
line = lines[end_line]
# Check if there's a semicolon immediately after the current range end
if end_char < len(line) and line[end_char] == ";":
# Extend range to include the semicolon
new_range = {"start": range_info["start"], "end": {"line": end_line, "character": end_char + 1}}
# Create modified symbol with extended range
extended_symbol = symbol.copy()
extended_symbol["range"] = new_range
# CRITICAL: Also update the location.range if it exists
if extended_symbol.get("location"):
location = extended_symbol["location"].copy()
if "range" in location:
location["range"] = new_range.copy()
extended_symbol["location"] = location
return extended_symbol
return symbol
@override
def request_document_symbols(
self, relative_file_path: str, include_body: bool = False
) -> tuple[list[ls_types.UnifiedSymbolInformation], list[ls_types.UnifiedSymbolInformation]]:
"""
Override to extend Nix symbol ranges to include trailing semicolons.
nixd provides expression-level ranges (excluding semicolons) but serena needs
statement-level ranges (including semicolons) for proper symbol replacement.
"""
# Get symbols from parent implementation
all_symbols, root_symbols = super().request_document_symbols(relative_file_path, include_body)
# Get file content for range extension
file_content = self.language_server.retrieve_full_file_content(relative_file_path)
# Extend ranges for all symbols recursively
def extend_symbol_and_children(symbol: ls_types.UnifiedSymbolInformation) -> ls_types.UnifiedSymbolInformation:
# Extend this symbol's range
extended = self._extend_nix_symbol_range_to_include_semicolon(symbol, file_content)
# Extend children recursively
if extended.get("children"):
extended["children"] = [extend_symbol_and_children(child) for child in extended["children"]]
return extended
# Apply range extension to all symbols
extended_all_symbols = [extend_symbol_and_children(sym) for sym in all_symbols]
extended_root_symbols = [extend_symbol_and_children(sym) for sym in root_symbols]
return extended_all_symbols, extended_root_symbols
@override
def is_ignored_dirname(self, dirname: str) -> bool:
# For Nix projects, we should ignore:
# - result: nix build output symlinks
# - result-*: multiple build outputs
# - .direnv: direnv cache
return super().is_ignored_dirname(dirname) or dirname in ["result", ".direnv"] or dirname.startswith("result-")
@staticmethod
def _get_nixd_version():
"""Get the installed nixd version or None if not found."""
try:
result = subprocess.run(["nixd", "--version"], capture_output=True, text=True, check=False)
if result.returncode == 0:
# nixd outputs version like: nixd 2.0.0
return result.stdout.strip()
except FileNotFoundError:
return None
return None
@staticmethod
def _check_nixd_installed():
"""Check if nixd is installed in the system."""
return shutil.which("nixd") is not None
@staticmethod
def _get_nixd_path():
"""Get the path to nixd executable."""
# First check if it's in PATH
nixd_path = shutil.which("nixd")
if nixd_path:
return nixd_path
# Check common installation locations
home = Path.home()
possible_paths = [
home / ".local" / "bin" / "nixd",
home / ".serena" / "language_servers" / "nixd" / "nixd",
home / ".nix-profile" / "bin" / "nixd",
Path("/usr/local/bin/nixd"),
Path("/run/current-system/sw/bin/nixd"), # NixOS system profile
Path("/opt/homebrew/bin/nixd"), # Homebrew on Apple Silicon
Path("/usr/local/opt/nixd/bin/nixd"), # Homebrew on Intel Mac
]
# Add Windows-specific paths
if platform.system() == "Windows":
possible_paths.extend(
[
home / "AppData" / "Local" / "nixd" / "nixd.exe",
home / ".serena" / "language_servers" / "nixd" / "nixd.exe",
]
)
for path in possible_paths:
if path.exists():
return str(path)
return None
@staticmethod
def _install_nixd_with_nix():
"""Install nixd using nix if available."""
# Check if nix is available
if not shutil.which("nix"):
return None
print("Installing nixd using nix... This may take a few minutes.")
try:
# Try to install nixd using nix profile
result = subprocess.run(
["nix", "profile", "install", "github:nix-community/nixd"],
capture_output=True,
text=True,
check=False,
timeout=600, # 10 minute timeout for building
)
if result.returncode == 0:
# Check if nixd is now in PATH
nixd_path = shutil.which("nixd")
if nixd_path:
print(f"Successfully installed nixd at: {nixd_path}")
return nixd_path
else:
# Try nix-env as fallback
result = subprocess.run(
["nix-env", "-iA", "nixpkgs.nixd"],
capture_output=True,
text=True,
check=False,
timeout=600,
)
if result.returncode == 0:
nixd_path = shutil.which("nixd")
if nixd_path:
print(f"Successfully installed nixd at: {nixd_path}")
return nixd_path
print(f"Failed to install nixd: {result.stderr}")
except subprocess.TimeoutExpired:
print("Nix install timed out after 10 minutes")
except Exception as e:
print(f"Error installing nixd with nix: {e}")
return None
@staticmethod
def _setup_runtime_dependency():
"""
Check if required Nix runtime dependencies are available.
Attempts to install nixd if not present.
"""
# First check if Nix is available (nixd needs it at runtime)
if not shutil.which("nix"):
print("WARNING: Nix is not installed. nixd requires Nix to function properly.")
raise RuntimeError("Nix is required for nixd. Please install Nix from https://nixos.org/download.html")
nixd_path = NixLanguageServer._get_nixd_path()
if not nixd_path:
print("nixd not found. Attempting to install...")
# Try to install with nix if available
nixd_path = NixLanguageServer._install_nixd_with_nix()
if not nixd_path:
raise RuntimeError(
"nixd (Nix Language Server) is not installed.\n"
"Please install nixd using one of the following methods:\n"
" - Using Nix flakes: nix profile install github:nix-community/nixd\n"
" - From nixpkgs: nix-env -iA nixpkgs.nixd\n"
" - On macOS with Homebrew: brew install nixd\n\n"
"After installation, make sure 'nixd' is in your PATH."
)
# Verify nixd works
try:
result = subprocess.run([nixd_path, "--version"], capture_output=True, text=True, check=False, timeout=5)
if result.returncode != 0:
raise RuntimeError(f"nixd failed to run: {result.stderr}")
except Exception as e:
raise RuntimeError(f"Failed to verify nixd installation: {e}")
return nixd_path
def __init__(
self, config: LanguageServerConfig, logger: LanguageServerLogger, repository_root_path: str, solidlsp_settings: SolidLSPSettings
):
nixd_path = self._setup_runtime_dependency()
super().__init__(
config,
logger,
repository_root_path,
ProcessLaunchInfo(cmd=nixd_path, cwd=repository_root_path),
"nix",
solidlsp_settings,
)
self.server_ready = threading.Event()
self.request_id = 0
@staticmethod
def _get_initialize_params(repository_absolute_path: str) -> InitializeParams:
"""
Returns the initialize params for nixd.
"""
root_uri = pathlib.Path(repository_absolute_path).as_uri()
initialize_params = {
"locale": "en",
"capabilities": {
"textDocument": {
"synchronization": {"didSave": True, "dynamicRegistration": True},
"definition": {"dynamicRegistration": True},
"references": {"dynamicRegistration": True},
"documentSymbol": {
"dynamicRegistration": True,
"hierarchicalDocumentSymbolSupport": True,
"symbolKind": {"valueSet": list(range(1, 27))},
},
"completion": {
"dynamicRegistration": True,
"completionItem": {
"snippetSupport": True,
"commitCharactersSupport": True,
"documentationFormat": ["markdown", "plaintext"],
"deprecatedSupport": True,
"preselectSupport": True,
},
},
"hover": {
"dynamicRegistration": True,
"contentFormat": ["markdown", "plaintext"],
},
"signatureHelp": {
"dynamicRegistration": True,
"signatureInformation": {
"documentationFormat": ["markdown", "plaintext"],
"parameterInformation": {"labelOffsetSupport": True},
},
},
"codeAction": {
"dynamicRegistration": True,
"codeActionLiteralSupport": {
"codeActionKind": {
"valueSet": [
"",
"quickfix",
"refactor",
"refactor.extract",
"refactor.inline",
"refactor.rewrite",
"source",
"source.organizeImports",
]
}
},
},
"rename": {"dynamicRegistration": True, "prepareSupport": True},
},
"workspace": {
"workspaceFolders": True,
"didChangeConfiguration": {"dynamicRegistration": True},
"configuration": True,
"symbol": {
"dynamicRegistration": True,
"symbolKind": {"valueSet": list(range(1, 27))},
},
},
},
"processId": os.getpid(),
"rootPath": repository_absolute_path,
"rootUri": root_uri,
"workspaceFolders": [
{
"uri": root_uri,
"name": os.path.basename(repository_absolute_path),
}
],
"initializationOptions": {
# nixd specific options
"nixpkgs": {"expr": "import <nixpkgs> { }"},
"formatting": {"command": ["nixpkgs-fmt"]}, # or ["alejandra"] or ["nixfmt"]
"options": {
"enable": True,
"target": {
"installable": "", # Will be auto-detected from flake.nix if present
},
},
},
}
return initialize_params
def _start_server(self):
"""Start nixd server process"""
def register_capability_handler(params):
return
def window_log_message(msg):
self.logger.log(f"LSP: window/logMessage: {msg}", logging.INFO)
def do_nothing(params):
return
self.server.on_request("client/registerCapability", register_capability_handler)
self.server.on_notification("window/logMessage", window_log_message)
self.server.on_notification("$/progress", do_nothing)
self.server.on_notification("textDocument/publishDiagnostics", do_nothing)
self.logger.log("Starting nixd server process", logging.INFO)
self.server.start()
initialize_params = self._get_initialize_params(self.repository_root_path)
self.logger.log(
"Sending initialize request from LSP client to LSP server and awaiting response",
logging.INFO,
)
init_response = self.server.send.initialize(initialize_params)
# Verify server capabilities
assert "textDocumentSync" in init_response["capabilities"]
assert "definitionProvider" in init_response["capabilities"]
assert "documentSymbolProvider" in init_response["capabilities"]
assert "referencesProvider" in init_response["capabilities"]
self.server.notify.initialized({})
self.completions_available.set()
# nixd server is typically ready immediately after initialization
self.server_ready.set()
self.server_ready.wait()
src/solidlsp/language_servers/omnisharp.py
+1 -2
View File
@@ -6,7 +6,6 @@ import json
import logging
import os
import pathlib
import stat
import threading
from collections.abc import Iterable
@@ -194,7 +193,7 @@ class OmniSharp(SolidLanguageServer):
FileUtils.download_and_extract_archive(logger, runtime_dependencies["OmniSharp"]["url"], omnisharp_ls_dir, "zip")
omnisharp_executable_path = os.path.join(omnisharp_ls_dir, runtime_dependencies["OmniSharp"]["binaryName"])
assert os.path.exists(omnisharp_executable_path)
os.chmod(omnisharp_executable_path, stat.S_IEXEC)
os.chmod(omnisharp_executable_path, 0o755)
razor_omnisharp_ls_dir = os.path.join(cls.ls_resources_dir(solidlsp_settings), "RazorOmnisharp")
if not os.path.exists(razor_omnisharp_ls_dir):
src/solidlsp/language_servers/perl_language_server.py
+225
View File
@@ -0,0 +1,225 @@
"""
Provides Perl specific instantiation of the LanguageServer class using Perl::LanguageServer.
Note: Windows is not supported as Nix itself doesn't support Windows natively.
"""
import logging
import os
import pathlib
import subprocess
import time
from overrides import override
from solidlsp.ls import SolidLanguageServer
from solidlsp.ls_config import LanguageServerConfig
from solidlsp.ls_logger import LanguageServerLogger
from solidlsp.ls_utils import PlatformId, PlatformUtils
from solidlsp.lsp_protocol_handler.lsp_types import InitializeParams
from solidlsp.lsp_protocol_handler.server import ProcessLaunchInfo
from solidlsp.settings import SolidLSPSettings
class PerlLanguageServer(SolidLanguageServer):
"""
Provides Perl specific instantiation of the LanguageServer class using Perl::LanguageServer.
"""
@staticmethod
def _get_perl_version():
"""Get the installed Perl version or None if not found."""
try:
result = subprocess.run(["perl", "-v"], capture_output=True, text=True, check=False)
if result.returncode == 0:
return result.stdout.strip()
except FileNotFoundError:
return None
return None
@staticmethod
def _get_perl_language_server_version():
"""Get the installed Perl::LanguageServer version or None if not found."""
try:
result = subprocess.run(
["perl", "-MPerl::LanguageServer", "-e", "print $Perl::LanguageServer::VERSION"],
capture_output=True,
text=True,
check=False,
)
if result.returncode == 0:
return result.stdout.strip()
except FileNotFoundError:
return None
return None
@override
def is_ignored_dirname(self, dirname: str) -> bool:
# For Perl projects, we should ignore:
# - blib: build library directory
# - local: local Perl module installation
# - .carton: Carton dependency manager cache
# - vendor: vendored dependencies
# - _build: Module::Build output
return super().is_ignored_dirname(dirname) or dirname in ["blib", "local", ".carton", "vendor", "_build", "cover_db"]
@classmethod
def _setup_runtime_dependencies(cls) -> str:
"""
Check if required Perl runtime dependencies are available.
Raises RuntimeError with helpful message if dependencies are missing.
"""
platform_id = PlatformUtils.get_platform_id()
valid_platforms = [
PlatformId.LINUX_x64,
PlatformId.LINUX_arm64,
PlatformId.OSX,
PlatformId.OSX_x64,
PlatformId.OSX_arm64,
]
if platform_id not in valid_platforms:
raise RuntimeError(f"Platform {platform_id} is not supported for Perl at the moment")
perl_version = cls._get_perl_version()
if not perl_version:
raise RuntimeError(
"Perl is not installed. Please install Perl from https://www.perl.org/get.html and make sure it is added to your PATH."
)
perl_ls_version = cls._get_perl_language_server_version()
if not perl_ls_version:
raise RuntimeError(
"Found a Perl version but Perl::LanguageServer is not installed.\n"
"Please install Perl::LanguageServer: cpanm Perl::LanguageServer\n"
"See: https://metacpan.org/pod/Perl::LanguageServer"
)
return "perl -MPerl::LanguageServer -e 'Perl::LanguageServer::run'"
def __init__(
self, config: LanguageServerConfig, logger: LanguageServerLogger, repository_root_path: str, solidlsp_settings: SolidLSPSettings
):
# Setup runtime dependencies before initializing
perl_ls_cmd = self._setup_runtime_dependencies()
super().__init__(
config,
logger,
repository_root_path,
ProcessLaunchInfo(cmd=perl_ls_cmd, cwd=repository_root_path),
"perl",
solidlsp_settings,
)
self.request_id = 0
@staticmethod
def _get_initialize_params(repository_absolute_path: str) -> InitializeParams:
"""
Returns the initialize params for Perl::LanguageServer.
Based on the expected structure from Perl::LanguageServer::Methods::_rpcreq_initialize.
"""
root_uri = pathlib.Path(repository_absolute_path).as_uri()
initialize_params = {
"processId": os.getpid(),
"rootPath": repository_absolute_path,
"rootUri": root_uri,
"capabilities": {
"textDocument": {
"synchronization": {"didSave": True, "dynamicRegistration": True},
"definition": {"dynamicRegistration": True},
"references": {"dynamicRegistration": True},
"documentSymbol": {"dynamicRegistration": True},
"hover": {"dynamicRegistration": True},
},
"workspace": {
"workspaceFolders": True,
"didChangeConfiguration": {"dynamicRegistration": True},
"symbol": {"dynamicRegistration": True},
},
},
"initializationOptions": {},
"workspaceFolders": [
{
"uri": root_uri,
"name": os.path.basename(repository_absolute_path),
}
],
}
return initialize_params
def _start_server(self):
"""Start Perl::LanguageServer process"""
def register_capability_handler(params):
return
def window_log_message(msg):
self.logger.log(f"LSP: window/logMessage: {msg}", logging.INFO)
def do_nothing(params):
return
def workspace_configuration_handler(params):
"""Handle workspace/configuration request from Perl::LanguageServer."""
self.logger.log(f"Received workspace/configuration request: {params}", logging.INFO)
perl_config = {
"perlInc": [self.repository_root_path, "."],
"fileFilter": [".pm", ".pl"],
"ignoreDirs": [".git", ".svn", "blib", "local", ".carton", "vendor", "_build", "cover_db"],
}
return [perl_config]
self.server.on_request("client/registerCapability", register_capability_handler)
self.server.on_request("workspace/configuration", workspace_configuration_handler)
self.server.on_notification("window/logMessage", window_log_message)
self.server.on_notification("$/progress", do_nothing)
self.server.on_notification("textDocument/publishDiagnostics", do_nothing)
self.logger.log("Starting Perl::LanguageServer process", logging.INFO)
self.server.start()
initialize_params = self._get_initialize_params(self.repository_root_path)
self.logger.log(
"Sending initialize request from LSP client to LSP server and awaiting response",
logging.INFO,
)
init_response = self.server.send.initialize(initialize_params)
self.logger.log(
"After sent initialize params",
logging.INFO,
)
# Verify server capabilities
assert "textDocumentSync" in init_response["capabilities"]
assert "definitionProvider" in init_response["capabilities"]
assert "referencesProvider" in init_response["capabilities"]
self.server.notify.initialized({})
# Send workspace configuration to Perl::LanguageServer
# Perl::LanguageServer requires didChangeConfiguration to set perlInc, fileFilter, and ignoreDirs
# See: Perl::LanguageServer::Methods::workspace::_rpcnot_didChangeConfiguration
perl_config = {
"settings": {
"perl": {
"perlInc": [self.repository_root_path, "."],
"fileFilter": [".pm", ".pl"],
"ignoreDirs": [".git", ".svn", "blib", "local", ".carton", "vendor", "_build", "cover_db"],
}
}
}
self.logger.log(f"Sending workspace/didChangeConfiguration notification with config: {perl_config}", logging.INFO)
self.server.notify.workspace_did_change_configuration(perl_config)
self.completions_available.set()
# Perl::LanguageServer needs time to index files and resolve cross-file references
# Without this delay, requests for definitions/references may return empty results
settling_time = 0.5
self.logger.log(f"Allowing {settling_time} seconds for Perl::LanguageServer to index files...", logging.INFO)
time.sleep(settling_time)
self.logger.log("Perl::LanguageServer settling period complete", logging.INFO)
src/solidlsp/language_servers/r_language_server.py
+172
View File
@@ -0,0 +1,172 @@
import logging
import os
import pathlib
import subprocess
import threading
from overrides import override
from solidlsp.ls import SolidLanguageServer
from solidlsp.ls_config import LanguageServerConfig
from solidlsp.ls_logger import LanguageServerLogger
from solidlsp.lsp_protocol_handler.lsp_types import InitializeParams
from solidlsp.lsp_protocol_handler.server import ProcessLaunchInfo
from solidlsp.settings import SolidLSPSettings
class RLanguageServer(SolidLanguageServer):
"""R Language Server implementation using the languageserver R package."""
@override
def _get_wait_time_for_cross_file_referencing(self) -> float:
return 5.0 # R language server needs extra time for workspace indexing in CI environments
@override
def is_ignored_dirname(self, dirname: str) -> bool:
# For R projects, ignore common directories
return super().is_ignored_dirname(dirname) or dirname in [
"renv", # R environment management
"packrat", # Legacy R package management
".Rproj.user", # RStudio project files
"vignettes", # Package vignettes (often large)
]
@staticmethod
def _check_r_installation():
"""Check if R and languageserver are available."""
try:
# Check R installation
result = subprocess.run(["R", "--version"], capture_output=True, text=True, check=False)
if result.returncode != 0:
raise RuntimeError("R is not installed or not in PATH")
# Check languageserver package
result = subprocess.run(
["R", "--vanilla", "--quiet", "--slave", "-e", "if (!require('languageserver', quietly=TRUE)) quit(status=1)"],
capture_output=True,
text=True,
check=False,
)
if result.returncode != 0:
raise RuntimeError(
"R languageserver package is not installed.\nInstall it with: R -e \"install.packages('languageserver')\""
)
except FileNotFoundError:
raise RuntimeError("R is not installed. Please install R from https://www.r-project.org/")
def __init__(
self, config: LanguageServerConfig, logger: LanguageServerLogger, repository_root_path: str, solidlsp_settings: SolidLSPSettings
):
# Check R installation
self._check_r_installation()
# R command to start language server
# Use --vanilla for minimal startup and --quiet to suppress all output except LSP
# Set specific options to improve parsing stability
r_cmd = 'R --vanilla --quiet --slave -e "options(languageserver.debug_mode = FALSE); languageserver::run()"'
super().__init__(
config,
logger,
repository_root_path,
ProcessLaunchInfo(cmd=r_cmd, cwd=repository_root_path),
"r",
solidlsp_settings,
)
self.server_ready = threading.Event()
@staticmethod
def _get_initialize_params(repository_absolute_path: str) -> InitializeParams:
"""Initialize params for R Language Server."""
root_uri = pathlib.Path(repository_absolute_path).as_uri()
initialize_params = {
"locale": "en",
"capabilities": {
"textDocument": {
"synchronization": {"didSave": True, "dynamicRegistration": True},
"completion": {
"dynamicRegistration": True,
"completionItem": {
"snippetSupport": True,
"commitCharactersSupport": True,
"documentationFormat": ["markdown", "plaintext"],
"deprecatedSupport": True,
"preselectSupport": True,
},
},
"hover": {"dynamicRegistration": True, "contentFormat": ["markdown", "plaintext"]},
"definition": {"dynamicRegistration": True},
"references": {"dynamicRegistration": True},
"documentSymbol": {
"dynamicRegistration": True,
"hierarchicalDocumentSymbolSupport": True,
"symbolKind": {"valueSet": list(range(1, 27))},
},
"formatting": {"dynamicRegistration": True},
"rangeFormatting": {"dynamicRegistration": True},
},
"workspace": {
"workspaceFolders": True,
"didChangeConfiguration": {"dynamicRegistration": True},
"symbol": {
"dynamicRegistration": True,
"symbolKind": {"valueSet": list(range(1, 27))},
},
},
},
"processId": os.getpid(),
"rootPath": repository_absolute_path,
"rootUri": root_uri,
"workspaceFolders": [
{
"uri": root_uri,
"name": os.path.basename(repository_absolute_path),
}
],
}
return initialize_params
def _start_server(self):
"""Start R Language Server process."""
def window_log_message(msg):
self.logger.log(f"R LSP: window/logMessage: {msg}", logging.INFO)
def do_nothing(params):
return
def register_capability_handler(params):
return
# Register LSP message handlers
self.server.on_request("client/registerCapability", register_capability_handler)
self.server.on_notification("window/logMessage", window_log_message)
self.server.on_notification("$/progress", do_nothing)
self.server.on_notification("textDocument/publishDiagnostics", do_nothing)
self.logger.log("Starting R Language Server process", logging.INFO)
self.server.start()
initialize_params = self._get_initialize_params(self.repository_root_path)
self.logger.log(
"Sending initialize request to R Language Server",
logging.INFO,
)
init_response = self.server.send.initialize(initialize_params)
# Verify server capabilities
capabilities = init_response.get("capabilities", {})
assert "textDocumentSync" in capabilities
if "completionProvider" in capabilities:
self.logger.log("R LSP completion provider available", logging.INFO)
if "definitionProvider" in capabilities:
self.logger.log("R LSP definition provider available", logging.INFO)
self.server.notify.initialized({})
self.completions_available.set()
# R Language Server is ready after initialization
self.server_ready.set()
src/solidlsp/language_servers/regal_server.py
+140
View File
@@ -0,0 +1,140 @@
"""Regal Language Server implementation for Rego policy files."""
import logging
import os
import shutil
import threading
from overrides import override
from solidlsp.ls import SolidLanguageServer
from solidlsp.ls_config import LanguageServerConfig
from solidlsp.ls_logger import LanguageServerLogger
from solidlsp.ls_utils import PathUtils
from solidlsp.lsp_protocol_handler.lsp_types import InitializeParams
from solidlsp.lsp_protocol_handler.server import ProcessLaunchInfo
from solidlsp.settings import SolidLSPSettings
class RegalLanguageServer(SolidLanguageServer):
"""
Provides Rego specific instantiation of the LanguageServer class using Regal.
Regal is the official linter and language server for Rego (Open Policy Agent's policy language).
See: https://github.com/StyraInc/regal
"""
@override
def is_ignored_dirname(self, dirname: str) -> bool:
return super().is_ignored_dirname(dirname) or dirname in [".regal", ".opa"]
def __init__(
self, config: LanguageServerConfig, logger: LanguageServerLogger, repository_root_path: str, solidlsp_settings: SolidLSPSettings
):
"""
Creates a RegalLanguageServer instance.
This class is not meant to be instantiated directly. Use LanguageServer.create() instead.
:param config: Language server configuration
:param logger: Logger instance
:param repository_root_path: Path to the repository root
:param solidlsp_settings: Settings for solidlsp
"""
# Regal should be installed system-wide (via CI or user installation)
regal_executable_path = shutil.which("regal")
if not regal_executable_path:
raise RuntimeError(
"Regal language server not found. Please install it from https://github.com/StyraInc/regal or via your package manager."
)
super().__init__(
config,
logger,
repository_root_path,
ProcessLaunchInfo(cmd=f"{regal_executable_path} language-server", cwd=repository_root_path),
"rego",
solidlsp_settings,
)
self.server_ready = threading.Event()
@staticmethod
def _get_initialize_params(repository_absolute_path: str) -> InitializeParams:
"""
Returns the initialize params for the Regal Language Server.
:param repository_absolute_path: Absolute path to the repository
:return: LSP initialization parameters
"""
root_uri = PathUtils.path_to_uri(repository_absolute_path)
return {
"processId": os.getpid(),
"locale": "en",
"rootPath": repository_absolute_path,
"rootUri": root_uri,
"capabilities": {
"textDocument": {
"synchronization": {"didSave": True, "dynamicRegistration": True},
"completion": {"dynamicRegistration": True, "completionItem": {"snippetSupport": True}},
"definition": {"dynamicRegistration": True},
"references": {"dynamicRegistration": True},
"documentSymbol": {
"dynamicRegistration": True,
"hierarchicalDocumentSymbolSupport": True,
"symbolKind": {"valueSet": list(range(1, 27))}, # type: ignore[arg-type]
},
"hover": {"dynamicRegistration": True, "contentFormat": ["markdown", "plaintext"]}, # type: ignore[list-item]
"codeAction": {"dynamicRegistration": True},
"formatting": {"dynamicRegistration": True},
},
"workspace": {
"workspaceFolders": True,
"didChangeConfiguration": {"dynamicRegistration": True},
"symbol": {"dynamicRegistration": True},
},
},
"workspaceFolders": [
{
"name": os.path.basename(repository_absolute_path),
"uri": root_uri,
}
],
}
def _start_server(self) -> None:
"""Start Regal language server process and wait for initialization."""
def register_capability_handler(params) -> None: # type: ignore[no-untyped-def]
return
def window_log_message(msg) -> None: # type: ignore[no-untyped-def]
self.logger.log(f"LSP: window/logMessage: {msg}", logging.INFO)
def do_nothing(params) -> None: # type: ignore[no-untyped-def]
return
self.server.on_request("client/registerCapability", register_capability_handler)
self.server.on_notification("window/logMessage", window_log_message)
self.server.on_notification("$/progress", do_nothing)
self.server.on_notification("textDocument/publishDiagnostics", do_nothing)
self.logger.log("Starting Regal language server process", logging.INFO)
self.server.start()
initialize_params = self._get_initialize_params(self.repository_root_path)
self.logger.log(
"Sending initialize request from LSP client to LSP server and awaiting response",
logging.INFO,
)
init_response = self.server.send.initialize(initialize_params)
# Verify server capabilities
assert "capabilities" in init_response
assert "textDocumentSync" in init_response["capabilities"]
self.server.notify.initialized({})
self.completions_available.set()
# Regal server is ready immediately after initialization
self.server_ready.set()
self.server_ready.wait()
src/solidlsp/language_servers/ruby_lsp.py
+477
View File
@@ -0,0 +1,477 @@
"""
Ruby LSP Language Server implementation using Shopify's ruby-lsp.
Provides modern Ruby language server capabilities with improved performance.
"""
import json
import logging
import os
import pathlib
import shutil
import subprocess
import threading
from overrides import override
from solidlsp.ls import SolidLanguageServer
from solidlsp.ls_config import LanguageServerConfig
from solidlsp.ls_logger import LanguageServerLogger
from solidlsp.lsp_protocol_handler.lsp_types import InitializeParams
from solidlsp.lsp_protocol_handler.server import ProcessLaunchInfo
from solidlsp.settings import SolidLSPSettings
class RubyLsp(SolidLanguageServer):
"""
Provides Ruby specific instantiation of the LanguageServer class using ruby-lsp.
Contains various configurations and settings specific to Ruby with modern LSP features.
"""
def __init__(
self, config: LanguageServerConfig, logger: LanguageServerLogger, repository_root_path: str, solidlsp_settings: SolidLSPSettings
):
"""
Creates a RubyLsp instance. This class is not meant to be instantiated directly.
Use LanguageServer.create() instead.
"""
ruby_lsp_executable = self._setup_runtime_dependencies(logger, config, repository_root_path)
super().__init__(
config,
logger,
repository_root_path,
ProcessLaunchInfo(cmd=ruby_lsp_executable, cwd=repository_root_path),
"ruby",
solidlsp_settings,
)
self.analysis_complete = threading.Event()
self.service_ready_event = threading.Event()
# Set timeout for ruby-lsp requests - ruby-lsp is fast
self.set_request_timeout(30.0) # 30 seconds for initialization and requests
@override
def is_ignored_dirname(self, dirname: str) -> bool:
"""Override to ignore Ruby-specific directories that cause performance issues."""
ruby_ignored_dirs = [
"vendor", # Ruby vendor directory
".bundle", # Bundler cache
"tmp", # Temporary files
"log", # Log files
"coverage", # Test coverage reports
".yardoc", # YARD documentation cache
"doc", # Generated documentation
"node_modules", # Node modules (for Rails with JS)
"storage", # Active Storage files (Rails)
"public/packs", # Webpacker output
"public/webpack", # Webpack output
"public/assets", # Rails compiled assets
]
return super().is_ignored_dirname(dirname) or dirname in ruby_ignored_dirs
@override
def _get_wait_time_for_cross_file_referencing(self) -> float:
"""Override to provide optimal wait time for ruby-lsp cross-file reference resolution.
ruby-lsp typically initializes quickly, but may need a brief moment
for cross-file analysis in larger projects.
"""
return 0.5 # 500ms should be sufficient for ruby-lsp
@staticmethod
def _find_executable_with_extensions(executable_name: str) -> str | None:
"""
Find executable with Windows-specific extensions (.bat, .cmd, .exe) if on Windows.
Returns the full path to the executable or None if not found.
"""
import platform
if platform.system() == "Windows":
# Try Windows-specific extensions first
for ext in [".bat", ".cmd", ".exe"]:
path = shutil.which(f"{executable_name}{ext}")
if path:
return path
# Fall back to default search
return shutil.which(executable_name)
else:
# Unix systems
return shutil.which(executable_name)
@staticmethod
def _setup_runtime_dependencies(logger: LanguageServerLogger, config: LanguageServerConfig, repository_root_path: str) -> list[str]:
"""
Setup runtime dependencies for ruby-lsp and return the command list to start the server.
Installation strategy: Bundler project > global ruby-lsp > gem install ruby-lsp
"""
# Detect rbenv-managed Ruby environment
# When .ruby-version exists, it indicates the project uses rbenv for version management.
# rbenv automatically reads .ruby-version to determine which Ruby version to use.
# Using "rbenv exec" ensures commands run with the correct Ruby version and its gems.
#
# Why rbenv is preferred over system Ruby:
# - Respects project-specific Ruby versions
# - Avoids bundler version mismatches between system and project
# - Ensures consistent environment across developers
#
# Fallback behavior:
# If .ruby-version doesn't exist or rbenv isn't installed, we fall back to system Ruby.
# This may cause issues if:
# - System Ruby version differs from what the project expects
# - System bundler version is incompatible with Gemfile.lock
# - Project gems aren't installed in system Ruby
ruby_version_file = os.path.join(repository_root_path, ".ruby-version")
use_rbenv = os.path.exists(ruby_version_file) and shutil.which("rbenv") is not None
if use_rbenv:
ruby_cmd = ["rbenv", "exec", "ruby"]
bundle_cmd = ["rbenv", "exec", "bundle"]
logger.log(f"Using rbenv-managed Ruby (found {ruby_version_file})", logging.INFO)
else:
ruby_cmd = ["ruby"]
bundle_cmd = ["bundle"]
if os.path.exists(ruby_version_file):
logger.log(
f"Found {ruby_version_file} but rbenv is not installed. "
"Using system Ruby. Consider installing rbenv for better version management: https://github.com/rbenv/rbenv",
logging.WARNING,
)
else:
logger.log("No .ruby-version file found, using system Ruby", logging.INFO)
# Check if Ruby is installed
try:
result = subprocess.run(ruby_cmd + ["--version"], check=True, capture_output=True, cwd=repository_root_path, text=True)
ruby_version = result.stdout.strip()
logger.log(f"Ruby version: {ruby_version}", logging.INFO)
# Extract version number for compatibility checks
import re
version_match = re.search(r"ruby (\d+)\.(\d+)\.(\d+)", ruby_version)
if version_match:
major, minor, patch = map(int, version_match.groups())
if major < 2 or (major == 2 and minor < 6):
logger.log(f"Warning: Ruby {major}.{minor}.{patch} detected. ruby-lsp works best with Ruby 2.6+", logging.WARNING)
except subprocess.CalledProcessError as e:
error_msg = e.stderr if isinstance(e.stderr, str) else e.stderr.decode() if e.stderr else "Unknown error"
raise RuntimeError(
f"Error checking Ruby installation: {error_msg}. Please ensure Ruby is properly installed and in PATH."
) from e
except FileNotFoundError as e:
raise RuntimeError(
"Ruby is not installed or not found in PATH. Please install Ruby using one of these methods:\n"
" - Using rbenv: rbenv install 3.0.0 && rbenv global 3.0.0\n"
" - Using RVM: rvm install 3.0.0 && rvm use 3.0.0 --default\n"
" - Using asdf: asdf install ruby 3.0.0 && asdf global ruby 3.0.0\n"
" - System package manager (brew install ruby, apt install ruby, etc.)"
) from e
# Check for Bundler project (Gemfile exists)
gemfile_path = os.path.join(repository_root_path, "Gemfile")
gemfile_lock_path = os.path.join(repository_root_path, "Gemfile.lock")
is_bundler_project = os.path.exists(gemfile_path)
if is_bundler_project:
logger.log("Detected Bundler project (Gemfile found)", logging.INFO)
# Check if bundle command is available using Windows-compatible search
bundle_path = RubyLsp._find_executable_with_extensions(bundle_cmd[0] if len(bundle_cmd) == 1 else "bundle")
if not bundle_path:
# Try common bundle executables
for bundle_executable in ["bin/bundle", "bundle"]:
if bundle_executable.startswith("bin/"):
bundle_full_path = os.path.join(repository_root_path, bundle_executable)
else:
bundle_full_path = RubyLsp._find_executable_with_extensions(bundle_executable)
if bundle_full_path and os.path.exists(bundle_full_path):
bundle_path = bundle_full_path if bundle_executable.startswith("bin/") else bundle_executable
break
if not bundle_path:
logger.log(
"Bundler project detected but 'bundle' command not found. Falling back to global ruby-lsp installation.",
logging.WARNING,
)
else:
# Check if ruby-lsp is in Gemfile.lock
ruby_lsp_in_bundle = False
if os.path.exists(gemfile_lock_path):
try:
with open(gemfile_lock_path) as f:
content = f.read()
ruby_lsp_in_bundle = "ruby-lsp" in content.lower()
except Exception as e:
logger.log(f"Warning: Could not read Gemfile.lock: {e}", logging.WARNING)
if ruby_lsp_in_bundle:
logger.log("Found ruby-lsp in Gemfile.lock", logging.INFO)
return bundle_cmd + ["exec", "ruby-lsp"]
else:
logger.log(
"ruby-lsp not found in Gemfile.lock. Consider adding 'gem \"ruby-lsp\"' to your Gemfile for better compatibility.",
logging.INFO,
)
# Fall through to global installation check
# Check if ruby-lsp is available globally using Windows-compatible search
ruby_lsp_path = RubyLsp._find_executable_with_extensions("ruby-lsp")
if ruby_lsp_path:
logger.log(f"Found ruby-lsp at: {ruby_lsp_path}", logging.INFO)
return [ruby_lsp_path]
# Try to install ruby-lsp globally
logger.log("ruby-lsp not found, attempting to install globally...", logging.INFO)
try:
subprocess.run(["gem", "install", "ruby-lsp"], check=True, capture_output=True, cwd=repository_root_path)
logger.log("Successfully installed ruby-lsp globally", logging.INFO)
# Find the newly installed ruby-lsp executable
ruby_lsp_path = RubyLsp._find_executable_with_extensions("ruby-lsp")
return [ruby_lsp_path] if ruby_lsp_path else ["ruby-lsp"]
except subprocess.CalledProcessError as e:
error_msg = e.stderr if isinstance(e.stderr, str) else e.stderr.decode() if e.stderr else str(e)
if is_bundler_project:
raise RuntimeError(
f"Failed to install ruby-lsp globally: {error_msg}\n"
"For Bundler projects, please add 'gem \"ruby-lsp\"' to your Gemfile and run 'bundle install'.\n"
"Alternatively, install globally: gem install ruby-lsp"
) from e
raise RuntimeError(f"Failed to install ruby-lsp: {error_msg}\nPlease try installing manually: gem install ruby-lsp") from e
@staticmethod
def _detect_rails_project(repository_root_path: str) -> bool:
"""
Detect if this is a Rails project by checking for Rails-specific files.
"""
rails_indicators = [
"config/application.rb",
"config/environment.rb",
"app/controllers/application_controller.rb",
"Rakefile",
]
for indicator in rails_indicators:
if os.path.exists(os.path.join(repository_root_path, indicator)):
return True
# Check for Rails in Gemfile
gemfile_path = os.path.join(repository_root_path, "Gemfile")
if os.path.exists(gemfile_path):
try:
with open(gemfile_path) as f:
content = f.read().lower()
if "gem 'rails'" in content or 'gem "rails"' in content:
return True
except Exception:
pass
return False
@staticmethod
def _get_ruby_exclude_patterns(repository_root_path: str) -> list[str]:
"""
Get Ruby and Rails-specific exclude patterns for better performance.
"""
base_patterns = [
"**/vendor/**", # Ruby vendor directory
"**/.bundle/**", # Bundler cache
"**/tmp/**", # Temporary files
"**/log/**", # Log files
"**/coverage/**", # Test coverage reports
"**/.yardoc/**", # YARD documentation cache
"**/doc/**", # Generated documentation
"**/.git/**", # Git directory
"**/node_modules/**", # Node modules (for Rails with JS)
"**/public/assets/**", # Rails compiled assets
]
# Add Rails-specific patterns if this is a Rails project
if RubyLsp._detect_rails_project(repository_root_path):
base_patterns.extend(
[
"**/app/assets/builds/**", # Rails 7+ CSS builds
"**/storage/**", # Active Storage
"**/public/packs/**", # Webpacker
"**/public/webpack/**", # Webpack
]
)
return base_patterns
def _get_initialize_params(self) -> InitializeParams:
"""
Returns ruby-lsp specific initialization parameters.
"""
exclude_patterns = self._get_ruby_exclude_patterns(self.repository_root_path)
initialize_params = {
"processId": os.getpid(),
"rootPath": self.repository_root_path,
"rootUri": pathlib.Path(self.repository_root_path).as_uri(),
"capabilities": {
"workspace": {
"workspaceEdit": {"documentChanges": True},
"configuration": True,
},
"window": {
"workDoneProgress": True,
},
"textDocument": {
"documentSymbol": {
"hierarchicalDocumentSymbolSupport": True,
"symbolKind": {"valueSet": list(range(1, 27))},
},
"completion": {
"completionItem": {
"snippetSupport": True,
"commitCharactersSupport": True,
}
},
},
},
"initializationOptions": {
# ruby-lsp enables all features by default, so we don't need to specify enabledFeatures
"experimentalFeaturesEnabled": False,
"featuresConfiguration": {},
"indexing": {
"includedPatterns": ["**/*.rb", "**/*.rake", "**/*.ru", "**/*.erb"],
"excludedPatterns": exclude_patterns,
},
},
}
return initialize_params
def _start_server(self) -> None:
"""
Starts the ruby-lsp Language Server for Ruby
"""
def register_capability_handler(params: dict) -> None:
assert "registrations" in params
for registration in params["registrations"]:
self.logger.log(f"Registered capability: {registration['method']}", logging.INFO)
return
def lang_status_handler(params: dict) -> None:
self.logger.log(f"LSP: language/status: {params}", logging.INFO)
if params.get("type") == "ready":
self.logger.log("ruby-lsp service is ready.", logging.INFO)
self.analysis_complete.set()
self.completions_available.set()
def execute_client_command_handler(params: dict) -> list:
return []
def do_nothing(params: dict) -> None:
return
def window_log_message(msg: dict) -> None:
self.logger.log(f"LSP: window/logMessage: {msg}", logging.INFO)
def progress_handler(params: dict) -> None:
# ruby-lsp sends progress notifications during indexing
self.logger.log(f"LSP: $/progress: {params}", logging.DEBUG)
if "value" in params:
value = params["value"]
# Check for completion indicators
if value.get("kind") == "end":
self.logger.log("ruby-lsp indexing complete ($/progress end)", logging.INFO)
self.analysis_complete.set()
self.completions_available.set()
elif value.get("kind") == "begin":
self.logger.log("ruby-lsp indexing started ($/progress begin)", logging.INFO)
elif "percentage" in value:
percentage = value.get("percentage", 0)
self.logger.log(f"ruby-lsp indexing progress: {percentage}%", logging.DEBUG)
# Handle direct progress format (fallback)
elif "token" in params and "value" in params:
token = params.get("token")
if isinstance(token, str) and "indexing" in token.lower():
value = params.get("value", {})
if value.get("kind") == "end" or value.get("percentage") == 100:
self.logger.log("ruby-lsp indexing complete (token progress)", logging.INFO)
self.analysis_complete.set()
self.completions_available.set()
def window_work_done_progress_create(params: dict) -> None:
"""Handle workDoneProgress/create requests from ruby-lsp"""
self.logger.log(f"LSP: window/workDoneProgress/create: {params}", logging.DEBUG)
return {}
self.server.on_request("client/registerCapability", register_capability_handler)
self.server.on_notification("language/status", lang_status_handler)
self.server.on_notification("window/logMessage", window_log_message)
self.server.on_request("workspace/executeClientCommand", execute_client_command_handler)
self.server.on_notification("$/progress", progress_handler)
self.server.on_request("window/workDoneProgress/create", window_work_done_progress_create)
self.server.on_notification("textDocument/publishDiagnostics", do_nothing)
self.logger.log("Starting ruby-lsp server process", logging.INFO)
self.server.start()
initialize_params = self._get_initialize_params()
self.logger.log(
"Sending initialize request from LSP client to LSP server and awaiting response",
logging.INFO,
)
self.logger.log(f"Sending init params: {json.dumps(initialize_params, indent=4)}", logging.INFO)
init_response = self.server.send.initialize(initialize_params)
self.logger.log(f"Received init response: {init_response}", logging.INFO)
# Verify expected capabilities
# Note: ruby-lsp may return textDocumentSync in different formats (number or object)
text_document_sync = init_response["capabilities"].get("textDocumentSync")
if isinstance(text_document_sync, int):
assert text_document_sync in [1, 2], f"Unexpected textDocumentSync value: {text_document_sync}"
elif isinstance(text_document_sync, dict):
# ruby-lsp returns an object with change property
assert "change" in text_document_sync, "textDocumentSync object should have 'change' property"
assert "completionProvider" in init_response["capabilities"]
self.server.notify.initialized({})
# Wait for ruby-lsp to complete its initial indexing
# ruby-lsp has fast indexing
self.logger.log("Waiting for ruby-lsp to complete initial indexing...", logging.INFO)
if self.analysis_complete.wait(timeout=30.0):
self.logger.log("ruby-lsp initial indexing complete, server ready", logging.INFO)
else:
self.logger.log("Timeout waiting for ruby-lsp indexing completion, proceeding anyway", logging.WARNING)
# Fallback: assume indexing is complete after timeout
self.analysis_complete.set()
self.completions_available.set()
def _handle_initialization_response(self, init_response):
"""
Handle the initialization response from ruby-lsp and validate capabilities.
"""
if "capabilities" in init_response:
capabilities = init_response["capabilities"]
# Validate textDocumentSync (ruby-lsp may return different formats)
text_document_sync = capabilities.get("textDocumentSync")
if isinstance(text_document_sync, int):
assert text_document_sync in [1, 2], f"Unexpected textDocumentSync value: {text_document_sync}"
elif isinstance(text_document_sync, dict):
# ruby-lsp returns an object with change property
assert "change" in text_document_sync, "textDocumentSync object should have 'change' property"
# Log important capabilities
important_capabilities = [
"completionProvider",
"hoverProvider",
"definitionProvider",
"referencesProvider",
"documentSymbolProvider",
"codeActionProvider",
"documentFormattingProvider",
"semanticTokensProvider",
]
for cap in important_capabilities:
if cap in capabilities:
self.logger.log(f"ruby-lsp {cap}: available", logging.DEBUG)
# Signal that the service is ready
self.service_ready_event.set()
src/solidlsp/language_servers/rust_analyzer.py
+57 -64
View File
@@ -5,7 +5,8 @@ Provides Rust specific instantiation of the LanguageServer class. Contains vario
import logging
import os
import pathlib
import stat
import shutil
import subprocess
import threading
from typing import cast
@@ -18,21 +19,72 @@ from solidlsp.lsp_protocol_handler.lsp_types import InitializeParams
from solidlsp.lsp_protocol_handler.server import ProcessLaunchInfo
from solidlsp.settings import SolidLSPSettings
from .common import RuntimeDependency, RuntimeDependencyCollection
class RustAnalyzer(SolidLanguageServer):
"""
Provides Rust specific instantiation of the LanguageServer class. Contains various configurations and settings specific to Rust.
"""
@staticmethod
def _get_rustup_version():
"""Get installed rustup version or None if not found."""
try:
result = subprocess.run(["rustup", "--version"], capture_output=True, text=True, check=False)
if result.returncode == 0:
return result.stdout.strip()
except FileNotFoundError:
return None
return None
@staticmethod
def _get_rust_analyzer_path():
"""Get rust-analyzer path via rustup or system PATH."""
# First try rustup
try:
result = subprocess.run(["rustup", "which", "rust-analyzer"], capture_output=True, text=True, check=False)
if result.returncode == 0:
return result.stdout.strip()
except FileNotFoundError:
pass
# Fallback to system PATH
return shutil.which("rust-analyzer")
@staticmethod
def _ensure_rust_analyzer_installed():
"""Ensure rust-analyzer is available, install via rustup if needed."""
path = RustAnalyzer._get_rust_analyzer_path()
if path:
return path
# Check if rustup is available
if not RustAnalyzer._get_rustup_version():
raise RuntimeError(
"Neither rust-analyzer nor rustup is installed.\n"
"Please install Rust via https://rustup.rs/ or install rust-analyzer separately."
)
# Try to install rust-analyzer component
result = subprocess.run(["rustup", "component", "add", "rust-analyzer"], check=False, capture_output=True, text=True)
if result.returncode != 0:
raise RuntimeError(f"Failed to install rust-analyzer via rustup: {result.stderr}")
# Try again after installation
path = RustAnalyzer._get_rust_analyzer_path()
if not path:
raise RuntimeError("rust-analyzer installation succeeded but binary not found in PATH")
return path
def __init__(
self, config: LanguageServerConfig, logger: LanguageServerLogger, repository_root_path: str, solidlsp_settings: SolidLSPSettings
):
"""
Creates a RustAnalyzer instance. This class is not meant to be instantiated directly. Use LanguageServer.create() instead.
"""
rustanalyzer_executable_path = self._setup_runtime_dependencies(logger, config, solidlsp_settings)
rustanalyzer_executable_path = self._ensure_rust_analyzer_installed()
logger.log(f"Using rust-analyzer at: {rustanalyzer_executable_path}", logging.INFO)
super().__init__(
config,
logger,
@@ -50,65 +102,6 @@ class RustAnalyzer(SolidLanguageServer):
def is_ignored_dirname(self, dirname: str) -> bool:
return super().is_ignored_dirname(dirname) or dirname in ["target"]
@classmethod
def _setup_runtime_dependencies(
cls, logger: LanguageServerLogger, config: LanguageServerConfig, solidlsp_settings: SolidLSPSettings
) -> str:
"""
Setup runtime dependencies for rust_analyzer and return the command to start the server.
"""
deps = RuntimeDependencyCollection(
[
RuntimeDependency(
id="RustAnalyzer",
description="RustAnalyzer for macOS (arm64)",
url="https://github.com/rust-lang/rust-analyzer/releases/download/2023-10-09/rust-analyzer-aarch64-apple-darwin.gz",
platform_id="osx-arm64",
archive_type="gz",
binary_name="rust_analyzer",
),
RuntimeDependency(
id="RustAnalyzer",
description="RustAnalyzer for Linux (x64)",
url="https://github.com/rust-lang/rust-analyzer/releases/download/2023-10-09/rust-analyzer-x86_64-unknown-linux-gnu.gz",
platform_id="linux-x64",
archive_type="gz",
binary_name="rust_analyzer",
),
RuntimeDependency(
id="RustAnalyzer",
description="RustAnalyzer for Linux (arm64)",
url="https://github.com/rust-lang/rust-analyzer/releases/download/2023-10-09/rust-analyzer-aarch64-unknown-linux-gnu.gz",
platform_id="linux-arm64",
archive_type="gz",
binary_name="rust_analyzer",
),
RuntimeDependency(
id="RustAnalyzer",
description="RustAnalyzer for Windows (x64)",
url="https://github.com/rust-lang/rust-analyzer/releases/download/2023-10-09/rust-analyzer-x86_64-pc-windows-msvc.zip",
platform_id="win-x64",
archive_type="zip",
binary_name="rust-analyzer.exe",
),
]
)
# assert platform_id.value in [
# "linux-x64",
# "win-x64",
# ], "Only linux-x64 and win-x64 platform is supported for in multilspy at the moment"
rustanalyzer_ls_dir = os.path.join(cls.ls_resources_dir(solidlsp_settings), "RustAnalyzer")
rustanalyzer_executable_path = deps.binary_path(rustanalyzer_ls_dir)
if not os.path.exists(rustanalyzer_ls_dir):
os.makedirs(rustanalyzer_ls_dir)
deps.install(logger, rustanalyzer_ls_dir)
assert os.path.exists(rustanalyzer_executable_path)
os.chmod(rustanalyzer_executable_path, stat.S_IEXEC)
return rustanalyzer_executable_path
@staticmethod
def _get_initialize_params(repository_absolute_path: str) -> InitializeParams:
"""
@@ -414,7 +407,7 @@ class RustAnalyzer(SolidLanguageServer):
"overrideCommand": None,
"useRustcWrapper": True,
},
"cfgs": {},
"cfgs": [],
"extraArgs": [],
"extraEnv": {},
"features": [],
src/solidlsp/language_servers/scala_language_server.py
+174
View File
@@ -0,0 +1,174 @@
"""
Provides Scala specific instantiation of the LanguageServer class. Contains various configurations and settings specific to Scala.
"""
import logging
import os
import pathlib
import shutil
import subprocess
from overrides import override
from solidlsp.ls import SolidLanguageServer
from solidlsp.ls_config import LanguageServerConfig
from solidlsp.ls_logger import LanguageServerLogger
from solidlsp.ls_utils import PlatformUtils
from solidlsp.lsp_protocol_handler.lsp_types import InitializeParams
from solidlsp.lsp_protocol_handler.server import ProcessLaunchInfo
from solidlsp.settings import SolidLSPSettings
if not PlatformUtils.get_platform_id().value.startswith("win"):
pass
class ScalaLanguageServer(SolidLanguageServer):
"""
Provides Scala specific instantiation of the LanguageServer class. Contains various configurations and settings specific to Scala.
"""
def __init__(
self, config: LanguageServerConfig, logger: LanguageServerLogger, repository_root_path: str, solidlsp_settings: SolidLSPSettings
):
"""
Creates a ScalaLanguageServer instance. This class is not meant to be instantiated directly. Use LanguageServer.create() instead.
"""
scala_lsp_executable_path = self._setup_runtime_dependencies(logger, config, solidlsp_settings)
super().__init__(
config,
logger,
repository_root_path,
ProcessLaunchInfo(cmd=scala_lsp_executable_path, cwd=repository_root_path),
config.code_language.value,
solidlsp_settings,
)
@override
def is_ignored_dirname(self, dirname: str) -> bool:
return super().is_ignored_dirname(dirname) or dirname in [
".bloop",
".metals",
"target",
]
@classmethod
def _setup_runtime_dependencies(
cls, logger: LanguageServerLogger, config: LanguageServerConfig, solidlsp_settings: SolidLSPSettings
) -> list[str]:
"""
Setup runtime dependencies for Scala Language Server and return the command to start the server.
"""
assert shutil.which("java") is not None, "JDK is not installed or not in PATH."
metals_home = os.path.join(cls.ls_resources_dir(solidlsp_settings), "metals-lsp")
os.makedirs(metals_home, exist_ok=True)
metals_executable = os.path.join(metals_home, "metals")
coursier_command_path = shutil.which("coursier")
cs_command_path = shutil.which("cs")
if not os.path.exists(metals_executable):
if not cs_command_path:
logger.log("'cs' command not found. Trying to install it using 'coursier'.", logging.INFO)
try:
logger.log("Running 'coursier setup --yes' to install 'cs'...", logging.INFO)
subprocess.run([coursier_command_path, "setup", "--yes"], check=True, capture_output=True, text=True)
except subprocess.CalledProcessError as e:
raise RuntimeError(f"Failed to set up 'cs' command with 'coursier setup'. Stderr: {e.stderr}")
cs_command_path = shutil.which("cs")
if not cs_command_path:
raise RuntimeError(
"'cs' command not found after running 'coursier setup'. Please check your PATH or install it manually."
)
logger.log("'cs' command installed successfully.", logging.INFO)
logger.log(f"metals executable not found at {metals_executable}, bootstrapping...", logging.INFO)
artifact = "org.scalameta:metals_2.13:1.6.2"
cmd = [
cs_command_path,
"bootstrap",
"--java-opt",
"-XX:+UseG1GC",
"--java-opt",
"-XX:+UseStringDeduplication",
"--java-opt",
"-Xss4m",
"--java-opt",
"-Xms100m",
"--java-opt",
"-Dmetals.client=Serena",
artifact,
"-o",
metals_executable,
"-f",
]
logger.log("Bootstrapping metals...", logging.INFO)
subprocess.run(cmd, cwd=metals_home, check=True)
logger.log("Bootstrapping metals finished.", logging.INFO)
return [metals_executable]
@staticmethod
def _get_initialize_params(repository_absolute_path: str) -> InitializeParams:
"""
Returns the initialize params for the Scala Language Server.
"""
root_uri = pathlib.Path(repository_absolute_path).as_uri()
initialize_params = {
"locale": "en",
"processId": os.getpid(),
"rootPath": repository_absolute_path,
"rootUri": root_uri,
"initializationOptions": {
"compilerOptions": {
"completionCommand": None,
"isCompletionItemDetailEnabled": True,
"isCompletionItemDocumentationEnabled": True,
"isCompletionItemResolve": True,
"isHoverDocumentationEnabled": True,
"isSignatureHelpDocumentationEnabled": True,
"overrideDefFormat": "ascli",
"snippetAutoIndent": False,
},
"debuggingProvider": True,
"decorationProvider": False,
"didFocusProvider": False,
"doctorProvider": False,
"executeClientCommandProvider": False,
"globSyntax": "uri",
"icons": "unicode",
"inputBoxProvider": False,
"isVirtualDocumentSupported": False,
"isExitOnShutdown": True,
"isHttpEnabled": True,
"openFilesOnRenameProvider": False,
"quickPickProvider": False,
"renameFileThreshold": 200,
"statusBarProvider": "false",
"treeViewProvider": False,
"testExplorerProvider": False,
"openNewWindowProvider": False,
"copyWorksheetOutputProvider": False,
"doctorVisibilityProvider": False,
},
}
return initialize_params
def _start_server(self):
"""
Starts the Scala Language Server
"""
self.logger.log("Starting Scala server process", logging.INFO)
self.server.start()
self.logger.log(
"Sending initialize request from LSP client to LSP server and awaiting response",
logging.INFO,
)
initialize_params = self._get_initialize_params(self.repository_root_path)
self.server.send.initialize(initialize_params)
self.server.notify.initialized({})
@override
def _get_wait_time_for_cross_file_referencing(self) -> float:
return 5
src/solidlsp/language_servers/solargraph.py
+238 -50
View File
@@ -7,7 +7,8 @@ import json
import logging
import os
import pathlib
import stat
import re
import shutil
import subprocess
import threading
from typing import cast
@@ -44,65 +45,230 @@ class Solargraph(SolidLanguageServer):
"ruby",
solidlsp_settings,
)
self.server_ready = threading.Event()
# Override internal language enum for file matching (excludes .erb files)
# while keeping LSP languageId as "ruby" for protocol compliance
from solidlsp.ls_config import Language
self.language = Language.RUBY_SOLARGRAPH
self.analysis_complete = threading.Event()
self.service_ready_event = threading.Event()
self.initialize_searcher_command_available = threading.Event()
self.resolve_main_method_available = threading.Event()
# Set timeout for Solargraph requests - Bundler environments may need more time
self.set_request_timeout(120.0) # 120 seconds for initialization and requests
@override
def is_ignored_dirname(self, dirname: str) -> bool:
return super().is_ignored_dirname(dirname) or dirname in ["vendor"]
ruby_ignored_dirs = [
"vendor", # Ruby vendor directory
".bundle", # Bundler cache
"tmp", # Temporary files
"log", # Log files
"coverage", # Test coverage reports
".yardoc", # YARD documentation cache
"doc", # Generated documentation
"node_modules", # Node modules (for Rails with JS)
"storage", # Active Storage files (Rails)
]
return super().is_ignored_dirname(dirname) or dirname in ruby_ignored_dirs
@staticmethod
def _setup_runtime_dependencies(logger: LanguageServerLogger, config: LanguageServerConfig, repository_root_path: str) -> str:
"""
Setup runtime dependencies for Solargraph and return the command to start the server.
"""
runtime_dependencies = [
{
"url": "https://rubygems.org/downloads/solargraph-0.51.1.gem",
"installCommand": "gem install solargraph -v 0.51.1",
"binaryName": "solargraph",
"archiveType": "gem",
}
]
dependency = runtime_dependencies[0]
# Check if Ruby is installed
try:
result = subprocess.run(["ruby", "--version"], check=True, capture_output=True, cwd=repository_root_path)
ruby_version = str(result.stdout).strip()
result = subprocess.run(["ruby", "--version"], check=True, capture_output=True, cwd=repository_root_path, text=True)
ruby_version = result.stdout.strip()
logger.log(f"Ruby version: {ruby_version}", logging.INFO)
# Extract version number for compatibility checks
version_match = re.search(r"ruby (\d+)\.(\d+)\.(\d+)", ruby_version)
if version_match:
major, minor, patch = map(int, version_match.groups())
if major < 2 or (major == 2 and minor < 6):
logger.log(f"Warning: Ruby {major}.{minor}.{patch} detected. Solargraph works best with Ruby 2.6+", logging.WARNING)
except subprocess.CalledProcessError as e:
raise RuntimeError(f"Error checking for Ruby installation: {e.stderr}") from e
error_msg = e.stderr.decode() if e.stderr else "Unknown error"
raise RuntimeError(
f"Error checking Ruby installation: {error_msg}. Please ensure Ruby is properly installed and in PATH."
) from e
except FileNotFoundError as e:
raise RuntimeError("Ruby is not installed. Please install Ruby before continuing.") from e
raise RuntimeError(
"Ruby is not installed or not found in PATH. Please install Ruby using one of these methods:\n"
" - Using rbenv: rbenv install 3.0.0 && rbenv global 3.0.0\n"
" - Using RVM: rvm install 3.0.0 && rvm use 3.0.0 --default\n"
" - Using asdf: asdf install ruby 3.0.0 && asdf global ruby 3.0.0\n"
" - System package manager (brew install ruby, apt install ruby, etc.)"
) from e
# Check if solargraph is installed
try:
result = subprocess.run(
["gem", "list", "^solargraph$", "-i"], check=False, capture_output=True, text=True, cwd=repository_root_path # type: ignore
# Helper function for Windows-compatible executable search
def find_executable_with_extensions(executable_name: str) -> str | None:
"""Find executable with Windows-specific extensions if on Windows."""
import platform
if platform.system() == "Windows":
for ext in [".bat", ".cmd", ".exe"]:
path = shutil.which(f"{executable_name}{ext}")
if path:
return path
return shutil.which(executable_name)
else:
return shutil.which(executable_name)
# Check for Bundler project (Gemfile exists)
gemfile_path = os.path.join(repository_root_path, "Gemfile")
gemfile_lock_path = os.path.join(repository_root_path, "Gemfile.lock")
is_bundler_project = os.path.exists(gemfile_path)
if is_bundler_project:
logger.log("Detected Bundler project (Gemfile found)", logging.INFO)
# Check if bundle command is available
bundle_path = find_executable_with_extensions("bundle")
if not bundle_path:
# Try common bundle executables
for bundle_cmd in ["bin/bundle", "bundle"]:
if bundle_cmd.startswith("bin/"):
bundle_full_path = os.path.join(repository_root_path, bundle_cmd)
else:
bundle_full_path = find_executable_with_extensions(bundle_cmd)
if bundle_full_path and os.path.exists(bundle_full_path):
bundle_path = bundle_full_path if bundle_cmd.startswith("bin/") else bundle_cmd
break
if not bundle_path:
raise RuntimeError(
"Bundler project detected but 'bundle' command not found. Please install Bundler:\n"
" - gem install bundler\n"
" - Or use your Ruby version manager's bundler installation\n"
" - Ensure the bundle command is in your PATH"
)
# Check if solargraph is in Gemfile.lock
solargraph_in_bundle = False
if os.path.exists(gemfile_lock_path):
try:
with open(gemfile_lock_path) as f:
content = f.read()
solargraph_in_bundle = "solargraph" in content.lower()
except Exception as e:
logger.log(f"Warning: Could not read Gemfile.lock: {e}", logging.WARNING)
if solargraph_in_bundle:
logger.log("Found solargraph in Gemfile.lock", logging.INFO)
return f"{bundle_path} exec solargraph"
else:
logger.log(
"solargraph not found in Gemfile.lock. Please add 'gem \"solargraph\"' to your Gemfile and run 'bundle install'",
logging.WARNING,
)
# Fall through to global installation check
# Check if solargraph is installed globally
# First, try to find solargraph in PATH (includes asdf shims) with Windows support
solargraph_path = find_executable_with_extensions("solargraph")
if solargraph_path:
logger.log(f"Found solargraph at: {solargraph_path}", logging.INFO)
return solargraph_path
# Fallback to gem exec (for non-Bundler projects or when global solargraph not found)
if not is_bundler_project:
runtime_dependencies = [
{
"url": "https://rubygems.org/downloads/solargraph-0.51.1.gem",
"installCommand": "gem install solargraph -v 0.51.1",
"binaryName": "solargraph",
"archiveType": "gem",
}
]
dependency = runtime_dependencies[0]
try:
result = subprocess.run(
["gem", "list", "^solargraph$", "-i"], check=False, capture_output=True, text=True, cwd=repository_root_path
)
if result.stdout.strip() == "false":
logger.log("Installing Solargraph...", logging.INFO)
subprocess.run(dependency["installCommand"].split(), check=True, capture_output=True, cwd=repository_root_path)
return "gem exec solargraph"
except subprocess.CalledProcessError as e:
error_msg = e.stderr.decode() if e.stderr else str(e)
raise RuntimeError(
f"Failed to check or install Solargraph: {error_msg}\nPlease try installing manually: gem install solargraph"
) from e
else:
raise RuntimeError(
"This appears to be a Bundler project, but solargraph is not available. "
"Please add 'gem \"solargraph\"' to your Gemfile and run 'bundle install'."
)
if str(result.stdout).strip() == "false":
logger.log("Installing Solargraph...", logging.INFO)
subprocess.run(dependency["installCommand"].split(), check=True, capture_output=True, cwd=repository_root_path)
# Get the gem executable path directly
result = subprocess.run(["gem", "which", "solargraph"], check=True, capture_output=True, text=True, cwd=repository_root_path) # type: ignore
gem_path = result.stdout.strip()
bin_dir = os.path.join(os.path.dirname(os.path.dirname(gem_path)), "bin") # type: ignore
executable_path = os.path.join(bin_dir, "solargraph")
@staticmethod
def _detect_rails_project(repository_root_path: str) -> bool:
"""
Detect if this is a Rails project by checking for Rails-specific files.
"""
rails_indicators = [
"config/application.rb",
"config/environment.rb",
"app/controllers/application_controller.rb",
"Rakefile",
]
if not os.path.exists(executable_path):
raise RuntimeError(f"Solargraph executable not found at {executable_path}")
for indicator in rails_indicators:
if os.path.exists(os.path.join(repository_root_path, indicator)):
return True
# Ensure the executable has the right permissions
os.chmod(executable_path, os.stat(executable_path).st_mode | stat.S_IEXEC)
# Check for Rails in Gemfile
gemfile_path = os.path.join(repository_root_path, "Gemfile")
if os.path.exists(gemfile_path):
try:
with open(gemfile_path) as f:
content = f.read().lower()
if "gem 'rails'" in content or 'gem "rails"' in content:
return True
except Exception:
pass
return executable_path
except subprocess.CalledProcessError as e:
raise RuntimeError(f"Failed to check or install Solargraph. {e.stderr}") from e
return False
@staticmethod
def _get_ruby_exclude_patterns(repository_root_path: str) -> list[str]:
"""
Get Ruby and Rails-specific exclude patterns for better performance.
"""
base_patterns = [
"**/vendor/**", # Ruby vendor directory (similar to node_modules)
"**/.bundle/**", # Bundler cache
"**/tmp/**", # Temporary files
"**/log/**", # Log files
"**/coverage/**", # Test coverage reports
"**/.yardoc/**", # YARD documentation cache
"**/doc/**", # Generated documentation
"**/.git/**", # Git directory
"**/node_modules/**", # Node modules (for Rails with JS)
"**/public/assets/**", # Rails compiled assets
]
# Add Rails-specific patterns if this is a Rails project
if Solargraph._detect_rails_project(repository_root_path):
rails_patterns = [
"**/public/packs/**", # Webpacker output
"**/public/webpack/**", # Webpack output
"**/storage/**", # Active Storage files
"**/tmp/cache/**", # Rails cache
"**/tmp/pids/**", # Process IDs
"**/tmp/sessions/**", # Session files
"**/tmp/sockets/**", # Socket files
"**/db/*.sqlite3", # SQLite databases
]
base_patterns.extend(rails_patterns)
return base_patterns
@staticmethod
def _get_initialize_params(repository_absolute_path: str) -> InitializeParams:
@@ -110,12 +276,27 @@ class Solargraph(SolidLanguageServer):
Returns the initialize params for the Solargraph Language Server.
"""
root_uri = pathlib.Path(repository_absolute_path).as_uri()
initialize_params = {
"capabilities": {},
"trace": "verbose",
exclude_patterns = Solargraph._get_ruby_exclude_patterns(repository_absolute_path)
initialize_params: InitializeParams = { # type: ignore
"processId": os.getpid(),
"rootPath": repository_absolute_path,
"rootUri": pathlib.Path(repository_absolute_path).as_uri(),
"rootUri": root_uri,
"initializationOptions": {
"exclude": exclude_patterns,
},
"capabilities": {
"workspace": {
"workspaceEdit": {"documentChanges": True},
},
"textDocument": {
"documentSymbol": {
"hierarchicalDocumentSymbolSupport": True,
"symbolKind": {"valueSet": list(range(1, 27))},
},
},
},
"trace": "verbose",
"workspaceFolders": [
{
"uri": root_uri,
@@ -138,12 +319,12 @@ class Solargraph(SolidLanguageServer):
self.resolve_main_method_available.set()
return
def lang_status_handler(params: dict) -> None:
# TODO: Should we wait for
# server -> client: {'jsonrpc': '2.0', 'method': 'language/status', 'params': {'type': 'ProjectStatus', 'message': 'OK'}}
# Before proceeding?
if params["type"] == "ServiceReady" and params["message"] == "ServiceReady":
self.service_ready_event.set()
def lang_status_handler(params):
self.logger.log(f"LSP: language/status: {params}", logging.INFO)
if params.get("type") == "ServiceReady" and params.get("message") == "Service is ready.":
self.logger.log("Solargraph service is ready.", logging.INFO)
self.analysis_complete.set()
self.completions_available.set()
def execute_client_command_handler(params: dict) -> list:
return []
@@ -180,7 +361,14 @@ class Solargraph(SolidLanguageServer):
"triggerCharacters": [".", ":", "@"],
}
self.server.notify.initialized({})
self.completions_available.set()
self.server_ready.set()
self.server_ready.wait()
# Wait for Solargraph to complete its initial workspace analysis
# This prevents issues by ensuring background tasks finish
self.logger.log("Waiting for Solargraph to complete initial workspace analysis...", logging.INFO)
if self.analysis_complete.wait(timeout=60.0):
self.logger.log("Solargraph initial analysis complete, server ready", logging.INFO)
else:
self.logger.log("Timeout waiting for Solargraph analysis completion, proceeding anyway", logging.WARNING)
# Fallback: assume analysis is complete after timeout
self.analysis_complete.set()
self.completions_available.set()
src/solidlsp/language_servers/sourcekit_lsp.py
+385
View File
@@ -0,0 +1,385 @@
import logging
import os
import pathlib
import subprocess
import threading
import time
from overrides import override
from solidlsp import ls_types
from solidlsp.ls import SolidLanguageServer
from solidlsp.ls_config import LanguageServerConfig
from solidlsp.ls_logger import LanguageServerLogger
from solidlsp.lsp_protocol_handler.lsp_types import InitializeParams
from solidlsp.lsp_protocol_handler.server import ProcessLaunchInfo
from solidlsp.settings import SolidLSPSettings
class SourceKitLSP(SolidLanguageServer):
"""
Provides Swift specific instantiation of the LanguageServer class using sourcekit-lsp.
"""
@override
def is_ignored_dirname(self, dirname: str) -> bool:
# For Swift projects, we should ignore:
# - .build: Swift Package Manager build artifacts
# - .swiftpm: Swift Package Manager metadata
# - node_modules: if the project has JavaScript components
# - dist/build: common output directories
return super().is_ignored_dirname(dirname) or dirname in [".build", ".swiftpm", "node_modules", "dist", "build"]
@staticmethod
def _get_sourcekit_lsp_version() -> str:
"""Get the installed sourcekit-lsp version or raise error if sourcekit was not found."""
try:
result = subprocess.run(["sourcekit-lsp", "-h"], capture_output=True, text=True, check=False)
if result.returncode == 0:
return result.stdout.strip()
else:
raise Exception(f"`sourcekit-lsp -h` resulted in: {result}")
except Exception as e:
raise RuntimeError(
"Could not find sourcekit-lsp, please install it as described in https://github.com/apple/sourcekit-lsp#installation"
"And make sure it is available on your PATH."
) from e
def __init__(
self, config: LanguageServerConfig, logger: LanguageServerLogger, repository_root_path: str, solidlsp_settings: SolidLSPSettings
):
sourcekit_version = self._get_sourcekit_lsp_version()
logger.log(f"Starting sourcekit lsp with version: {sourcekit_version}", logging.INFO)
super().__init__(
config,
logger,
repository_root_path,
ProcessLaunchInfo(cmd="sourcekit-lsp", cwd=repository_root_path),
"swift",
solidlsp_settings,
)
self.server_ready = threading.Event()
self.request_id = 0
self._did_sleep_before_requesting_references = False
self._initialization_timestamp = None
@staticmethod
def _get_initialize_params(repository_absolute_path: str) -> InitializeParams:
"""
Returns the initialize params for the Swift Language Server.
"""
root_uri = pathlib.Path(repository_absolute_path).as_uri()
initialize_params = {
"capabilities": {
"general": {
"markdown": {"parser": "marked", "version": "1.1.0"},
"positionEncodings": ["utf-16"],
"regularExpressions": {"engine": "ECMAScript", "version": "ES2020"},
"staleRequestSupport": {
"cancel": True,
"retryOnContentModified": [
"textDocument/semanticTokens/full",
"textDocument/semanticTokens/range",
"textDocument/semanticTokens/full/delta",
],
},
},
"notebookDocument": {"synchronization": {"dynamicRegistration": True, "executionSummarySupport": True}},
"textDocument": {
"callHierarchy": {"dynamicRegistration": True},
"codeAction": {
"codeActionLiteralSupport": {
"codeActionKind": {
"valueSet": [
"",
"quickfix",
"refactor",
"refactor.extract",
"refactor.inline",
"refactor.rewrite",
"source",
"source.organizeImports",
]
}
},
"dataSupport": True,
"disabledSupport": True,
"dynamicRegistration": True,
"honorsChangeAnnotations": True,
"isPreferredSupport": True,
"resolveSupport": {"properties": ["edit"]},
},
"codeLens": {"dynamicRegistration": True},
"colorProvider": {"dynamicRegistration": True},
"completion": {
"completionItem": {
"commitCharactersSupport": True,
"deprecatedSupport": True,
"documentationFormat": ["markdown", "plaintext"],
"insertReplaceSupport": True,
"insertTextModeSupport": {"valueSet": [1, 2]},
"labelDetailsSupport": True,
"preselectSupport": True,
"resolveSupport": {"properties": ["documentation", "detail", "additionalTextEdits"]},
"snippetSupport": True,
"tagSupport": {"valueSet": [1]},
},
"completionItemKind": {
"valueSet": [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25]
},
"completionList": {"itemDefaults": ["commitCharacters", "editRange", "insertTextFormat", "insertTextMode", "data"]},
"contextSupport": True,
"dynamicRegistration": True,
"insertTextMode": 2,
},
"declaration": {"dynamicRegistration": True, "linkSupport": True},
"definition": {"dynamicRegistration": True, "linkSupport": True},
"diagnostic": {"dynamicRegistration": True, "relatedDocumentSupport": False},
"documentHighlight": {"dynamicRegistration": True},
"documentLink": {"dynamicRegistration": True, "tooltipSupport": True},
"documentSymbol": {
"dynamicRegistration": True,
"hierarchicalDocumentSymbolSupport": True,
"labelSupport": True,
"symbolKind": {
"valueSet": [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26]
},
"tagSupport": {"valueSet": [1]},
},
"foldingRange": {
"dynamicRegistration": True,
"foldingRange": {"collapsedText": False},
"foldingRangeKind": {"valueSet": ["comment", "imports", "region"]},
"lineFoldingOnly": True,
"rangeLimit": 5000,
},
"formatting": {"dynamicRegistration": True},
"hover": {"contentFormat": ["markdown", "plaintext"], "dynamicRegistration": True},
"implementation": {"dynamicRegistration": True, "linkSupport": True},
"inlayHint": {
"dynamicRegistration": True,
"resolveSupport": {"properties": ["tooltip", "textEdits", "label.tooltip", "label.location", "label.command"]},
},
"inlineValue": {"dynamicRegistration": True},
"linkedEditingRange": {"dynamicRegistration": True},
"onTypeFormatting": {"dynamicRegistration": True},
"publishDiagnostics": {
"codeDescriptionSupport": True,
"dataSupport": True,
"relatedInformation": True,
"tagSupport": {"valueSet": [1, 2]},
"versionSupport": False,
},
"rangeFormatting": {"dynamicRegistration": True, "rangesSupport": True},
"references": {"dynamicRegistration": True},
"rename": {
"dynamicRegistration": True,
"honorsChangeAnnotations": True,
"prepareSupport": True,
"prepareSupportDefaultBehavior": 1,
},
"selectionRange": {"dynamicRegistration": True},
"semanticTokens": {
"augmentsSyntaxTokens": True,
"dynamicRegistration": True,
"formats": ["relative"],
"multilineTokenSupport": False,
"overlappingTokenSupport": False,
"requests": {"full": {"delta": True}, "range": True},
"serverCancelSupport": True,
"tokenModifiers": [
"declaration",
"definition",
"readonly",
"static",
"deprecated",
"abstract",
"async",
"modification",
"documentation",
"defaultLibrary",
],
"tokenTypes": [
"namespace",
"type",
"class",
"enum",
"interface",
"struct",
"typeParameter",
"parameter",
"variable",
"property",
"enumMember",
"event",
"function",
"method",
"macro",
"keyword",
"modifier",
"comment",
"string",
"number",
"regexp",
"operator",
"decorator",
],
},
"signatureHelp": {
"contextSupport": True,
"dynamicRegistration": True,
"signatureInformation": {
"activeParameterSupport": True,
"documentationFormat": ["markdown", "plaintext"],
"parameterInformation": {"labelOffsetSupport": True},
},
},
"synchronization": {"didSave": True, "dynamicRegistration": True, "willSave": True, "willSaveWaitUntil": True},
"typeDefinition": {"dynamicRegistration": True, "linkSupport": True},
"typeHierarchy": {"dynamicRegistration": True},
},
"window": {
"showDocument": {"support": True},
"showMessage": {"messageActionItem": {"additionalPropertiesSupport": True}},
"workDoneProgress": True,
},
"workspace": {
"applyEdit": True,
"codeLens": {"refreshSupport": True},
"configuration": True,
"diagnostics": {"refreshSupport": True},
"didChangeConfiguration": {"dynamicRegistration": True},
"didChangeWatchedFiles": {"dynamicRegistration": True, "relativePatternSupport": True},
"executeCommand": {"dynamicRegistration": True},
"fileOperations": {
"didCreate": True,
"didDelete": True,
"didRename": True,
"dynamicRegistration": True,
"willCreate": True,
"willDelete": True,
"willRename": True,
},
"foldingRange": {"refreshSupport": True},
"inlayHint": {"refreshSupport": True},
"inlineValue": {"refreshSupport": True},
"semanticTokens": {"refreshSupport": False},
"symbol": {
"dynamicRegistration": True,
"resolveSupport": {"properties": ["location.range"]},
"symbolKind": {
"valueSet": [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26]
},
"tagSupport": {"valueSet": [1]},
},
"workspaceEdit": {
"changeAnnotationSupport": {"groupsOnLabel": True},
"documentChanges": True,
"failureHandling": "textOnlyTransactional",
"normalizesLineEndings": True,
"resourceOperations": ["create", "rename", "delete"],
},
"workspaceFolders": True,
},
},
"clientInfo": {"name": "Visual Studio Code", "version": "1.102.2"},
"initializationOptions": {
"backgroundIndexing": True,
"backgroundPreparationMode": "enabled",
"textDocument/codeLens": {"supportedCommands": {"swift.debug": "swift.debug", "swift.run": "swift.run"}},
"window/didChangeActiveDocument": True,
"workspace/getReferenceDocument": True,
"workspace/peekDocuments": True,
},
"locale": "en",
"processId": os.getpid(),
"rootPath": repository_absolute_path,
"rootUri": root_uri,
"workspaceFolders": [
{
"uri": root_uri,
"name": os.path.basename(repository_absolute_path),
}
],
}
return initialize_params
def _start_server(self):
"""Start sourcekit-lsp server process"""
def register_capability_handler(_params):
return
def window_log_message(msg):
self.logger.log(f"LSP: window/logMessage: {msg}", logging.INFO)
def do_nothing(_params):
return
self.server.on_request("client/registerCapability", register_capability_handler)
self.server.on_notification("window/logMessage", window_log_message)
self.server.on_notification("$/progress", do_nothing)
self.server.on_notification("textDocument/publishDiagnostics", do_nothing)
self.logger.log("Starting sourcekit-lsp server process", logging.INFO)
self.server.start()
initialize_params = self._get_initialize_params(self.repository_root_path)
self.logger.log(
"Sending initialize request from LSP client to LSP server and awaiting response",
logging.INFO,
)
init_response = self.server.send.initialize(initialize_params)
capabilities = init_response["capabilities"]
self.logger.log(f"SourceKit LSP capabilities: {list(capabilities.keys())}", logging.INFO)
assert "textDocumentSync" in capabilities, "textDocumentSync capability missing"
assert "definitionProvider" in capabilities, "definitionProvider capability missing"
self.server.notify.initialized({})
self.completions_available.set()
self.server_ready.set()
self.server_ready.wait()
# Mark initialization timestamp for smarter delay calculation
self._initialization_timestamp = time.time()
@override
def request_references(self, relative_file_path: str, line: int, column: int) -> list[ls_types.Location]:
# SourceKit LSP needs initialization + indexing time after startup
# before it can provide accurate reference information. This sleep
# prevents race conditions where references might not be available yet.
# CI environments need extra time for project indexing and cross-file analysis
if not self._did_sleep_before_requesting_references:
# Calculate minimum delay based on how much time has passed since initialization
if self._initialization_timestamp:
elapsed = time.time() - self._initialization_timestamp
# Increased CI delay for project indexing: 15s CI, 5s local
base_delay = 15 if os.getenv("CI") else 5
remaining_delay = max(2, base_delay - elapsed)
else:
# Fallback if initialization timestamp is missing
remaining_delay = 15 if os.getenv("CI") else 5
self.logger.log(
f"Sleeping {remaining_delay:.1f}s before requesting references for the first time (CI needs extra indexing time)",
logging.INFO,
)
time.sleep(remaining_delay)
self._did_sleep_before_requesting_references = True
# Get references with retry logic for CI stability
references = super().request_references(relative_file_path, line, column)
# In CI, if no references found, retry once after additional delay
if os.getenv("CI") and not references:
self.logger.log("No references found in CI - retrying after additional 5s delay", logging.INFO)
time.sleep(5)
references = super().request_references(relative_file_path, line, column)
return references
src/solidlsp/language_servers/terraform_ls.py
+10 -3
View File
@@ -1,7 +1,6 @@
import logging
import os
import shutil
import stat
import threading
from typing import cast
@@ -84,6 +83,14 @@ class TerraformLS(SolidLanguageServer):
archive_type="zip",
binary_name="terraform-ls",
),
RuntimeDependency(
id="TerraformLS",
description="terraform-ls for Linux (ARM64)",
url="https://releases.hashicorp.com/terraform-ls/0.36.5/terraform-ls_0.36.5_linux_arm64.zip",
platform_id="linux-arm64",
archive_type="zip",
binary_name="terraform-ls",
),
RuntimeDependency(
id="TerraformLS",
description="terraform-ls for Linux (x64)",
@@ -102,7 +109,7 @@ class TerraformLS(SolidLanguageServer):
),
]
)
dependency = deps.single_for_current_platform()
dependency = deps.get_single_dep_for_current_platform()
terraform_ls_executable_path = deps.binary_path(cls.ls_resources_dir(solidlsp_settings))
if not os.path.exists(terraform_ls_executable_path):
@@ -113,7 +120,7 @@ class TerraformLS(SolidLanguageServer):
# Make the executable file executable on Unix-like systems
if platform_id.value != "win-x64":
os.chmod(terraform_ls_executable_path, stat.S_IEXEC | stat.S_IREAD)
os.chmod(terraform_ls_executable_path, 0o755)
return terraform_ls_executable_path
src/solidlsp/language_servers/typescript_language_server.py
+6 -13
View File
@@ -75,7 +75,7 @@ class TypeScriptLanguageServer(SolidLanguageServer):
@classmethod
def _setup_runtime_dependencies(
cls, logger: LanguageServerLogger, config: LanguageServerConfig, solidlsp_settings: SolidLSPSettings
) -> str:
) -> list[str]:
"""
Setup runtime dependencies for TypeScript Language Server and return the command to start the server.
"""
@@ -97,13 +97,13 @@ class TypeScriptLanguageServer(SolidLanguageServer):
RuntimeDependency(
id="typescript",
description="typescript package",
command="npm install --prefix ./ typescript@5.5.4",
command=["npm", "install", "--prefix", "./", "typescript@5.5.4"],
platform_id="any",
),
RuntimeDependency(
id="typescript-language-server",
description="typescript-language-server package",
command="npm install --prefix ./ typescript-language-server@4.3.3",
command=["npm", "install", "--prefix", "./", "typescript-language-server@4.3.3"],
platform_id="any",
),
]
@@ -133,7 +133,7 @@ class TypeScriptLanguageServer(SolidLanguageServer):
raise FileNotFoundError(
f"typescript-language-server executable not found at {tsserver_executable_path}, something went wrong with the installation."
)
return f"{tsserver_executable_path} --stdio"
return [tsserver_executable_path, "--stdio"]
@staticmethod
def _get_initialize_params(repository_absolute_path: str) -> InitializeParams:
@@ -252,12 +252,5 @@ class TypeScriptLanguageServer(SolidLanguageServer):
self.completions_available.set()
@override
# For some reason, the LS may need longer to process this, so we just retry
def _send_references_request(self, relative_file_path: str, line: int, column: int) -> list[lsp_types.Location] | None:
# TODO: The LS doesn't return references contained in other files if it doesn't sleep. This is
# despite the LS having processed requests already. I don't know what causes this, but sleeping
# one second helps. It may be that sleeping only once is enough but that's hard to reliably test.
# It may be that even this 1sec is not enough in larger TS projects, at some point we should find what
# causes this and solve it.
sleep(1)
return super()._send_references_request(relative_file_path, line, column)
def _get_wait_time_for_cross_file_referencing(self) -> float:
return 1
src/solidlsp/language_servers/vts_language_server.py
+2 -4
View File
@@ -233,8 +233,6 @@ class VtsLanguageServer(SolidLanguageServer):
self.server_ready.set()
self.completions_available.set()
# VTS may need longer to process references, so we include a sleep similar to TypeScript LS
@override
def _send_references_request(self, relative_file_path: str, line: int, column: int) -> list[lsp_types.Location] | None:
sleep(1)
return super()._send_references_request(relative_file_path, line, column)
def _get_wait_time_for_cross_file_referencing(self) -> float:
return 1
src/solidlsp/language_servers/zls.py
+252
View File
@@ -0,0 +1,252 @@
"""
Provides Zig specific instantiation of the LanguageServer class using ZLS (Zig Language Server).
"""
import logging
import os
import pathlib
import platform
import shutil
import subprocess
import threading
from overrides import override
from solidlsp.ls import SolidLanguageServer
from solidlsp.ls_config import LanguageServerConfig
from solidlsp.ls_logger import LanguageServerLogger
from solidlsp.lsp_protocol_handler.lsp_types import InitializeParams
from solidlsp.lsp_protocol_handler.server import ProcessLaunchInfo
from solidlsp.settings import SolidLSPSettings
class ZigLanguageServer(SolidLanguageServer):
"""
Provides Zig specific instantiation of the LanguageServer class using ZLS.
"""
@override
def is_ignored_dirname(self, dirname: str) -> bool:
# For Zig projects, we should ignore:
# - zig-cache: build cache directory
# - zig-out: default build output directory
# - .zig-cache: alternative cache location
# - node_modules: if the project has JavaScript components
return super().is_ignored_dirname(dirname) or dirname in ["zig-cache", "zig-out", ".zig-cache", "node_modules", "build", "dist"]
@staticmethod
def _get_zig_version():
"""Get the installed Zig version or None if not found."""
try:
result = subprocess.run(["zig", "version"], capture_output=True, text=True, check=False)
if result.returncode == 0:
return result.stdout.strip()
except FileNotFoundError:
return None
return None
@staticmethod
def _get_zls_version():
"""Get the installed ZLS version or None if not found."""
try:
result = subprocess.run(["zls", "--version"], capture_output=True, text=True, check=False)
if result.returncode == 0:
return result.stdout.strip()
except FileNotFoundError:
return None
return None
@staticmethod
def _check_zls_installed():
"""Check if ZLS is installed in the system."""
return shutil.which("zls") is not None
@staticmethod
def _setup_runtime_dependency():
"""
Check if required Zig runtime dependencies are available.
Raises RuntimeError with helpful message if dependencies are missing.
"""
# Check for Windows and provide error message
if platform.system() == "Windows":
raise RuntimeError(
"Windows is not supported by ZLS in this integration. "
"Cross-file references don't work reliably on Windows. Reason unknown."
)
zig_version = ZigLanguageServer._get_zig_version()
if not zig_version:
raise RuntimeError(
"Zig is not installed. Please install Zig from https://ziglang.org/download/ and make sure it is added to your PATH."
)
if not ZigLanguageServer._check_zls_installed():
zls_version = ZigLanguageServer._get_zls_version()
if not zls_version:
raise RuntimeError(
"Found Zig but ZLS (Zig Language Server) is not installed.\n"
"Please install ZLS from https://github.com/zigtools/zls\n"
"You can install it via:\n"
" - Package managers (brew install zls, scoop install zls, etc.)\n"
" - Download pre-built binaries from GitHub releases\n"
" - Build from source with: zig build -Doptimize=ReleaseSafe\n\n"
"After installation, make sure 'zls' is added to your PATH."
)
return True
def __init__(
self, config: LanguageServerConfig, logger: LanguageServerLogger, repository_root_path: str, solidlsp_settings: SolidLSPSettings
):
self._setup_runtime_dependency()
super().__init__(
config,
logger,
repository_root_path,
ProcessLaunchInfo(cmd="zls", cwd=repository_root_path),
"zig",
solidlsp_settings,
)
self.server_ready = threading.Event()
self.request_id = 0
@staticmethod
def _get_initialize_params(repository_absolute_path: str) -> InitializeParams:
"""
Returns the initialize params for the Zig Language Server.
"""
root_uri = pathlib.Path(repository_absolute_path).as_uri()
initialize_params = {
"locale": "en",
"capabilities": {
"textDocument": {
"synchronization": {"didSave": True, "dynamicRegistration": True},
"definition": {"dynamicRegistration": True},
"references": {"dynamicRegistration": True},
"documentSymbol": {
"dynamicRegistration": True,
"hierarchicalDocumentSymbolSupport": True,
"symbolKind": {"valueSet": list(range(1, 27))},
},
"completion": {
"dynamicRegistration": True,
"completionItem": {
"snippetSupport": True,
"commitCharactersSupport": True,
"documentationFormat": ["markdown", "plaintext"],
"deprecatedSupport": True,
"preselectSupport": True,
},
},
"hover": {
"dynamicRegistration": True,
"contentFormat": ["markdown", "plaintext"],
},
},
"workspace": {
"workspaceFolders": True,
"didChangeConfiguration": {"dynamicRegistration": True},
"configuration": True,
},
},
"processId": os.getpid(),
"rootPath": repository_absolute_path,
"rootUri": root_uri,
"workspaceFolders": [
{
"uri": root_uri,
"name": os.path.basename(repository_absolute_path),
}
],
"initializationOptions": {
# ZLS specific options based on schema.json
# Critical paths for ZLS to understand the project
"zig_exe_path": shutil.which("zig"), # Path to zig executable
"zig_lib_path": None, # Let ZLS auto-detect
"build_runner_path": None, # Let ZLS use its built-in runner
"global_cache_path": None, # Let ZLS use default cache
# Build configuration
"enable_build_on_save": True, # Enable to analyze project structure
"build_on_save_args": ["build"],
# Features
"enable_snippets": True,
"enable_argument_placeholders": True,
"semantic_tokens": "full",
"warn_style": False,
"highlight_global_var_declarations": False,
"skip_std_references": False,
"prefer_ast_check_as_child_process": True,
"completion_label_details": True,
# Inlay hints configuration
"inlay_hints_show_variable_type_hints": True,
"inlay_hints_show_struct_literal_field_type": True,
"inlay_hints_show_parameter_name": True,
"inlay_hints_show_builtin": True,
"inlay_hints_exclude_single_argument": True,
"inlay_hints_hide_redundant_param_names": False,
"inlay_hints_hide_redundant_param_names_last_token": False,
},
}
return initialize_params
def _start_server(self):
"""Start ZLS server process"""
def register_capability_handler(params):
return
def window_log_message(msg):
self.logger.log(f"LSP: window/logMessage: {msg}", logging.INFO)
def do_nothing(params):
return
self.server.on_request("client/registerCapability", register_capability_handler)
self.server.on_notification("window/logMessage", window_log_message)
self.server.on_notification("$/progress", do_nothing)
self.server.on_notification("textDocument/publishDiagnostics", do_nothing)
self.logger.log("Starting ZLS server process", logging.INFO)
self.server.start()
initialize_params = self._get_initialize_params(self.repository_root_path)
self.logger.log(
"Sending initialize request from LSP client to LSP server and awaiting response",
logging.INFO,
)
init_response = self.server.send.initialize(initialize_params)
# Verify server capabilities
assert "textDocumentSync" in init_response["capabilities"]
assert "definitionProvider" in init_response["capabilities"]
assert "documentSymbolProvider" in init_response["capabilities"]
assert "referencesProvider" in init_response["capabilities"]
self.server.notify.initialized({})
self.completions_available.set()
# ZLS server is ready after initialization
self.server_ready.set()
self.server_ready.wait()
# Open build.zig if it exists to help ZLS understand project structure
build_zig_path = os.path.join(self.repository_root_path, "build.zig")
if os.path.exists(build_zig_path):
try:
with open(build_zig_path, encoding="utf-8") as f:
content = f.read()
uri = pathlib.Path(build_zig_path).as_uri()
self.server.notify.did_open_text_document(
{
"textDocument": {
"uri": uri,
"languageId": "zig",
"version": 1,
"text": content,
}
}
)
self.logger.log("Opened build.zig to provide project context to ZLS", logging.INFO)
except Exception as e:
self.logger.log(f"Failed to open build.zig: {e}", logging.WARNING)
Loaded 100 of 242 files, more files were not shown because too many files have changed in this diff. Show more