An LSP server can end a symbol range one line past EOF: the convention
for a range covering whole lines ends it at the start of the following,
non-existent line. SymbolBody.get_text indexes self._lines[end_line]
directly, so that well-defined case raised IndexError.
get_text now corrects exactly that case (end_line == len(lines), end_col=0) to end
at the actual last line, in get_text itself so it also applies to a
cached SymbolBody, not just a freshly constructed one. Any other
out-of-range end_line now raises InvalidTextLocationError.
Fixes#1498
`search_text` mapped the match's exclusive end index to a line number via
`get_line_from_index`, which resolves an index at a line start to the following
line. A match ending with a newline was therefore attributed one line too many,
and `context_lines_after` was shifted by the same amount.
FileUtils.read_file has two return paths with different newline semantics.
The primary path, open(file_path, encoding=encoding).read(), uses the default
newline=None and therefore applies universal newlines, so CR never reaches
callers. The charset_normalizer fallback decodes the raw bytes instead
(match.raw.decode(match.encoding)), which performs no newline translation, so
any file that fails to decode with the project's configured encoding re-enters
Serena carrying its CR characters.
Normalize to LF at the fallback's return, matching the primary path.
- read via `ReadFileTool`
- searched via `SearchForPatternTool`
when using plugin version 2023.3.3+ (yet unreleased)
This is enabled via JetBrainsFileProxy (which reads the file through the IDE)
read_file determined line indices via Python's splitlines, which breaks on
\f, \v, \x1c-\x1e, \x85, U+2028 and U+2029 in addition to the LSP-compliant
line breaks counted by delete_lines/replace_lines. In a file containing any
of those characters the indices reported by read_file referred to different
physical lines than the editing tools operate on, so an edit made after the
required read_file could silently hit the wrong line.
Use TextUtils.split_lines, as already applied to search_text and
MatchedConsecutiveLines.from_file_contents. Files whose line breaks are all
LSP-compliant are decomposed exactly as before.
delete_lines/replace_lines address the position one line past the last line
(line == number of lines, col 0). get_index_from_line_col cannot resolve that
position when the file has no trailing newline (no closing newline to count),
so delete_text_between_positions raised InvalidTextLocationError and the delete
crashed on a very common file shape.
Clamp that past-EOF end position to end-of-file in delete_text_between_positions,
mirroring the guard insert_text_at_position already has. A file with no trailing
newline now deletes its last line identically to one with a trailing newline.
Adds TextUtils unit tests and a CHANGELOG entry.
stepping line by line instead of character by character,
delegating the scan for line breaks to native string search
(~15-100x speedup for a full pass over typical source files)
* Add helper class TextStepper to facilitate computations
* Reimplement TextUtils methods based on TextStepper
* Adhere to LSP definition of newlines being defined by either "\n", "\r\n" or "\r" on its own.
Previously, only "\n" was considered.
* Add util method TextUtils.split_lines, allowing usages of Python's splitlines to be replaced
The cpu matrix job had no timeout-minutes, so a hung job runs until
GitHub's default 6-hour limit. This is not hypothetical: on run
29064865197 (2026-07-10), the native (ubuntu-latest) job hung and ran
from 02:36:34 to 08:36:50 before GitHub cancelled it at exactly the
6-hour cap, blocking the PR's status the whole time and burning a
runner for 6 hours.
Healthy runs are far below the new limit: across the 8 most recent
successful runs of this workflow, the slowest job was 16 minutes
(niche on ubuntu-latest), with catch-all on windows-latest typically
11-14 minutes. 60 minutes leaves ~4x headroom for cache-cold runs
(language-server downloads, npm installs) while cutting the worst-case
hang from 6 hours to 1.
Part of the timeout-hardening discussed in #1695.
Dominik Jain