* perf(search_text): precompute line offsets for O(log n) line lookup

search_for_pattern resolved each match's line number by rescanning the
file from the beginning (TextUtils.get_line_from_index walks a
TextStepper from index 0), making coordinate resolution O(n) per match
and O(n*m) for m matches. search_text now precomputes line start
offsets once per file (O(n)) and resolves each match via binary search
(O(log n) per match).

The precomputed table mirrors TextStepper line semantics exactly:
\n, \r\n and bare \r are line separators, and an index pointing at the
\n of a \r\n pair resolves to (next line, 0) — the same rule
TextUtils.get_line_col_from_index implements, now pinned by regression
tests (including the mixed line-ending edge cases from the
insert_text_at_position fix).

On a synthetic 12k-line file (494 KB) with 723 matches, match-coordinate
resolution drops from ~1764 ms to ~2 ms (~1000x); the one-off
precompute costs ~1.1 ms. Small files regress nothing (100 lines, 5
matches: 70 µs -> 11 µs). Results are byte-for-byte identical to the
previous implementation.

Note: compatible with PR #1899 (interruptible agent regexes): that PR
touches the re.compile call site in search_text and the ContentReplacer
helpers, this PR touches the post-match line-coordinate resolution; the
two changes are complementary and rebase cleanly onto each other.

* perf(search_text): add reproducible benchmark script

scripts/profile_search_text.py regenerates the synthetic benchmark
content with a fixed seed (no real-world code), asserts both resolution
paths produce identical results, and prints the before/after timings
quoted in the PR description. Default: 12k lines / ~1830 matches;
optional CLI args scale the file down for quick runs.

* refactor(ls): encapsulate cached text coordinates in TextCoordinates abstraction

Address review feedback on the search_text line-lookup PR: the previously
added module-level helpers duplicated TextStepper's line-ending logic and
exposed raw offsets as a low-level data structure shared between two
functions, violating the project's software design principles (each
concern in exactly one home; dataclasses instead of tuples for simple
data storage).

Introduce in solidlsp/ls_utils.py:

- LineCol: frozen, kw-only dataclass for a 0-based line/column position
- TextCoordinates: encapsulates the cached table of line start offsets,
  built via TextStepper (step_line + line_start_idx), with a single
  public method line_col_at_index() resolving an index via binary
  search, including the CRLF rule (an index pointing at the '\n' of a
  '\r\n' pair denotes the beginning of the next line, column 0) and
  InvalidTextLocationError for out-of-range indices

search_text now instantiates TextCoordinates once per file and resolves
each match's coordinates through the public method; the private helpers
(_compute_line_starts, _line_col_at_index) are removed.

Equivalence with TextUtils.get_line_col_from_index verified across
mixed line endings, empty content and boundary indices (0 mismatches).

Tests are behaviour-anchored: the implementation-coupled parametrized
test importing the private helpers is replaced by public-contract tests
of TextCoordinates (test/solidlsp/test_ls_utils.py) and absolute line
assertions through the public search_text API. The benchmark script
uses the public API as well.

Benchmark (synthetic 12k lines, ~723 matches, best of 5): coordinate
resolution ~1764 ms -> ~2 ms; precompute ~1.1 ms; small files unchanged.

Semantics are unchanged; results are identical to the previously
proposed implementation and to the upstream reference implementation.

* fix(test): rename test_ls_utils to test_text_coordinates to avoid basename collision

The new test file collided with the pre-existing upstream
test/solidlsp/util/test_ls_utils.py: pytest (prepend import mode, no
__init__.py in the test dirs) requires unique basenames, and collecting
both modules produced an import-file-mismatch error that aborted
collection with exit code 2 across every CI matrix job.

Rename to test_text_coordinates.py (unique, and matches the class under
test). Full-suite collection verified locally (2252 tests, 0 errors).
This commit is contained in:
Guillermo authored and Dominik Jain committed 2026-09-23 14:03:08 +02:00
1 parent 704e8c3d19
commit fa79a2260f
6 files changed
+329 -4

No files matched your search

+110
View File
@@ -0,0 +1,110 @@
# SPDX-License-Identifier: GPL-3.0-or-later
"""Benchmark for search_text line-coordinate resolution (see PR "perf(search_text): precompute line offsets").
Reproduces the before/after numbers quoted in the PR description:
- "before" resolves each match's line number via ``TextUtils.get_line_from_index``,
which walks a TextStepper from index 0 (O(n) per match, O(n*m) for m matches)
- "after" precomputes line start offsets once (O(n)) and resolves each match via
binary search (O(log n) per match)
The content is fully synthetic and generated with a fixed seed: it contains no
code from any real project. Both paths must produce identical results; the script
asserts that before timing anything.
Usage: uv run python scripts/profile_search_text.py [num_lines] [num_matches]
"""
import random
import re
import sys
import time
import solidlsp # noqa: F401 # imported first: solidlsp must resolve before serena.util.text_utils (known circular-import window)
from solidlsp.ls_utils import TextCoordinates, TextUtils
def generate_synthetic_content(num_lines: int, match_identifier: str, match_frequency: float = 0.15) -> str:
"""
Generates synthetic pseudo-code content with a fixed seed.
:param num_lines: number of lines to generate
:param match_identifier: the identifier the benchmark pattern will search for
:param match_frequency: fraction of lines that reference the match identifier
:return: the synthetic content as a single string
"""
rng = random.Random(1234) # fixed seed for reproducibility
filler_identifiers = ["var_global_contador", "otra_funcion_aleatoria", "gestion_operaciones", "ajuste_niveles"]
lines = []
for i in range(num_lines):
kind = rng.random()
if kind < match_frequency:
lines.append(f" double resultado_{i} = {match_identifier}({rng.randint(1, 999)});")
elif kind < match_frequency + 0.15:
lines.append(f" if ({filler_identifiers[rng.randrange(len(filler_identifiers))]} > {rng.randint(0, 100)}) {{")
elif kind < match_frequency + 0.25:
lines.append(" }")
elif kind < match_frequency + 0.40:
lines.append(f"// comment line {i} with filler text {rng.randint(1000, 9999)}")
else:
lines.append(f" int local_variable_{i} = {rng.randint(0, 5000)};")
return "\n".join(lines) + "\n"
def resolve_before(content: str, compiled_pattern: re.Pattern) -> list[tuple[int, int]]:
"""Resolves line coordinates the way upstream currently does (TextStepper from index 0 per match)."""
results = []
for match in compiled_pattern.finditer(content):
start_pos, end_pos = match.start(), match.end()
start_line_num = TextUtils.get_line_from_index(content, start_pos)
end_line_num = TextUtils.get_line_from_index(content, end_pos)
if end_line_num > start_line_num and TextUtils.get_line_col_from_index(content, end_pos)[1] == 0:
end_line_num -= 1
results.append((start_line_num, end_line_num))
return results
def resolve_after(content: str, compiled_pattern: re.Pattern) -> list[tuple[int, int]]:
"""Resolves line coordinates the way this PR proposes (cached line starts + binary search)."""
coordinates = TextCoordinates(content)
results = []
for match in compiled_pattern.finditer(content):
start_pos, end_pos = match.start(), match.end()
s = coordinates.line_col_at_index(start_pos).line
end_loc = coordinates.line_col_at_index(end_pos)
e = end_loc.line
if e > s and end_loc.col == 0:
e -= 1
results.append((s, e))
return results
def main() -> None:
num_lines = int(sys.argv[1]) if len(sys.argv) > 1 else 12000
match_identifier = "funcion_calcula_parametro"
content = generate_synthetic_content(num_lines, match_identifier)
compiled_pattern = re.compile(match_identifier)
num_matches = len(list(compiled_pattern.finditer(content)))
print(f"synthetic file: {content.count(chr(10)):,} lines, {len(content):,} chars, {num_matches} matches")
before = resolve_before(content, compiled_pattern)
after = resolve_after(content, compiled_pattern)
assert before == after, "implementations disagree; this script only measures identical work"
print(f"results identical: OK ({len(before)} matches)")
for name, fn in [
("before (TextStepper, O(n) per match)", lambda: resolve_before(content, compiled_pattern)),
("after (bisect, O(log n) per match)", lambda: resolve_after(content, compiled_pattern)),
]:
times = []
for _ in range(5):
t0 = time.perf_counter()
fn()
times.append(time.perf_counter() - t0)
print(f"{name}: {min(times) * 1000:.2f} ms (best of 5)")
if __name__ == "__main__":
main()