insert_text_at_position returns the position of the end of the inserted text.
That position was computed by _get_updated_position_from_line_and_column_and_edit,
which steps the inserted text in isolation and adds the resulting line/column
delta to the start position.
An insertion is not independent of its surroundings. Under the LSP newline
definition adopted in 9208a3ae, a "\n" inserted directly after an existing "\r"
merges with it into a single "\r\n" sequence, so the text gains no line while the
isolated arithmetic still counts one:
insert_text_at_position("\r", 1, 0, "\n") -> ("\r\n", 2, 0); actual (1, 0)
insert_text_at_position("x\ry", 1, 0, "\n") -> ("x\r\ny", 2, 0); actual (1, 0)
insert_text_at_position("a\r", 2, 0, "X") -> ("a\r\nX", 2, 1); actual (1, 1)
The third case reaches the same fault through the one-line-past-EOF branch, which
prepends a "\n" that then merges with the trailing "\r". The converse interaction
was wrong too: inserting between the "\r" and the "\n" of a "\r\n" splits it, and
the isolated arithmetic reported (0, 2) -- a column on a line the split had just
emptied. In every case the resulting text was correct; only the position was wrong.
9208a3ae centralised line/column computation on TextStepper and reimplemented the
TextUtils methods on it. This helper was rebased onto TextStepper in that commit
but kept its original shape: it steps one string, and the string it steps is the
insertion rather than the result. Once a newline can be formed by two adjacent
characters, no computation over the inserted text alone can be correct.
The position is now determined from the resulting text via get_line_col_from_index,
the same function that resolves positions everywhere else, which removes the last
computation in TextUtils that did not go through the centralised path. The helper
had no other callers and is deleted, along with the line/col adjustment in the
past-EOF branch that existed only to feed it.
The column is clamped to the end of the text before both the splice and the
derivation. This is not a new tolerance: the splice already clamped implicitly
(text[:i] with i > len(text) yields the whole text), so an out-of-range column has
always appended successfully. Making the clamp explicit keeps the position derived
from the same index the splice used; without it, deriving from the result would
newly raise for input that previously worked.
Columns are offsets into the Python string (code points), which the docstring now
states. LSP columns are UTF-16 code units and nothing converts between the two at
the boundary -- a real defect, but a pre-existing and separate one: the previous
arithmetic did not preserve UTF-16 units either, since it added a code-point length
to a code-unit column and so produced a value in neither unit.
"\U0001f600" + "\U0001f600" at column 2
old (0, 3); UTF-16 (0, 4); code points (0, 2)
Seven regression tests cover the boundary, mid-document and past-EOF merges, the
newline split, an out-of-range column, a column past a non-BMP character, and an
unaffected "\n"-terminated control. The six that pin changed behaviour were each
verified to fail before the change; the control passes either way.
delete_lines/replace_lines address the position one line past the last line
(line == number of lines, col 0). get_index_from_line_col cannot resolve that
position when the file has no trailing newline (no closing newline to count),
so delete_text_between_positions raised InvalidTextLocationError and the delete
crashed on a very common file shape.
Clamp that past-EOF end position to end-of-file in delete_text_between_positions,
mirroring the guard insert_text_at_position already has. A file with no trailing
newline now deletes its last line identically to one with a trailing newline.
Adds TextUtils unit tests and a CHANGELOG entry.
stepping line by line instead of character by character,
delegating the scan for line breaks to native string search
(~15-100x speedup for a full pass over typical source files)
* Add helper class TextStepper to facilitate computations
* Reimplement TextUtils methods based on TextStepper
* Adhere to LSP definition of newlines being defined by either "\n", "\r\n" or "\r" on its own.
Previously, only "\n" was considered.
* Add util method TextUtils.split_lines, allowing usages of Python's splitlines to be replaced
xor