mirror of
https://github.com/tiennm99/DocsGPT.git
synced 2026-10-18 06:13:54 +00:00
fix(attachments,usage): validate content behind a BOM, judge by the live parser table
Review follow-ups. A BOM told the sniff which encoding to read, but was also taken as the verdict: three prepended bytes let any binary through, including as notes.txt. A BOM now only selects the test — UTF-8 falls through to the byte rules on the remainder, UTF-16/32 decode and judge the characters (NUL, unprintable, or replacement chars from bytes the decoder could not read). Real Notepad-Unicode text still passes, mp4-behind-a-BOM does not, in either language. The gate treated the full parser table as a given, but without docling the fallback extractor has no .tif/.tiff/.bmp/.webp/.vtt/.xml handler, so those suffixes skipped the content check and reached the plain-text fallthrough — the original bug, one install away. The worker now passes the keys of the extractor it actually built, making the second gate stricter than the route's static one rather than a copy of it. Cache bins: extraction coerced a missing value to 0 and only non-zero bins were recorded, so a provider reporting cached_tokens=0 persisted as NULL — indistinguishable from "not reported", and OpenAI reports exactly that on every uncached request. Bins are now carried as Optional and recorded when not None, which is what the nullable columns and the NULL-means-unknown comment already assumed. Anthropic's cache_read/cache_creation bins had the same shape and are fixed alongside; the int-or-None coercion is shared in llm/base.py.
This commit is contained in:
1 parent
f7cd94668f
commit
9db11f18f5
10 files changed
+341
-66
No files matched your search
@@ -1583,5 +1583,26 @@ def test_record_responses_metadata_captures_cache_write_bin(monkeypatch):
|
||||
),
|
||||
))
|
||||
# A pure cache-write turn (first request of a prefix) has reads=0 and
|
||||
# writes>0; the details must still be recorded.
|
||||
assert llm._last_usage["prompt_tokens_details"] == {"cache_write_tokens": 9}
|
||||
# writes>0; both bins were reported, so both are recorded — a reported
|
||||
# zero is "no cache hits", not "unknown".
|
||||
assert llm._last_usage["prompt_tokens_details"] == {
|
||||
"cached_tokens": 0,
|
||||
"cache_write_tokens": 9,
|
||||
}
|
||||
|
||||
|
||||
@pytest.mark.unit
|
||||
def test_record_responses_metadata_omits_unreported_cache_bins(monkeypatch):
|
||||
"""A provider that says nothing about caching must persist as NULL, not 0."""
|
||||
llm = _make_llm(monkeypatch, _responses_caps())
|
||||
llm._record_responses_metadata(_ns(
|
||||
id="resp_2",
|
||||
usage=_ns(
|
||||
input_tokens=10,
|
||||
output_tokens=7,
|
||||
total_tokens=17,
|
||||
input_tokens_details=None,
|
||||
output_tokens_details=None,
|
||||
),
|
||||
))
|
||||
assert "prompt_tokens_details" not in llm._last_usage
|
||||
Reference in new issue
Block a user