mirror of
https://github.com/tiennm99/DocsGPT.git
synced 2026-10-15 22:14:46 +00:00
fix: accurate per-call token usage and oversized-context guards
Token accounting: - Drain each tool round's provider stream to exhaustion before running tools and recursing, so the usage decorator persists exactly one token_usage row per LLM call, at call end. Previously every round's generator was abandoned mid-iteration and flushed together at request teardown, writing N near-identical rows stamped with the final round's provider counts (duplicate billing). - Consume the Chat Completions include_usage terminal chunk (it arrives after finish_reason and was never read) so streamed calls record provider-exact token counts instead of tiktoken estimates. - Claim provider-reported usage once per call (_last_usage_claimed) so a late-finalized generator can never adopt another call's counts. Oversized-context guards: - Enforce Responses API function_call/function_call_output pairing in the input builder (drop unpaired items; bypassed for store-mode previous_response_id chaining where calls are matched server-side). - Hard pre-send context gate: shrink oversized tool results and refuse payloads that cannot fit the model's window before dispatch, so a hopeless request is never sent or billed. - Cap a single tool result entering the LLM context (TOOL_RESULT_MAX_TOKENS, default 20000); the tool journal and persistence keep the full result. Applied on the resume/continuation path too. - Skip the fallback attempt when the payload cannot fit the fallback model's context window (10% estimation slack). - Compression: never save a compression point that does not reduce tokens; bound oversized verbatim fields kept after a compression point (COMPRESSION_RECENT_FIELD_MAX_TOKENS, default 8000). Robustness fixes from review: - Google parallel function calls: complete index-less ToolCalls are no longer merged into one another (dict arguments raised TypeError on +=; second call could execute with the first call's arguments). - Trailing-frame failures after a delivered answer no longer error the stream or restream the whole answer from the fallback (_stream_reached_finish). - In-memory compression falls back to minimal pruning when the summary is not smaller than the original. - keep<=0 guard in the middle-truncation helpers (a tiny cap returned marker + full text). Frontend: tooltip on the Analytics tokens stat card explaining that agent tool loops re-send conversation context on every step.
This commit is contained in:
1 parent
a7fd6d7c02
commit
ca5f80995d
16 files changed
+1626
-119
No files matched your search
@@ -210,6 +210,17 @@ class TestCompressionService:
|
||||
"""
|
||||
compression_service.llm.gen.return_value = mock_summary
|
||||
|
||||
# Real compressions run near a context-window threshold, so the
|
||||
# original is always far larger than the summary. Pad the fixture
|
||||
# accordingly — a summary bigger than the original is rejected by
|
||||
# the negative-savings guard.
|
||||
sample_conversation["queries"][0]["response"] = (
|
||||
"Python is a high-level programming language. " * 40
|
||||
)
|
||||
sample_conversation["queries"][1]["response"] = (
|
||||
"You can install Python from python.org. " * 40
|
||||
)
|
||||
|
||||
# Compress first 2 queries
|
||||
result = compression_service.compress_conversation(
|
||||
conversation=sample_conversation, compress_up_to_index=1
|
||||
|
||||
Reference in new issue
Block a user