mirror of
https://github.com/tiennm99/DocsGPT.git
synced 2026-10-23 06:17:44 +00:00
Alex
04d358ab69
In store mode every user turn chained onto the previous response, so the provider's stored transcript grew without bound (measured: 889k prompt tokens for a 37k-token saved history) while every local guard, the compression pipeline included, measured the saved history. Each chained tool round also re-sent the system message, which the server appends rather than dedupes, and a saved compression point was applied exactly once, in the turn that made it. Chaining is now bounded. A turn starts from the saved history when the previous turn's reported prompt reached the chain budget (default: the model's context window), when the conversation was compressed after that turn was produced, or when OPENAI_RESPONSES_CHAIN_ACROSS_TURNS is off. Chained rounds omit an unchanged system head (hash carried in the persisted Responses state). truncation="auto" is available behind a setting as a backstop against a chain that outgrows the model's window. Compression: a saved point is applied at every turn start; the threshold counts the summary plus the queries after the point instead of the raw history; re-compression summarises only the tail on top of the last point; the mid-execution path marks itself persisted and resets the provider chain so the rebuilt messages are the context; an empty summary is rejected; the visible "[Context Compression Summary]" rows are no longer replayed as history; appending the same point twice is a no-op. Cache hints: a per-user prompt_cache_key and an optional prompt_cache_retention on Responses API calls. Measured on Azure with the same client shape as production (stateless OpenAI client, server-side tools, PDF part): tokens billed on the sixth turn fell from 58k to 35k, tool rounds add tens of tokens instead of ~2.8k, the turn after a compression reused the saved summary in under two seconds instead of re-summarising, and the round after a mid-execution compression started from the compressed context instead of the full stored transcript.