mirror of
https://github.com/tiennm99/DocsGPT.git
synced 2026-10-14 12:16:20 +00:00
fix(llm,compression): bound cross-turn Responses chaining and make compression stick
In store mode every user turn chained onto the previous response, so the provider's stored transcript grew without bound (measured: 889k prompt tokens for a 37k-token saved history) while every local guard, the compression pipeline included, measured the saved history. Each chained tool round also re-sent the system message, which the server appends rather than dedupes, and a saved compression point was applied exactly once, in the turn that made it. Chaining is now bounded. A turn starts from the saved history when the previous turn's reported prompt reached the chain budget (default: the model's context window), when the conversation was compressed after that turn was produced, or when OPENAI_RESPONSES_CHAIN_ACROSS_TURNS is off. Chained rounds omit an unchanged system head (hash carried in the persisted Responses state). truncation="auto" is available behind a setting as a backstop against a chain that outgrows the model's window. Compression: a saved point is applied at every turn start; the threshold counts the summary plus the queries after the point instead of the raw history; re-compression summarises only the tail on top of the last point; the mid-execution path marks itself persisted and resets the provider chain so the rebuilt messages are the context; an empty summary is rejected; the visible "[Context Compression Summary]" rows are no longer replayed as history; appending the same point twice is a no-op. Cache hints: a per-user prompt_cache_key and an optional prompt_cache_retention on Responses API calls. Measured on Azure with the same client shape as production (stateless OpenAI client, server-side tools, PDF part): tokens billed on the sixth turn fell from 58k to 35k, tool rounds add tens of tokens instead of ~2.8k, the turn after a compression reused the saved summary in under two seconds instead of re-summarising, and the round after a mid-execution compression started from the compressed context instead of the full stored transcript.
This commit is contained in:
1 parent
4b56e56649
commit
04d358ab69
19 files changed
+1099
-38
No files matched your search
@@ -66,6 +66,9 @@ For OpenAI models that support it, DocsGPT can call the newer **Responses API**
|
||||
|
||||
- `reasoning_effort` — per-model reasoning effort hint (for example `medium`) declared in the model catalog.
|
||||
- `OPENAI_RESPONSES_STORE` (default `false`) — when `true`, lets OpenAI persist Responses API state server-side and chains calls with `previous_response_id`. When `false`, DocsGPT requests encrypted reasoning items and persists those ciphertext items itself so reasoning survives tool calls and later conversation turns without OpenAI retaining the response.
|
||||
- `OPENAI_RESPONSES_CHAIN_ACROSS_TURNS` (default `true`) and `OPENAI_RESPONSES_CHAIN_BUDGET_TOKENS` (default: the model's `context_window`) — in store mode a new user turn chains onto the previous response, which keeps the provider's stored transcript and its prompt cache warm. The chain is bounded: once the previous turn's reported prompt reaches the budget, or after the conversation history has been compressed, the next turn starts from DocsGPT's own saved history instead. Chained tool rounds do not re-send an unchanged system message.
|
||||
- `OPENAI_RESPONSES_TRUNCATION_AUTO` (default `false`) — send `truncation: "auto"` so the provider trims middle items instead of failing a request that exceeds the model's window.
|
||||
- `OPENAI_PROMPT_CACHE_KEY` (default `true`) and `OPENAI_PROMPT_CACHE_RETENTION` (default unset) — prompt-cache hints on Responses API calls: a per-user `prompt_cache_key`, and a `prompt_cache_retention` value such as `24h` where the provider supports it.
|
||||
- `V1_SESSION_TTL_SECONDS` (default `86400`) — lifetime of the hashed Redis mapping used to associate OpenAI-compatible client session headers with hidden DocsGPT conversations. Explicit DocsGPT conversation IDs always take precedence.
|
||||
|
||||
See [App Configuration](/Deploying/DocsGPT-Settings) for the full settings reference.
|
||||
|
||||
Reference in new issue
Block a user