Parse documents on a Celery parsing worker via a read_document tool

Replace the sandbox Docling extractor with read_document, backed by the in-process
backend parser (the same one ingestion uses) and offloaded to a dedicated
'parsing' Celery queue so it can run on GPU-capable workers with predictable RAM.
The tool resolves the input ref under the run-scoped gate, enqueues the parse,
and awaits it with a timeout (degrading to an error rather than hanging); the
worker independently re-resolves the artifact through the same gate and never
trusts a raw path. Untrusted files get the upload path's safeguards (extension
whitelist, size cap, sanitized temp file, cleanup). Options: output
(markdown/text/structured/chunks), ocr, pages, engine, max_chars, include_tables,
persist, json_schema. The workflow native-file 'extract' fallback now uses the
same worker path, so document parsing no longer needs the sandbox and works on
every backend.

Also fixes the branch's periodic-task test (the sandbox reaper made it 12) and
points the dev and e2e Celery workers at the parsing queue.
This commit is contained in:
Alex committed 2026-06-25 13:24:12 +01:00
1 parent 273c830b55
commit 37d93cbd86
25 files changed
+1611 -1275

No files matched your search

+5 -1
View File
@@ -29,9 +29,13 @@ serves only the WSGI Flask app — it omits `/mcp` and the reconnect reader
### Celery (Task Queue)
```bash
celery -A application.app.celery worker -l INFO
celery -A application.app.celery worker -l INFO -Q docsgpt,parsing
```
The `parsing` queue serves document parsing (the `read_document` tool / workflow
native-file parse); without it those calls hang `DOCUMENT_PARSE_TIMEOUT` then
error. A dedicated `-Q parsing` worker can be GPU-enabled for heavier parsers.
## Github Codespaces Instructions
### 1. Make Ports Public: