mirror of
https://github.com/tiennm99/DocsGPT.git
synced 2026-10-15 14:16:05 +00:00
Parse documents on a Celery parsing worker via a read_document tool
Replace the sandbox Docling extractor with read_document, backed by the in-process backend parser (the same one ingestion uses) and offloaded to a dedicated 'parsing' Celery queue so it can run on GPU-capable workers with predictable RAM. The tool resolves the input ref under the run-scoped gate, enqueues the parse, and awaits it with a timeout (degrading to an error rather than hanging); the worker independently re-resolves the artifact through the same gate and never trusts a raw path. Untrusted files get the upload path's safeguards (extension whitelist, size cap, sanitized temp file, cleanup). Options: output (markdown/text/structured/chunks), ocr, pages, engine, max_chars, include_tables, persist, json_schema. The workflow native-file 'extract' fallback now uses the same worker path, so document parsing no longer needs the sandbox and works on every backend. Also fixes the branch's periodic-task test (the sandbox reaper made it 12) and points the dev and e2e Celery workers at the parsing queue.
This commit is contained in:
1 parent
273c830b55
commit
37d93cbd86
25 files changed
+1611
-1275
No files matched your search
@@ -276,7 +276,7 @@ class TestSetupPeriodicTasks:
|
||||
|
||||
setup_periodic_tasks(sender)
|
||||
|
||||
assert sender.add_periodic_task.call_count == 11
|
||||
assert sender.add_periodic_task.call_count == 12
|
||||
|
||||
calls = sender.add_periodic_task.call_args_list
|
||||
|
||||
@@ -308,6 +308,9 @@ class TestSetupPeriodicTasks:
|
||||
# schedule runs cleanup (24h)
|
||||
assert calls[10][0][0] == timedelta(hours=24)
|
||||
assert calls[10][1].get("name") == "cleanup-schedule-runs"
|
||||
# sandbox session reaper (60s)
|
||||
assert calls[11][0][0] == timedelta(seconds=60)
|
||||
assert calls[11][1].get("name") == "reap-sandbox-sessions"
|
||||
|
||||
|
||||
class TestMcpOauthTask:
|
||||
|
||||
Reference in new issue
Block a user