fix(worker): know you are in a worker from any thread, not only the task's

Celery records the executing task on the thread that runs it, so a thread that
task starts sees none. The embeddings client and read_document both decided
"am I in a worker?" from that alone, and from any other thread took the
web-process branch: dispatch to the worker they were running in and block on
the result. Celery refuses that get() ("Never call result.get() within a
task!"), so the embed failed and latched the 30s dispatch cooldown for every
caller after it; with joins allowed, read_document would instead wait on a
parsing queue only its own busy process serves.

Threads inside tasks are not hypothetical: per-source retrieval fans out to a
pool, so a scheduled or webhook agent searching several sources embedded from
pool threads. Graph extraction did too, which failed every chunk of a build.

in_worker() in celery_init answers for the whole process. Celery's
task_join_will_block is process-wide and set for every blocking pool (prefork,
solo, threads) -- exactly the condition under which dispatch-and-wait goes
wrong; eventlet/gevent leave it unset, so the task's own thread still counts
through current_worker_task. Verified with real workers on each blocking pool:
from a thread a task started, the old check dispatched and hit the error, the
new one embedded locally.
This commit is contained in:
Alex committed 2026-09-19 14:07:42 +01:00
1 parent 5e67f8c927
commit 15bdda8554
6 files changed
+117 -16

No files matched your search

@@ -357,9 +357,9 @@ def test_malformed_json_schema_rejected_before_enqueue(monkeypatch):
@pytest.mark.unit
def test_dispatch_inline_when_in_worker(monkeypatch):
_stub_repo(monkeypatch, found=True, conv="conv-1", run=None)
# Inside a worker current_task is truthy -> parse inline, never enqueue (else the
# Inside a worker -> parse inline, never enqueue (else the
# parsing queue self-deadlocks the worker that also serves it).
monkeypatch.setattr(rd, "current_task", object())
monkeypatch.setattr(rd, "in_worker", lambda: True)
import docsgpt.api.user.tasks as tasks
monkeypatch.setattr(
@@ -387,8 +387,8 @@ def test_dispatch_inline_when_in_worker(monkeypatch):
@pytest.mark.unit
def test_dispatch_enqueues_when_not_in_worker(monkeypatch):
_stub_repo(monkeypatch, found=True, conv="conv-1", run=None)
# Web process: current_task falsy -> dispatch to the parsing queue, never inline.
monkeypatch.setattr(rd, "current_task", None)
# Web process -> dispatch to the parsing queue, never inline.
monkeypatch.setattr(rd, "in_worker", lambda: False)
captured = _patch_task(monkeypatch, payload={"status": "ok", "content": "queued", "truncated": False})
import docsgpt.worker as worker
@@ -414,9 +414,9 @@ _TIMED_OUT = "document parsing timed out after"
def _inline(monkeypatch, run_parse, *, timeout=0.2) -> ReadDocumentTool:
"""Drive the inline branch (current_task truthy) with a patched parse window."""
"""Drive the inline (in-worker) branch with a patched parse window."""
_stub_repo(monkeypatch, found=True, conv="conv-1", run=None)
monkeypatch.setattr(rd, "current_task", object())
monkeypatch.setattr(rd, "in_worker", lambda: True)
import docsgpt.api.user.tasks as tasks
monkeypatch.setattr(