- CI installs the backend requirements from docsgpt/; the old cd into
application/ silently installed nothing.
- The root .dockerignore re-admits only application/__init__.py. An upgraded
checkout may still hold gitignored application/{inputs,indexes,vectors,.env}
from the old layout, and the directory rule shipped them into the image.
- The compose files keep the host bind mounts on application/{indexes,inputs,
vectors}, so an upgrade does not start with empty data. The move comes with
the packaging work, together with an upgrade note.
- The alias loader puts the real docsgpt spec back on the shared module object
after import (the import machinery stamped the alias spec on it, which made
importlib.reload rename the module and skip re-execution) and delegates
get_code/get_source/get_filename to the target loader, so
python -m application.<name> runs.
- Each legacy application.* task name is registered as its own task object,
a subclass carrying the old name. Registering the same object under two
keys made Celery's tracer log every run under whichever name it built last.
- The redbeat key prefix stays redbeat:docsgpt:; the three schedule_syncs
entries get stable names instead. redbeat tracks its static entries and
deletes the ones that vanish from beat_schedule at start-up, and rewrites the
task path of named entries in place, so neither a prefix bump nor a cleanup
pass is needed (checked against redbeat 2.4.2 with a seeded Redis).
The backend import package is now docsgpt, the name it will carry on PyPI;
application was far too generic to install into anyone's site-packages.
git mv plus a mechanical rewrite of every import, dotted string and path
reference: 734 Python files, the compose files, Dockerfile, workflows, docs,
setup scripts, devcontainer, k8s manifests, vscode config, pytest and coverage
config, .gitignore. Behaviour is unchanged.
Kept for one release:
- A top-level application package whose meta-path finder resolves
application.x.y to the already-imported docsgpt.x.y object, so old imports
and entry points (celery -A application.app.celery,
uvicorn application.asgi:asgi_app) keep working with a FutureWarning.
- Celery registers every application.* task name as an alias of its
docsgpt.* task on start-up, so messages queued by the previous release still
run. The redbeat key prefix moves to redbeat:docsgpt:v2: so schedule entries
the previous release wrote are left unread instead of firing twice.
The backend image builds from the repository root (docker build -f
docsgpt/Dockerfile .) so it can ship the alias package; a root .dockerignore
allow-lists docsgpt/ and application/ and keeps caches, local data, .env
files, the sample index files and the Dockerfile out. Compose and the image
workflows point at the new context.
The Vite dev server (docker-compose.yaml's dev target) lets an empty process
variable override .env.development, so listing every VITE_* as ${VAR:-}
wiped the banner and Google client id defaults. A bare name in the
environment list reaches the container only when the variable is set in the
shell or the --env-file; unset ones are omitted. Applied to all four compose
files.
- The frontend image ran the Vite dev server in development mode, so
.env.development supplied its defaults (notification banner, Google client
id, local API host). The static build only loads .env.production, so the
build stage now copies .env.development in as the baseline and the compose
files pass every VITE_* the app reads through from .env; the runtime script
skips empty values so a blank passthrough keeps the build-time default.
.dockerignore kept only the .local variants out.
- VITE_DISABLE_SOURCE_FE disables sources only when it is the string true.
- DoclingParser: find_spec raises when docling itself is absent; the install
hint now covers that path, with a regression test.
- verify_offline: direct tests for verify(); the PR image check builds and
verifies the -docling variant as well as slim.
- Workflows this branch adds or rewrites pin actions by commit, pass the
release tag through env instead of template expansion, and do not persist
checkout credentials.
- OCR guide no longer claims pre-built images never include docling.
Backend (arc53/docsgpt): 4.5 GB compressed -> 0.9 GB with both embedding
models and tiktoken baked in.
- torch/transformers gone from the default install (docling extra only).
- Ubuntu 24.04 ships python3.12: no deadsnakes PPA, no software-properties-
common; every pin is a wheel, so no gcc/g++/rust in the builder.
- COPY --chown and a prefetch that runs as the process user replace the
trailing chown -R, which duplicated the 600 MB model layer.
- .dockerignore keeps __pycache__, .coverage, local indexes and .env out.
- EXTRAS build arg (INSTALL_DOCLING kept as an alias); the docling variant
also bakes docling's layout/table/RapidOCR models (DOCLING_ARTIFACTS_PATH)
and tesseract, and drops only the discovery documents of Google APIs the
app never builds.
- FLASK_DEBUG env removed (unused); OCI labels added.
Frontend (arc53/docsgpt-fe): 302 MB Vite dev server -> 25 MB static build
behind nginx. VITE_* variables are injected at container start into
/config.js and read through src/env.ts, so the image no longer needs a
rebuild per deployment; docker-compose.yaml keeps hot reload via the dev
target.
Publishing: every release and develop build now pushes a slim tag and a
-docling tag (docling engine + models + tesseract). docker-compose-hub.yaml
takes DOCSGPT_IMAGE_TAG / DOCSGPT_IMAGE_VARIANT; docker-compose-standalone.yaml
runs the stack from pre-built images without a checkout and is attached to
each release. setup.sh selects the -docling variant for OCR instead of
requiring a local build. A new workflow builds the image on PRs that touch
it and runs verify_offline under --network none; lint checks the exported
requirements match uv.lock.
Conflicts, and how each was taken:
- application/core/settings.py — ours. The renamed OCR_ENABLED /
OCR_ATTACHMENTS_ENABLED / OCR_MIN_CHARS_PER_PAGE accept main's
DOCLING_OCR_* spellings as AliasChoices, so nothing is dropped.
- application/Dockerfile — both. Main's install layers plus the
INSTALL_DOCLING build arg.
- application/parser/file/constants.py — both imports.
- deployment/docker-compose.yaml — both. The INSTALL_DOCLING /
INSTALL_TESSERACT build args on backend and worker, and main's
-Q docsgpt,parsing,embeddings, which query embedding needs.
- tests/conftest.py — theirs. Both sides fixed the same pytest-postgresql
9.0.0 autocommit= breakage; main's spelling is the one already on main.
- application/requirements.txt — the comments claimed different reasons
torch is in core. Main's is the true one now: it removed
sentence-transformers, so docling is torch's only remaining consumer.
Two things the merge broke without conflicting:
- onnxruntime. This branch moved it out of core into the docling extra;
main meanwhile made it the runtime local embeddings execute on
(fastembed). Git took the deletion, leaving fastembed with no pinned
runtime in a repo that pins everything. Restored to core, and no longer
pinned twice from the extra.
- The frontend copy of ATTACHMENT_PARSER_EXTENSIONS. The backend list is
derived and picked up the anydoc suffixes; the hand-kept frontend mirror
did not, so the composer would refuse files the API accepts.
tests/parser/file/test_constants.py is what caught it.
ruff, pytest (9897 passed), frontend build and docs build all pass. The
image build is unverified: no Docker daemon on this machine.
Query embedding moved to the Celery worker, but nothing that ships was
updated to consume the queue it dispatches to.
- Add `embeddings` to every worker `-Q` list (compose x3, k8s, devcontainer,
sandbox README). Without it a search blocked for EMBEDDINGS_DELEGATE_TIMEOUT
and then answered with no retrieved context, because classic_rag swallows the
dispatch error and skips the source -- bad answers, not an error.
- Skip the task_postrun heap reclaim for the embed task. The full gc.collect()
was written for docling/torch parses; on a worker holding the ONNX model it
measured ~86ms against ~8ms for the embed itself, a 9x slowdown of the round
trip for a task that allocates a few kilobytes.
- Resolve the installation pin in the re-embed script. It never imports
application.app, so an install pinned in app_metadata with no EMBEDDINGS_NAME
set -- every stock k8s deployment, whose manifests carry no embedding config
-- would rewrite its whole index with the legacy default and stamp
sources.model to match, then be told by the boot warning to run it again.
- Fail fast for 30s after a failed dispatch. fanout.embed_questions falls back
to letting each store embed its own query, so one dead-worker retrieval paid
the timeout once in the fan-out and again per source.
- Forget the task result. Nothing reads it back: the key is per-dispatch UUID,
not content-addressed, so a repeated query mints another. Left alone every
search leaked ~17KB for result_expires (7 days) into the Redis the broker
shares -- on the bundled k8s manifest (1Gi, no maxmemory policy) that is an
OOMKill that takes the broker with it.
- Release the model ensure_vector_schema loads to read the width of an
unregistered model, in a process that delegates and would never call it.
The width still comes from the model, not the table, so the mismatch check
the hook exists for keeps working.
- Correct the docs that said otherwise: embeddings.md claimed the standard
deployment worked unchanged, upgrading.mdx said no action was needed, and
the settings table listed none of the three delegation settings.
Replace the sandbox Docling extractor with read_document, backed by the in-process
backend parser (the same one ingestion uses) and offloaded to a dedicated
'parsing' Celery queue so it can run on GPU-capable workers with predictable RAM.
The tool resolves the input ref under the run-scoped gate, enqueues the parse,
and awaits it with a timeout (degrading to an error rather than hanging); the
worker independently re-resolves the artifact through the same gate and never
trusts a raw path. Untrusted files get the upload path's safeguards (extension
whitelist, size cap, sanitized temp file, cleanup). Options: output
(markdown/text/structured/chunks), ocr, pages, engine, max_chars, include_tables,
persist, json_schema. The workflow native-file 'extract' fallback now uses the
same worker path, so document parsing no longer needs the sandbox and works on
every backend.
Also fixes the branch's periodic-task test (the sandbox reaper made it 12) and
points the dev and e2e Celery workers at the parsing queue.
Run each kernel under a scrubbed environment so untrusted code can never read
the host's secrets. A custom 'docsgpt-python' kernelspec launches ipykernel
through a wrapper that keeps only what the kernel needs (PATH, HOME, LANG, and
the Jupyter runtime/data dirs), dropping API keys, tokens, the database URL, and
the gateway token. The app selects this kernel by name via SANDBOX_KERNEL_NAME,
so the distinct name is never shadowed by the stock python3 spec. Per-session
workspaces are created mode 0700 (defense in depth under the shared uid). The
README documents the runner as a single trust domain and points to the Daytona
backend for per-tenant isolation.
Introduce a pluggable CodeSandbox abstraction with a SandboxManager and a
Jupyter Kernel Gateway backend: the app is a client of a single always-on
runner that executes code in stateful in-process kernels (no child-container
spawning, no docker socket). Sessions bind to a conversation or workflow run
with an agent-selectable TTL clamped by a global cap; execution enforces a
wall-clock deadline with interrupt-on-timeout and capped output, and file
transfer is workspace-contained with size and integrity checks. Adds a
docsgpt-sandbox docker-compose service (resource-capped, read-only, internal
network) plus settings, with a real local-gateway integration test.
* fixes setup scripts
fixes to env handling in setup script plus other minor fixes
* Remove var declarations
Declarations such as `LLM_PROVIDER=$LLM_PROVIDER` override .env variables in compose
Similar issue is present in the frontend - need to choose either to switch to separate frontend env or keep as is.
* Manage apikeys in settings
1. More pydantic management of api keys.
2. Clean up of variable declarations from docker compose files, used to block .env imports. Now should be managed ether by settings.py defaults or .env
Alex