The Windows installer only checked that uv.exe exists after running the uv
installer, so a failed install that left an older uv.exe behind was accepted;
it now fails on a nonzero exit code.
sg runs its command with /bin/sh, which need not be bash, so the handoff
after installing Docker quotes each argument as POSIX single quotes instead
of with bash's printf %q.
Both installers download the pinned uv installer to a file and run it only
when its sha256 matches the value pinned next to UV_VERSION; bumping the
version means bumping the hash. Astral publishes checksums for the uv
binaries but not for the installer scripts, so the hash is pinned here.
get.docker.com is still only downloaded in full before running: its content
changes over time and it publishes no checksum.
- install.sh saves the get.docker.com and uv installers to a file and runs
them only after the download finished, so a cut-off transfer runs nothing.
- Neither installer prints DOCSGPT_PACKAGE, which may be a URL with
credentials.
- The CI step assigns the wheel path before exporting it, so a missing wheel
fails instead of installing from PyPI.
- Docker-Deploying shows one code block per platform; Quickstart names the
/opt/docsgpt home used for root on Linux.
deployment/install.sh (curl | bash) and install.ps1 (irm | iex) check for
Docker, install uv when it is missing or older than 0.8 (pinned 0.12.15 via
Astral's installer), install or upgrade the docsgpt package with
`uv tool install`, and hand the terminal to `docsgpt up` with any arguments.
On Linux without Docker the shell installer offers get.docker.com. Both run
entirely inside a function, so a download cut short runs nothing.
Releases attach both scripts next to the Compose file, which is where
docs.ac/install and docs.ac/install.ps1 will point. installer-lint.yml runs
shellcheck and the PowerShell parser; docker-image-verify.yml now installs
through install.sh. README, Quickstart, Docker-Deploying and the changelog
lead with the one-liner.
Docker-Deploying gains a `docsgpt up` section, Pip-Install and Upgrading
describe the ~/.docsgpt/server data home, and the changelog covers both.
docker-image-verify.yml installs the wheel and runs `docsgpt up`, `status`,
a second `up` that must keep the secrets, and `uninstall --purge` against
the image it built. The standalone Compose file maps host.docker.internal
to the host gateway, so a model server on a Linux host is reachable the way
`docsgpt up` suggests.
The backend image builds the web UI with scripts/build_frontend.sh and
serves it through docsgpt/ui.py, so the standalone Compose file drops the
frontend container. UI and API share port 7091, published on 127.0.0.1
unless DOCSGPT_BIND says otherwise. POSTGRES_PASSWORD is configurable, and
an optional https profile puts Caddy in front of a public domain.
docker-image-verify.yml starts the standalone stack on the image it built
and checks the API, the UI, /config.js and a client-side route on one port.
deployment/k8s/deployments/sandbox-deploy.yaml pulls arc53/docsgpt-sandbox,
which has never been pushed anywhere: Compose builds the runner from the
checkout (`build: ./sandbox`), but Kubernetes cannot build, so enabling code
execution on a cluster failed on an image that does not exist.
Build and push it like the other two images: `develop` on a push to main that
touches deployment/sandbox, and `<version>` plus `latest` when the release
workflow calls it. Release and develop live in one file here rather than two,
because the runner changes rarely and the only difference is which tags move.
The tag comes from the inputs and the release payload, not from
`github.event_name`, which is `push` when backend-release calls this.
- The post-task reclaim skip recognises the legacy application.* embed name,
so query embeds queued by the previous release do not pay a full collect.
- The Azure compose file mounts host data on /app/{indexes,inputs,vectors},
where the process actually reads and writes; it mounted /app/application/...
before the rename and /app/docsgpt/... after it, and nothing wrote to either.
- The offline image check triggers on docsgpt/requirements*.txt again;
dependabot's pip entry points at docsgpt/.
- install_hint() and its docstring name docsgpt/requirements-<extra>.txt;
the test asserts the full path.
- application/vectors/ stays ignored: the compose files still mount it.
- The durability QA script quiets the docsgpt logger tree.
- Upgrade note: the three renamed source-sync entries start their timers
from the upgrade.
- CI installs the backend requirements from docsgpt/; the old cd into
application/ silently installed nothing.
- The root .dockerignore re-admits only application/__init__.py. An upgraded
checkout may still hold gitignored application/{inputs,indexes,vectors,.env}
from the old layout, and the directory rule shipped them into the image.
- The compose files keep the host bind mounts on application/{indexes,inputs,
vectors}, so an upgrade does not start with empty data. The move comes with
the packaging work, together with an upgrade note.
- The alias loader puts the real docsgpt spec back on the shared module object
after import (the import machinery stamped the alias spec on it, which made
importlib.reload rename the module and skip re-execution) and delegates
get_code/get_source/get_filename to the target loader, so
python -m application.<name> runs.
- Each legacy application.* task name is registered as its own task object,
a subclass carrying the old name. Registering the same object under two
keys made Celery's tracer log every run under whichever name it built last.
- The redbeat key prefix stays redbeat:docsgpt:; the three schedule_syncs
entries get stable names instead. redbeat tracks its static entries and
deletes the ones that vanish from beat_schedule at start-up, and rewrites the
task path of named entries in place, so neither a prefix bump nor a cleanup
pass is needed (checked against redbeat 2.4.2 with a seeded Redis).
The backend import package is now docsgpt, the name it will carry on PyPI;
application was far too generic to install into anyone's site-packages.
git mv plus a mechanical rewrite of every import, dotted string and path
reference: 734 Python files, the compose files, Dockerfile, workflows, docs,
setup scripts, devcontainer, k8s manifests, vscode config, pytest and coverage
config, .gitignore. Behaviour is unchanged.
Kept for one release:
- A top-level application package whose meta-path finder resolves
application.x.y to the already-imported docsgpt.x.y object, so old imports
and entry points (celery -A application.app.celery,
uvicorn application.asgi:asgi_app) keep working with a FutureWarning.
- Celery registers every application.* task name as an alias of its
docsgpt.* task on start-up, so messages queued by the previous release still
run. The redbeat key prefix moves to redbeat:docsgpt:v2: so schedule entries
the previous release wrote are left unread instead of firing twice.
The backend image builds from the repository root (docker build -f
docsgpt/Dockerfile .) so it can ship the alias package; a root .dockerignore
allow-lists docsgpt/ and application/ and keeps caches, local data, .env
files, the sample index files and the Dockerfile out. Compose and the image
workflows point at the new context.
The Vite dev server (docker-compose.yaml's dev target) lets an empty process
variable override .env.development, so listing every VITE_* as ${VAR:-}
wiped the banner and Google client id defaults. A bare name in the
environment list reaches the container only when the variable is set in the
shell or the --env-file; unset ones are omitted. Applied to all four compose
files.
- The frontend image ran the Vite dev server in development mode, so
.env.development supplied its defaults (notification banner, Google client
id, local API host). The static build only loads .env.production, so the
build stage now copies .env.development in as the baseline and the compose
files pass every VITE_* the app reads through from .env; the runtime script
skips empty values so a blank passthrough keeps the build-time default.
.dockerignore kept only the .local variants out.
- VITE_DISABLE_SOURCE_FE disables sources only when it is the string true.
- DoclingParser: find_spec raises when docling itself is absent; the install
hint now covers that path, with a regression test.
- verify_offline: direct tests for verify(); the PR image check builds and
verifies the -docling variant as well as slim.
- Workflows this branch adds or rewrites pin actions by commit, pass the
release tag through env instead of template expansion, and do not persist
checkout credentials.
- OCR guide no longer claims pre-built images never include docling.
settings.EMBEDDINGS_NAME still defaults to mpnet so an upgraded deployment
keeps its index; new installs only get granite because setup.sh writes it to
.env. The standalone compose always starts on fresh volumes, so it defaults
EMBEDDINGS_NAME to granite (overridable from .env or the shell), and the
Docker guide's hand-written .env examples name the model too.
The worker hands finished indexes to the API through an internal endpoint
that rejects every request while INTERNAL_KEY is unset, so a .env written by
hand from the docs' minimal example made every upload end in a 401. setup.sh
generates the key; the standalone compose header and the Docker guide now
generate one too.
A named volume mounted on /app/inputs, /app/indexes or /app/vectors inherits
the ownership of the image directory, so the image creates them as appuser;
before this the volume came up root-owned and every upload failed with a
permission error unless the container ran as root. docker-compose-standalone.yaml
still runs backend and worker as root, like docker-compose-hub.yaml, so it
also works with image tags that predate these directories.
requirements-docling.txt adds the PyTorch CPU index, which uv resolves only
with UV_INDEX_STRATEGY=unsafe-best-match (pip is unaffected); the file header
and the docs say so and point uv users at uv sync --extra docling.
Backend (arc53/docsgpt): 4.5 GB compressed -> 0.9 GB with both embedding
models and tiktoken baked in.
- torch/transformers gone from the default install (docling extra only).
- Ubuntu 24.04 ships python3.12: no deadsnakes PPA, no software-properties-
common; every pin is a wheel, so no gcc/g++/rust in the builder.
- COPY --chown and a prefetch that runs as the process user replace the
trailing chown -R, which duplicated the 600 MB model layer.
- .dockerignore keeps __pycache__, .coverage, local indexes and .env out.
- EXTRAS build arg (INSTALL_DOCLING kept as an alias); the docling variant
also bakes docling's layout/table/RapidOCR models (DOCLING_ARTIFACTS_PATH)
and tesseract, and drops only the discovery documents of Google APIs the
app never builds.
- FLASK_DEBUG env removed (unused); OCI labels added.
Frontend (arc53/docsgpt-fe): 302 MB Vite dev server -> 25 MB static build
behind nginx. VITE_* variables are injected at container start into
/config.js and read through src/env.ts, so the image no longer needs a
rebuild per deployment; docker-compose.yaml keeps hot reload via the dev
target.
Publishing: every release and develop build now pushes a slim tag and a
-docling tag (docling engine + models + tesseract). docker-compose-hub.yaml
takes DOCSGPT_IMAGE_TAG / DOCSGPT_IMAGE_VARIANT; docker-compose-standalone.yaml
runs the stack from pre-built images without a checkout and is attached to
each release. setup.sh selects the -docling variant for OCR instead of
requiring a local build. A new workflow builds the image on PRs that touch
it and runs verify_offline under --network none; lint checks the exported
requirements match uv.lock.
Conflicts, and how each was taken:
- application/core/settings.py — ours. The renamed OCR_ENABLED /
OCR_ATTACHMENTS_ENABLED / OCR_MIN_CHARS_PER_PAGE accept main's
DOCLING_OCR_* spellings as AliasChoices, so nothing is dropped.
- application/Dockerfile — both. Main's install layers plus the
INSTALL_DOCLING build arg.
- application/parser/file/constants.py — both imports.
- deployment/docker-compose.yaml — both. The INSTALL_DOCLING /
INSTALL_TESSERACT build args on backend and worker, and main's
-Q docsgpt,parsing,embeddings, which query embedding needs.
- tests/conftest.py — theirs. Both sides fixed the same pytest-postgresql
9.0.0 autocommit= breakage; main's spelling is the one already on main.
- application/requirements.txt — the comments claimed different reasons
torch is in core. Main's is the true one now: it removed
sentence-transformers, so docling is torch's only remaining consumer.
Two things the merge broke without conflicting:
- onnxruntime. This branch moved it out of core into the docling extra;
main meanwhile made it the runtime local embeddings execute on
(fastembed). Git took the deletion, leaving fastembed with no pinned
runtime in a repo that pins everything. Restored to core, and no longer
pinned twice from the extra.
- The frontend copy of ATTACHMENT_PARSER_EXTENSIONS. The backend list is
derived and picked up the anydoc suffixes; the hand-kept frontend mirror
did not, so the composer would refuse files the API accepts.
tests/parser/file/test_constants.py is what caught it.
ruff, pytest (9897 passed), frontend build and docs build all pass. The
image build is unverified: no Docker daemon on this machine.
The sentence dates from when DOCLING_OCR_ENABLED=true meant docling plus
RapidOCR. Under the new defaults OCR_ENABLED=true resolves to the native
backend with OCR_ENGINE=tesseract, a CPU subprocess that GPU libraries do not
accelerate. Name what actually uses a GPU: docling's torch-backed layout and
table models on the worker, or a DeepSeek-OCR endpoint off it.
Both setup scripts start with `compose pull && compose up -d`. In the local
compose file backend and worker are build-only services, so `up -d` builds
only when no image exists yet: a rerun that switches OCR on wrote
INSTALL_TESSERACT=true to .env and then reused the image built without it,
leaving OCR_ENABLED=true with no engine. Build explicitly on the local
compose path; the hub path stays pull-only, its services have no build stage.
The OCR message named OCR_ENGINE=deepseek but not OCR_DEEPSEEK_URL, whose
default (localhost:11434) resolves to the container, not the host.
deployment/sandbox/README.md still described Docling as already present in
application/requirements.txt.
Query embedding moved to the Celery worker, but nothing that ships was
updated to consume the queue it dispatches to.
- Add `embeddings` to every worker `-Q` list (compose x3, k8s, devcontainer,
sandbox README). Without it a search blocked for EMBEDDINGS_DELEGATE_TIMEOUT
and then answered with no retrieved context, because classic_rag swallows the
dispatch error and skips the source -- bad answers, not an error.
- Skip the task_postrun heap reclaim for the embed task. The full gc.collect()
was written for docling/torch parses; on a worker holding the ONNX model it
measured ~86ms against ~8ms for the embed itself, a 9x slowdown of the round
trip for a task that allocates a few kilobytes.
- Resolve the installation pin in the re-embed script. It never imports
application.app, so an install pinned in app_metadata with no EMBEDDINGS_NAME
set -- every stock k8s deployment, whose manifests carry no embedding config
-- would rewrite its whole index with the legacy default and stamp
sources.model to match, then be told by the boot warning to run it again.
- Fail fast for 30s after a failed dispatch. fanout.embed_questions falls back
to letting each store embed its own query, so one dead-worker retrieval paid
the timeout once in the fan-out and again per source.
- Forget the task result. Nothing reads it back: the key is per-dispatch UUID,
not content-addressed, so a repeated query mints another. Left alone every
search leaked ~17KB for result_expires (7 days) into the Redis the broker
shares -- on the bundled k8s manifest (1Gi, no maxmemory policy) that is an
OOMKill that takes the broker with it.
- Release the model ensure_vector_schema loads to read the width of an
unregistered model, in a process that delegates and would never call it.
The width still comes from the model, not the table, so the mismatch check
the hook exists for keeps working.
- Correct the docs that said otherwise: embeddings.md claimed the standard
deployment worked unchanged, upgrading.mdx said no action was needed, and
the settings table listed none of the three delegation settings.
Native fs events do not propagate from Windows hosts into Linux
containers, causing Vite's HMR to stop detecting file changes.
When the DOCKER environment variable is set, Chokidar now falls back
to polling with a 300ms interval, which reliably detects changes
across the Docker volume mount boundary.
Fixes#2456
Replace the sandbox Docling extractor with read_document, backed by the in-process
backend parser (the same one ingestion uses) and offloaded to a dedicated
'parsing' Celery queue so it can run on GPU-capable workers with predictable RAM.
The tool resolves the input ref under the run-scoped gate, enqueues the parse,
and awaits it with a timeout (degrading to an error rather than hanging); the
worker independently re-resolves the artifact through the same gate and never
trusts a raw path. Untrusted files get the upload path's safeguards (extension
whitelist, size cap, sanitized temp file, cleanup). Options: output
(markdown/text/structured/chunks), ocr, pages, engine, max_chars, include_tables,
persist, json_schema. The workflow native-file 'extract' fallback now uses the
same worker path, so document parsing no longer needs the sandbox and works on
every backend.
Also fixes the branch's periodic-task test (the sandbox reaper made it 12) and
points the dev and e2e Celery workers at the parsing queue.
Run each kernel under a scrubbed environment so untrusted code can never read
the host's secrets. A custom 'docsgpt-python' kernelspec launches ipykernel
through a wrapper that keeps only what the kernel needs (PATH, HOME, LANG, and
the Jupyter runtime/data dirs), dropping API keys, tokens, the database URL, and
the gateway token. The app selects this kernel by name via SANDBOX_KERNEL_NAME,
so the distinct name is never shadowed by the stock python3 spec. Per-session
workspaces are created mode 0700 (defense in depth under the shared uid). The
README documents the runner as a single trust domain and points to the Daytona
backend for per-tenant isolation.
Expose a user's artifacts through the MCP server as readable resources: each is
listed under an artifact:// URI with its mime type and read on demand as inline
text or a base64 blob, bounded by a size cap and scoped strictly to the owning
principal resolved from the request's API key, so no artifact is served across
tenants. Ship the network-level egress controls the sandbox runner needs but
cannot self-apply: a Kubernetes NetworkPolicy that allows public egress while
denying RFC1918, link-local, and cloud-metadata ranges, an optional
docker-compose egress overlay, and runner network-hardening docs.
Add a document extractor tool with an extract_document action that converts an
input artifact (PDF, docx, pptx, ...) into structured JSON by running a fixed
Docling program in the sandbox (the document and parameters travel as data, so
nothing in them can execute). Output is a compact payload bounded by an input
size cap, a head-and-tail markdown window, and per-table caps, can be validated
against a JSON schema, and is persisted as a data artifact by reference. Docling
is heavy, so it stays out of the base image and ships in an opt-in sandbox image
variant.
Introduce a pluggable CodeSandbox abstraction with a SandboxManager and a
Jupyter Kernel Gateway backend: the app is a client of a single always-on
runner that executes code in stateful in-process kernels (no child-container
spawning, no docker socket). Sessions bind to a conversation or workflow run
with an agent-selectable TTL clamped by a global cap; execution enforces a
wall-clock deadline with interrupt-on-timeout and capped output, and file
transfer is workspace-contained with size and integrity checks. Adds a
docsgpt-sandbox docker-compose service (resource-capped, read-only, internal
network) plus settings, with a real local-gateway integration test.
* fixes setup scripts
fixes to env handling in setup script plus other minor fixes
* Remove var declarations
Declarations such as `LLM_PROVIDER=$LLM_PROVIDER` override .env variables in compose
Similar issue is present in the frontend - need to choose either to switch to separate frontend env or keep as is.
* Manage apikeys in settings
1. More pydantic management of api keys.
2. Clean up of variable declarations from docker compose files, used to block .env imports. Now should be managed ether by settings.py defaults or .env
Alex