The widget now ships with the docs agent's embed token, so the site works
without any environment configuration. NEXT_PUBLIC_DOCSGPT_API_KEY still
overrides it for staging.
The key is public either way. NEXT_PUBLIC_* is inlined at build time, so
it lands in the client bundle whichever way it is supplied, and every
documented DocsGPT embed carries its key in the page. Abuse is bounded by
the agent's daily request and token limits, enforced by check_usage on
the answer and stream routes.
Adds a floating "Ask the docs" bubble to every docs page. docsgpt-react
was already a dependency but nothing imported it, so no widget rendered.
The widget reads window and navigator on mount, so it is lazy-loaded
with ssr: false from a client wrapper. Its light/dark theme follows the
site toggle via next-themes, promoted here from a transitive dependency
to a declared one.
The key comes from NEXT_PUBLIC_DOCSGPT_API_KEY and the host from
NEXT_PUBLIC_DOCSGPT_API_HOST. With no key set the wrapper renders
nothing rather than falling through to the demo key the package defaults
to.
Bumping docsgpt-react to ^0.7.0 also drops a duplicate React. The old
^0.6.3 pin depended on React 18 and npm installed a nested 18.3.1 beside
the app's 19.2.8, which would have thrown an invalid hook call on first
render.
Parcel's library build concatenates modules, so the directive at the top
of DocsGPTWidget.tsx landed a thousand lines into dist/module.js and
dist/main.js. webpack rejects a 'use client' that is not the first
expression in the file, which broke every Next.js consumer of
docsgpt-react:
x The "use client" directive must be placed before other expressions.
1168 | 'use client';
The directive could not do its job from inside a bundle anyway. A React
Server Components boundary has to be declared by the importing module,
so consumers wrap the widget in their own client component instead.
The chat selection was inferred by diffing the refreshed source list
against the snapshot taken before the upload, so any source that appeared
meanwhile -- another upload finishing, a new team share -- could be picked
instead. The upload response already carries the id; match on it, and
select nothing when it is absent rather than guessing.
Drops the now-unused sourceDocs read and its stale dependency.
An agent YAML sitting in the repo root is personal config -- chat ids,
webhook targets -- and nothing tracked matches the pattern, so keep it
out of accidental `git add -A`.
selectedSourceIdsFromAgent preferred the extra sources over the legacy
single source instead of merging them. Since every save writes both
fields, opening an agent that held both and saving -- even just a rename
-- silently detached the primary source. Retrieval reads both, so merge
them.
The edit form also reported unsaved changes the moment it loaded: the
snapshot taken after the fetch defaulted the retriever to "classic" and
left the default model empty, while the effects that run straight after
rewrite both. Normalise the snapshot through the same serializer and
rules the effects apply.
A source attached to a team-shared agent that the caller cannot list --
the owner's private source -- rendered no picker row, so it showed as
attached with nothing to switch it off. Append rows for selected ids
missing from the list, and memoise the label lookup the trigger and the
picker now share.
Uploading from the agent builder repointed the open conversation onto the
new source and persisted it, because the upload modal always dispatched
the chat selection. Gate that behind a prop the builder opts out of. The
wiki source is created synchronously and never refreshed the source list,
so the builder selected an id the picker could not render and the trigger
read "External KB"; refresh on every completion path that skips ingest
tracking.
Removing the synthetic "Default" source made an empty source list
reachable, which the stored-selection reconciler skipped, leaving a
deleted source checked forever with no way to clear it. Treat an empty
list as a real answer and only skip when the list has not loaded.
Finally, finish the picker extraction: the chat input still hand-rolled
the item mapping, so team users saw the same sources grouped in the agent
builder and flat in chat. Point it at the shared helper, with its own
locale keys, and make the sources link a router navigation instead of a
full page reload.
PinAgent looked agents up with no ownership, share or team predicate, and
PinnedAgents returned those rows through a hand-rolled dict that bypassed
the blanking _format_agent_output applies. Anyone holding an agent id --
a share-link recipient learns the real UUID -- could pin it and keep
reading its name, description, prompt, tools, type, status and masked key
after the share was revoked.
Add _user_may_pin: the owner, a team grantee, a recipient of a still-live
share link, or a system template. PinAgent checks it before pinning and
PinnedAgents re-checks every row on read, so a revoked grant drops out of
the list instead of persisting. Unpinning stays open regardless of current
access, or a revoked share would strand a pin the user cannot clear. The
masked key is now owner-only, matching _format_agent_output.
Adds the first cross-user pin tests: every existing one pinned the
caller's own agent.
Review follow-ups:
- The agents list and the pinned list dropped agents that had neither a
source nor a retriever. Publishing such an agent is now allowed, so it
vanished from the list right after publish. Both filters are gone; a
source-less agent skips retrieval and is still runnable. Tests cover
both listings.
- The agent form selected an uploaded source by diffing the source
catalog before and after the upload, which would also pick up any
source created or shared in the meantime. Upload now passes the id of
the source it created to onSuccessfulUpload, and the form selects only
that id.
The sources list used to start with a fake "Default" entry that had no id
and, at run time, meant "no source, skip retrieval". The agent form
pre-selected it, snapped back to it when the last source was deselected,
and refused to publish without it, so a new agent always looked like it
had a knowledge base when it had none.
Backend
- /api/sources returns only ingested sources; no placeholder row.
- Publishing an agent no longer requires a source on create or update.
The legacy "default" value is still accepted and maps to NULL.
Frontend
- The agent source picker starts empty, can be cleared, and shows a hint
that a source-less agent answers from the model and its tools only.
- The picker groups sources into "Your sources" and "Shared with team"
when any team-shared source exists, shows "N sources selected" for a
multi-selection, and gets the same "Go to Sources" / "Upload new"
footer as the chat picker. A source uploaded from the form is selected
when it lands.
- Source selection serialisation and the picker id live in one helper
shared with the chat picker; the four copies in the form are gone.
- The client no longer seeds a placeholder source in the store, and the
dead auto-select of a "default" document is removed.
The script copied .env.development over frontend/.env.production.local and
deleted it on exit, destroying a local override. An existing file is now
set aside before the build and put back by the exit trap, whatever the
build's outcome; without one, nothing is left behind.
pip install docsgpt now brings the web UI with it: `docsgpt api` serves the
API and the UI on one port.
- scripts/build_frontend.sh builds the frontend into docsgpt/static
(gitignored) the way the frontend image does: .env.development as the
production baseline, and index.html loading /config.js ahead of the
bundle. hatch admits the directory into the wheel and the sdist through
`artifacts`; the package workflows run the script before `uv build` and
fail if the wheel lacks the UI. The backend image keeps ignoring it.
- docsgpt/ui.py serves the build in front of Flask: files as they are,
hashed assets immutable, Flask's own path prefixes (taken from its URL map,
so new blueprints need no registration) passed through, every other GET
rendered as index.html for the client-side router. /config.js is generated
per request with VITE_API_HOST and VITE_BASE_URL set to the page's origin,
VITE_* environment variables winning. SERVE_UI=false leaves the API alone.
- docsgpt api configures gunicorn in code (gunicorn.app.base.Application)
instead of rewriting sys.argv, so the SIGUSR2 re-exec that gunicorn uses
for zero-downtime upgrades runs the docsgpt console script again and
works; verified with a live handover.
- Docs: the pip page says the UI is included, that DOCSGPT_HOME and
DOCSGPT_ENV_FILE are process environment variables rather than .env
entries, and the settings page describes SERVE_UI.
A caller passing a target other than pypi or testpypi would skip both
publish jobs and upload nothing; the build job now fails with the value
instead. Empty (the release event) stays valid.
Run 34260211236 on the pull request called pypi-publish.yml through
workflow_call aimed at TestPyPI: the trusted-publishing exchange succeeded,
attestations were generated, and both archives were uploaded (already
present, so skipped). Same-repository reusable workflows pass PyPI's
job_workflow_ref check, which names the called file.
- backend-release checks the tag and the release separately and publishes
only when it created the release in this run. A run that pushed the tag and
then failed to create the release used to skip both on re-run.
- pypi-publish accepts a `target` from callers (default pypi) so the
reusable-workflow path can be rehearsed against TestPyPI; the TestPyPI job
skips files that already exist there.
- A temporary workflow, to be removed before merge, calls pypi-publish the
way backend-release does, aimed at TestPyPI, on this pull request: PyPI
verifies the job_workflow_ref claim, which names the called file, so the
run proves the trusted-publishing path through workflow_call.
- zizmor-action v0.6.3: the pinned v0.5.2 bundles zizmor 1.23, which cannot
parse the `uses: $/...` self-repository syntax and aborted the audit.
backend-release creates the GitHub release with GITHUB_TOKEN, and GitHub
never starts workflows from events that token produces, so the
`release: published` triggers on the Docker and PyPI publish workflows only
fired for releases made by hand. 0.18.0 got no Docker images for that
reason, and 0.19.0 reached PyPI by a manual run.
backend-release now calls both publish workflows after creating the release,
passing the version it tagged. Both workflows gain a `workflow_call` trigger
with a `version` input and read the tag from it or from the release event,
so a release created by hand still publishes through the release trigger.
The Docker Hub credentials are passed by name; the PyPI job authenticates
through trusted publishing, which matches the called workflow's filename and
environment, so the publisher configuration is unchanged.
Permissions in backend-release move from the workflow to the jobs: the
release job writes contents; the Docker call writes contents (the compose
file attached to the release) and packages; the PyPI call gets an OIDC token.
- The image pins DOCSGPT_HOME=/app: it ships no checkout, so the data home
no longer depends on the working directory.
- api, worker, beat and migrate print the data home and env file they
resolved, so an API and a worker started from different directories show
it.
- The worker passes -Q only when asked; a bare worker consumes every
configured queue, which honours EMBEDDINGS_QUEUE and DOCUMENT_PARSE_QUEUE.
- The worker runs through celery.start and returns its exit code; click
usage errors print usage and exit 2 instead of a traceback.
- Windows: solo pool and no embedded scheduler (celery rejects -B there),
with a pointer to the new `docsgpt beat` command, which runs the
scheduler on its own.
- prefetch_models and verify_offline parse their arguments, so --help is
help rather than a model name.
- A DOCSGPT_ENV_FILE that is not a file raises instead of booting with
defaults.
- pipx: replacing the CUDA torch inside the pipx environment needs
pip's --force-reinstall (pipx inject, with or without --force, leaves a
satisfied requirement alone), and --no-deps so torch's dependencies are not
reinstalled from the PyTorch index, which carries older copies of them.
- Upgrading: the embedded Milvus and LanceDB default paths now resolve under
the checkout (or DOCSGPT_HOME) instead of the start directory; the note
names the old location and the settings to pin.
- docsgpt api binds 127.0.0.1 by default, like gunicorn and uvicorn do;
--host 0.0.0.0 exposes it. The docs say so.
- The embedded Milvus and LanceDB defaults derive from the data home, so
they follow DOCSGPT_HOME like the faiss indexes and uploads do. A checkout
run from its root and the Docker image resolve to the same paths as before.
- The docs and the pyproject comment describe the CPU torch install as two
steps (torch and torchvision from the PyTorch CPU index first, then the
docling extra): pip picks the highest version across indexes, so
--extra-index-url only yields the CPU build while that index keeps pace
with PyPI.
- AGENTS.md separates DOCSGPT_HOME (moves the data home) from
DOCSGPT_ENV_FILE (selects the .env file); the docs example uses a password
placeholder.
pip install docsgpt (extras: docling, milvus) installs the backend with a
docsgpt command: api, worker, migrate, prefetch-models, verify-offline,
reembed. Second step of the PyPI work after the package rename.
- hatchling build; the version comes from docsgpt/version.py. The wheel is
the docsgpt package with the data it reads at runtime (prompts, model
catalogs, seed config, alembic.ini and migrations) and without the
Dockerfile, the exported requirements, the sample index and local runtime
data. The application import alias stays checkout-only. uv sync installs
the package editable now that [tool.uv] package = false is gone.
- docsgpt/cli.py: api (gunicorn + BoundedDrainUvicornWorker with the image's
flags, --reload for uvicorn), worker (Celery worker with beat embedded,
--no-beat/-Q/--concurrency/--pool, solo pool on macOS), migrate, and
argument pass-through to the maintenance scripts. --help imports no app.
- docsgpt/core/paths.py: runtime data lives in a data home (DOCSGPT_HOME,
else the checkout, else cwd); DOCSGPT_ENV_FILE overrides the env file.
Settings, the dotenv load, LocalStorage and the internal upload route use
it instead of "three directories above this file", which is site-packages
for an installed package. A checkout and the Docker image behave as before.
- [project] dependencies are compatible ranges so the package installs next
to other packages; uv.lock resolves to the same versions and the exported
requirements files are unchanged.
- package-build.yml builds and checks the wheel on PRs and installs it into
a clean venv; pypi-publish.yml publishes on a published release through
trusted publishing (environment pypi), or to TestPyPI on a manual run.
- Docs: Deploying -> Install with pip. AGENTS.md notes the package.
- The post-task reclaim skip recognises the legacy application.* embed name,
so query embeds queued by the previous release do not pay a full collect.
- The Azure compose file mounts host data on /app/{indexes,inputs,vectors},
where the process actually reads and writes; it mounted /app/application/...
before the rename and /app/docsgpt/... after it, and nothing wrote to either.
- The offline image check triggers on docsgpt/requirements*.txt again;
dependabot's pip entry points at docsgpt/.
- install_hint() and its docstring name docsgpt/requirements-<extra>.txt;
the test asserts the full path.
- application/vectors/ stays ignored: the compose files still mount it.
- The durability QA script quiets the docsgpt logger tree.
- Upgrade note: the three renamed source-sync entries start their timers
from the upgrade.
- CI installs the backend requirements from docsgpt/; the old cd into
application/ silently installed nothing.
- The root .dockerignore re-admits only application/__init__.py. An upgraded
checkout may still hold gitignored application/{inputs,indexes,vectors,.env}
from the old layout, and the directory rule shipped them into the image.
- The compose files keep the host bind mounts on application/{indexes,inputs,
vectors}, so an upgrade does not start with empty data. The move comes with
the packaging work, together with an upgrade note.
- The alias loader puts the real docsgpt spec back on the shared module object
after import (the import machinery stamped the alias spec on it, which made
importlib.reload rename the module and skip re-execution) and delegates
get_code/get_source/get_filename to the target loader, so
python -m application.<name> runs.
- Each legacy application.* task name is registered as its own task object,
a subclass carrying the old name. Registering the same object under two
keys made Celery's tracer log every run under whichever name it built last.
- The redbeat key prefix stays redbeat:docsgpt:; the three schedule_syncs
entries get stable names instead. redbeat tracks its static entries and
deletes the ones that vanish from beat_schedule at start-up, and rewrites the
task path of named entries in place, so neither a prefix bump nor a cleanup
pass is needed (checked against redbeat 2.4.2 with a seeded Redis).
The backend import package is now docsgpt, the name it will carry on PyPI;
application was far too generic to install into anyone's site-packages.
git mv plus a mechanical rewrite of every import, dotted string and path
reference: 734 Python files, the compose files, Dockerfile, workflows, docs,
setup scripts, devcontainer, k8s manifests, vscode config, pytest and coverage
config, .gitignore. Behaviour is unchanged.
Kept for one release:
- A top-level application package whose meta-path finder resolves
application.x.y to the already-imported docsgpt.x.y object, so old imports
and entry points (celery -A application.app.celery,
uvicorn application.asgi:asgi_app) keep working with a FutureWarning.
- Celery registers every application.* task name as an alias of its
docsgpt.* task on start-up, so messages queued by the previous release still
run. The redbeat key prefix moves to redbeat:docsgpt:v2: so schedule entries
the previous release wrote are left unread instead of firing twice.
The backend image builds from the repository root (docker build -f
docsgpt/Dockerfile .) so it can ship the alias package; a root .dockerignore
allow-lists docsgpt/ and application/ and keeps caches, local data, .env
files, the sample index files and the Dockerfile out. Compose and the image
workflows point at the new context.
The build and manifest jobs of both image workflows declare
environment: docker-hub, so the registry credentials can be scoped to it and
protection rules applied in the repository settings (the environment is
created on first use). The image namespace is the public arc53 the compose
files pull from, not the login secret, which now only authenticates.
zizmor flags a secret expanded inside with: or run:. The image workflows
put DOCKER_USERNAME in a job-level env and use that in tags, labels, cache
refs and the manifest loop; login-action keeps its username input.
The Vite dev server (docker-compose.yaml's dev target) lets an empty process
variable override .env.development, so listing every VITE_* as ${VAR:-}
wiped the banner and Google client id defaults. A bare name in the
environment list reaches the container only when the variable is set in the
shell or the --env-file; unset ones are omitted. Applied to all four compose
files.
- The frontend image ran the Vite dev server in development mode, so
.env.development supplied its defaults (notification banner, Google client
id, local API host). The static build only loads .env.production, so the
build stage now copies .env.development in as the baseline and the compose
files pass every VITE_* the app reads through from .env; the runtime script
skips empty values so a blank passthrough keeps the build-time default.
.dockerignore kept only the .local variants out.
- VITE_DISABLE_SOURCE_FE disables sources only when it is the string true.
- DoclingParser: find_spec raises when docling itself is absent; the install
hint now covers that path, with a regression test.
- verify_offline: direct tests for verify(); the PR image check builds and
verifies the -docling variant as well as slim.
- Workflows this branch adds or rewrites pin actions by commit, pass the
release tag through env instead of template expansion, and do not persist
checkout credentials.
- OCR guide no longer claims pre-built images never include docling.
settings.EMBEDDINGS_NAME still defaults to mpnet so an upgraded deployment
keeps its index; new installs only get granite because setup.sh writes it to
.env. The standalone compose always starts on fresh volumes, so it defaults
EMBEDDINGS_NAME to granite (overridable from .env or the shell), and the
Docker guide's hand-written .env examples name the model too.
OMP_NUM_THREADS=4 capped torch in the old image, but FastEmbed runs on
onnxruntime, which ignores it and sizes its pool to the host's core count.
A CPU-limited container still reports every host core, so embedding ran with
64 threads on a 4-CPU sandbox and took up to 12x longer per file. The image
now sets EMBEDDINGS_THREADS=4 next to the existing caps; the settings guide
documents the knob.
The worker hands finished indexes to the API through an internal endpoint
that rejects every request while INTERNAL_KEY is unset, so a .env written by
hand from the docs' minimal example made every upload end in a 401. setup.sh
generates the key; the standalone compose header and the Docker guide now
generate one too.
A named volume mounted on /app/inputs, /app/indexes or /app/vectors inherits
the ownership of the image directory, so the image creates them as appuser;
before this the volume came up root-owned and every upload failed with a
permission error unless the container ran as root. docker-compose-standalone.yaml
still runs backend and worker as root, like docker-compose-hub.yaml, so it
also works with image tags that predate these directories.
requirements-docling.txt adds the PyTorch CPU index, which uv resolves only
with UV_INDEX_STRATEGY=unsafe-best-match (pip is unaffected); the file header
and the docs say so and point uv users at uv sync --extra docling.
Backend (arc53/docsgpt): 4.5 GB compressed -> 0.9 GB with both embedding
models and tiktoken baked in.
- torch/transformers gone from the default install (docling extra only).
- Ubuntu 24.04 ships python3.12: no deadsnakes PPA, no software-properties-
common; every pin is a wheel, so no gcc/g++/rust in the builder.
- COPY --chown and a prefetch that runs as the process user replace the
trailing chown -R, which duplicated the 600 MB model layer.
- .dockerignore keeps __pycache__, .coverage, local indexes and .env out.
- EXTRAS build arg (INSTALL_DOCLING kept as an alias); the docling variant
also bakes docling's layout/table/RapidOCR models (DOCLING_ARTIFACTS_PATH)
and tesseract, and drops only the discovery documents of Google APIs the
app never builds.
- FLASK_DEBUG env removed (unused); OCI labels added.
Frontend (arc53/docsgpt-fe): 302 MB Vite dev server -> 25 MB static build
behind nginx. VITE_* variables are injected at container start into
/config.js and read through src/env.ts, so the image no longer needs a
rebuild per deployment; docker-compose.yaml keeps hot reload via the dev
target.
Publishing: every release and develop build now pushes a slim tag and a
-docling tag (docling engine + models + tesseract). docker-compose-hub.yaml
takes DOCSGPT_IMAGE_TAG / DOCSGPT_IMAGE_VARIANT; docker-compose-standalone.yaml
runs the stack from pre-built images without a checkout and is attached to
each release. setup.sh selects the -docling variant for OCR instead of
requiring a local build. A new workflow builds the image on PRs that touch
it and runs verify_offline under --network none; lint checks the exported
requirements match uv.lock.
Three things still reached the network from a container whose models were
baked in:
- tiktoken fetched cl100k_base from openaipublic.blob.core.windows.net on
every fresh container (its cache defaulted to /tmp), and token accounting
calls it on every chat. prefetch_models now warms it too; the image sets
TIKTOKEN_CACHE_DIR.
- The chunker loaded its tokenizer with Tokenizer.from_pretrained, which
revalidates the revision with a HEAD request per process start and stalls
for the etag timeout (10 s) when huggingface.co is unreachable. It now reads
tokenizer.json from the hub cache first and only downloads on a miss; the
repo-metadata read for models outside the registry does the same.
- tldextract fetched the public suffix list on the first web crawl; the
bundled snapshot is used instead.
application/scripts/verify_offline.py exercises these paths (and docling's
conversion when the extra is installed) so an image can be checked with
docker run --network none.
requirements.txt pinned torch and transformers in core although only docling
needs them, and on Linux torch pulls the CUDA 13 stack: 2.7 GB of the 3.0 GB
wheel download. Direct dependencies now live in pyproject.toml, uv.lock pins
everything, and application/requirements*.txt are exported from the lock by
scripts/export_requirements.sh (each file is the core set plus one extra).
The docling extra pins torch/torchvision/transformers itself and, on Linux,
resolves torch from the CPU-only PyTorch index (no nvidia packages). milvus
(pymilvus + milvus-lite, which pulls pyarrow) is the second extra.
application/core/optional_deps.py is the one place install hints come from;
the milvus store and the docling call sites use it so a missing extra fails
with the exact command to run.
An unchained request with no system message staged None, and the record
step skipped the commit, so the previous head hash survived a transcript
that never received a head; a later chained request restoring that head
would have omitted it. The staged value is now committed as-is, None
included. Also pins each rejection predicate of is_usable_compression_point
with its own test.
Alex