Next 16.3.5 and React 19.3 in docs/, plus the transitive refresh that came
with them (mermaid 11.17, katex 0.16.47, shiki 3.23, styled-components 6.5).
Two overrides were needed:
- zod ~4.3.6. nextra 4.6.1 declares `zod: ^4.1.12` but its MDX prop
validation breaks on zod >= 4.4.0, failing prerender for every page with
"Invalid input: expected nonoptional, received undefined -> at children".
Bisected: 4.3.6 builds, 4.4.0 through 4.6.2 all fail. nextra 4.6.1 is the
latest release, so there is no upstream fix to take yet. Drop this pin
once nextra ships one.
- @xmldom/xmldom ^0.9.12. speech-rule-engine 4.1.4 (via mathjax-full ->
better-react-mathjax -> nextra) pins xmldom at exactly 0.9.10, so
`npm audit fix` cannot reach it. 0.9.12 clears 13 advisories including
GHSA-6gmq-8vp8-gcm6 (high).
Docs build passes, 48 pages indexed, 0 advisories.
`uv lock --upgrade` after the major bumps; every direct dependency is now at
its latest release.
What stays behind is capped upstream, not by this project. The notable one:
huggingface-hub sits at 1.16.1 rather than 1.31.0 because 1.31 wants
click>=8.4.2 and gTTS 2.5.4 — the newest gTTS — caps click below 8.2.
marshmallow is held at 3.x by dataclasses-json, websockets by google-genai,
typer and semchunk by docling-core, portalocker by qdrant-client, and
pydantic-core is pinned exactly by pydantic.
Verified beyond the suite: `docsgpt verify-offline` passes (tiktoken
encoding, both tokenizers, both embedding models, docling PDF conversion),
and the wheel builds and installs into a clean environment resolved from the
declared ranges rather than the lock, with the ASGI app importing there.
Lifts the deliberate `transformers<5.9` cap. The cap existed because 5.9
broke docling's PDF layout model on Apple Silicon; 5.17 does not. Checked by
converting a four-document benchmark (two papers, a 30-page table corpus, a
10-page report) on this machine with both stacks: the Markdown is
byte-identical on three, and the fourth differs only by one dropped
`<!-- image -->` placeholder on a figure. Tables still render as GFM tables
and no text is lost.
With that cap gone tokenizers can move too — transformers 5.8.1 pinned it at
<=0.23.0 and no such release exists, which is what held it at 0.22.2.
torchvision follows torch.
Linux still resolves torch and torchvision from the CPU-only PyTorch index,
so the docling Docker variant does not pick up the CUDA stack.
No code change needed: the server side (`FastMCP`, `@mcp.tool`,
`get_http_headers`, `http_app`) and the client side (`Client`, `BearerAuth`,
the three transports, and the `mcp.client.auth` / `mcp.shared.auth` OAuth
types imported directly) all kept their signatures.
Verified beyond the suite by driving the real `/mcp` mount over a live
uvicorn socket with the fastmcp client: tool discovery, a `search_docs` call
and bearer-token extraction all round-trip, with the token reaching
`search()` intact.
fastmcp 4 and mcp 2.2 also moved to httpx2, so the retry tuple's comment now
places them on that side of the split.
Both SDKs' 1.x/3.x majors run on httpx2 (the maintained fork of httpx by its
original author, published by Pydantic at github.com/pydantic/httpx2, version
line 2.x) instead of httpx, which is what makes these two bumps one change.
httpx2 is now a declared dependency because two modules import it directly.
Everything else in the 0.x -> 1.x / 2.x -> 3.x change lists is absent here:
no Text Completions, no `with_raw_response`, no raw `output_format` dicts, no
Bedrock client, and `requires-python` is already 3.12.
Three code changes, all forced by the bump:
The BYOM DNS-pinning client in `docsgpt/security/safe_url.py` is handed to
`OpenAI(http_client=...)`, and the SDK rejects an old-httpx client at
construction — which would have taken the SSRF guard offline. It is built on
httpx2 now, and its `sni_hostname` extension carries a `str` rather than
ascii bytes: httpcore passes the value straight to
`ssl.SSLContext.wrap_socket`, and the truststore backend httpx2 uses for the
default system trust store encodes it instead of accepting bytes. Bytes
therefore failed every real handshake while passing the existing tests, which
stub the transport out; the test now pins the type and says why.
The stream-retry error tuple in `docsgpt/llm/base.py` named `httpx`
exceptions only. The two libraries' exception classes are unrelated types, so
after the bump the retry silently stopped firing for openai and anthropic
while still working for google-genai and elevenlabs. It now covers both
stacks, with a parametrized test over each.
anthropic 1.x dropped temperature/top_p/top_k from `messages.create`'s
signature (passing one raises TypeError) without dropping them from the API,
so the provider forwards them through `extra_body`. The wire request is
unchanged and a model that rejects them 400s exactly as before.
Major bumps whose ceilings had to move. redis 8.1.0, tiktoken 0.14.0 and
daytona 0.211.2 needed no code change (the Daytona client, filesystem and
process signatures the sandbox calls are unchanged; redis 8 was checked
against a live server through the app's own sync and async clients).
reportlab 5.0.1 is test-only.
openapi-parser 2.0.0 is a rewrite onto pydantic spec models: `paths` is now
a dict keyed by URL rather than a list of objects carrying their own `url`,
and a path item exposes one field per HTTP method instead of an `operations`
list. `OpenAPI3Parser` reads both accordingly, iterating methods in the
order the spec declares them, and its rendered output is byte-identical to
before. The rewrite also drops prance, openapi-spec-validator and five more
transitive packages.
tokenizers stays at 0.22.2: transformers 5.8.1 caps it at <=0.23.0 and no
such release exists, so it moves with the transformers cap or not at all.
`uv lock --upgrade` plus the two code changes the new versions need.
firecrawl-anydoc 0.2.4 raises a dedicated `NeedsOcrError` where 0.2.3 raised
`UnsupportedError("... OCR is required")`, so the anydoc parser no longer
recognised a scanned PDF: the fallback still ran, but a near-empty result was
stored as an empty document instead of failing with the OCR_ENABLED hint.
`_needs_ocr` now accepts both spellings and looks the class up lazily, so an
older anydoc keeps working. 0.2.4 also refuses the CID-font NDA fixture
outright rather than dropping its Chinese column silently, so the PDF
trust-check tests stub that dropped output against the fixture's real bytes
(the check's own inputs) and a new test pins the refusal path.
ruff 0.16 widened its implicit default rule set, turning the dev-group bump
into 7131 findings across the tree. `.ruff.toml` now states the historical
selection (E4, E7, E9, F) explicitly and the CI pin moves to the locked
0.16.7, so lint no longer drifts with the version.
The published 0.7.0 bundle carries the stranded 'use client' directive
that breaks the webpack build, so the docs deploy has been failing since
the widget landed on main. 0.7.1 ships the fix.
The lockfile pinned 0.7.0 exactly and Vercel installs from the lockfile,
so publishing the fix was not enough on its own. Raising the range to
^0.7.1 keeps a fresh install off the broken build.
The widget now ships with the docs agent's embed token, so the site works
without any environment configuration. NEXT_PUBLIC_DOCSGPT_API_KEY still
overrides it for staging.
The key is public either way. NEXT_PUBLIC_* is inlined at build time, so
it lands in the client bundle whichever way it is supplied, and every
documented DocsGPT embed carries its key in the page. Abuse is bounded by
the agent's daily request and token limits, enforced by check_usage on
the answer and stream routes.
Adds a floating "Ask the docs" bubble to every docs page. docsgpt-react
was already a dependency but nothing imported it, so no widget rendered.
The widget reads window and navigator on mount, so it is lazy-loaded
with ssr: false from a client wrapper. Its light/dark theme follows the
site toggle via next-themes, promoted here from a transitive dependency
to a declared one.
The key comes from NEXT_PUBLIC_DOCSGPT_API_KEY and the host from
NEXT_PUBLIC_DOCSGPT_API_HOST. With no key set the wrapper renders
nothing rather than falling through to the demo key the package defaults
to.
Bumping docsgpt-react to ^0.7.0 also drops a duplicate React. The old
^0.6.3 pin depended on React 18 and npm installed a nested 18.3.1 beside
the app's 19.2.8, which would have thrown an invalid hook call on first
render.
Parcel's library build concatenates modules, so the directive at the top
of DocsGPTWidget.tsx landed a thousand lines into dist/module.js and
dist/main.js. webpack rejects a 'use client' that is not the first
expression in the file, which broke every Next.js consumer of
docsgpt-react:
x The "use client" directive must be placed before other expressions.
1168 | 'use client';
The directive could not do its job from inside a bundle anyway. A React
Server Components boundary has to be declared by the importing module,
so consumers wrap the widget in their own client component instead.
The chat selection was inferred by diffing the refreshed source list
against the snapshot taken before the upload, so any source that appeared
meanwhile -- another upload finishing, a new team share -- could be picked
instead. The upload response already carries the id; match on it, and
select nothing when it is absent rather than guessing.
Drops the now-unused sourceDocs read and its stale dependency.
An agent YAML sitting in the repo root is personal config -- chat ids,
webhook targets -- and nothing tracked matches the pattern, so keep it
out of accidental `git add -A`.
selectedSourceIdsFromAgent preferred the extra sources over the legacy
single source instead of merging them. Since every save writes both
fields, opening an agent that held both and saving -- even just a rename
-- silently detached the primary source. Retrieval reads both, so merge
them.
The edit form also reported unsaved changes the moment it loaded: the
snapshot taken after the fetch defaulted the retriever to "classic" and
left the default model empty, while the effects that run straight after
rewrite both. Normalise the snapshot through the same serializer and
rules the effects apply.
A source attached to a team-shared agent that the caller cannot list --
the owner's private source -- rendered no picker row, so it showed as
attached with nothing to switch it off. Append rows for selected ids
missing from the list, and memoise the label lookup the trigger and the
picker now share.
Uploading from the agent builder repointed the open conversation onto the
new source and persisted it, because the upload modal always dispatched
the chat selection. Gate that behind a prop the builder opts out of. The
wiki source is created synchronously and never refreshed the source list,
so the builder selected an id the picker could not render and the trigger
read "External KB"; refresh on every completion path that skips ingest
tracking.
Removing the synthetic "Default" source made an empty source list
reachable, which the stored-selection reconciler skipped, leaving a
deleted source checked forever with no way to clear it. Treat an empty
list as a real answer and only skip when the list has not loaded.
Finally, finish the picker extraction: the chat input still hand-rolled
the item mapping, so team users saw the same sources grouped in the agent
builder and flat in chat. Point it at the shared helper, with its own
locale keys, and make the sources link a router navigation instead of a
full page reload.
PinAgent looked agents up with no ownership, share or team predicate, and
PinnedAgents returned those rows through a hand-rolled dict that bypassed
the blanking _format_agent_output applies. Anyone holding an agent id --
a share-link recipient learns the real UUID -- could pin it and keep
reading its name, description, prompt, tools, type, status and masked key
after the share was revoked.
Add _user_may_pin: the owner, a team grantee, a recipient of a still-live
share link, or a system template. PinAgent checks it before pinning and
PinnedAgents re-checks every row on read, so a revoked grant drops out of
the list instead of persisting. Unpinning stays open regardless of current
access, or a revoked share would strand a pin the user cannot clear. The
masked key is now owner-only, matching _format_agent_output.
Adds the first cross-user pin tests: every existing one pinned the
caller's own agent.
Review follow-ups:
- The agents list and the pinned list dropped agents that had neither a
source nor a retriever. Publishing such an agent is now allowed, so it
vanished from the list right after publish. Both filters are gone; a
source-less agent skips retrieval and is still runnable. Tests cover
both listings.
- The agent form selected an uploaded source by diffing the source
catalog before and after the upload, which would also pick up any
source created or shared in the meantime. Upload now passes the id of
the source it created to onSuccessfulUpload, and the form selects only
that id.
The sources list used to start with a fake "Default" entry that had no id
and, at run time, meant "no source, skip retrieval". The agent form
pre-selected it, snapped back to it when the last source was deselected,
and refused to publish without it, so a new agent always looked like it
had a knowledge base when it had none.
Backend
- /api/sources returns only ingested sources; no placeholder row.
- Publishing an agent no longer requires a source on create or update.
The legacy "default" value is still accepted and maps to NULL.
Frontend
- The agent source picker starts empty, can be cleared, and shows a hint
that a source-less agent answers from the model and its tools only.
- The picker groups sources into "Your sources" and "Shared with team"
when any team-shared source exists, shows "N sources selected" for a
multi-selection, and gets the same "Go to Sources" / "Upload new"
footer as the chat picker. A source uploaded from the form is selected
when it lands.
- Source selection serialisation and the picker id live in one helper
shared with the chat picker; the four copies in the form are gone.
- The client no longer seeds a placeholder source in the store, and the
dead auto-select of a "default" document is removed.
The script copied .env.development over frontend/.env.production.local and
deleted it on exit, destroying a local override. An existing file is now
set aside before the build and put back by the exit trap, whatever the
build's outcome; without one, nothing is left behind.
pip install docsgpt now brings the web UI with it: `docsgpt api` serves the
API and the UI on one port.
- scripts/build_frontend.sh builds the frontend into docsgpt/static
(gitignored) the way the frontend image does: .env.development as the
production baseline, and index.html loading /config.js ahead of the
bundle. hatch admits the directory into the wheel and the sdist through
`artifacts`; the package workflows run the script before `uv build` and
fail if the wheel lacks the UI. The backend image keeps ignoring it.
- docsgpt/ui.py serves the build in front of Flask: files as they are,
hashed assets immutable, Flask's own path prefixes (taken from its URL map,
so new blueprints need no registration) passed through, every other GET
rendered as index.html for the client-side router. /config.js is generated
per request with VITE_API_HOST and VITE_BASE_URL set to the page's origin,
VITE_* environment variables winning. SERVE_UI=false leaves the API alone.
- docsgpt api configures gunicorn in code (gunicorn.app.base.Application)
instead of rewriting sys.argv, so the SIGUSR2 re-exec that gunicorn uses
for zero-downtime upgrades runs the docsgpt console script again and
works; verified with a live handover.
- Docs: the pip page says the UI is included, that DOCSGPT_HOME and
DOCSGPT_ENV_FILE are process environment variables rather than .env
entries, and the settings page describes SERVE_UI.
A caller passing a target other than pypi or testpypi would skip both
publish jobs and upload nothing; the build job now fails with the value
instead. Empty (the release event) stays valid.
Run 34260211236 on the pull request called pypi-publish.yml through
workflow_call aimed at TestPyPI: the trusted-publishing exchange succeeded,
attestations were generated, and both archives were uploaded (already
present, so skipped). Same-repository reusable workflows pass PyPI's
job_workflow_ref check, which names the called file.
- backend-release checks the tag and the release separately and publishes
only when it created the release in this run. A run that pushed the tag and
then failed to create the release used to skip both on re-run.
- pypi-publish accepts a `target` from callers (default pypi) so the
reusable-workflow path can be rehearsed against TestPyPI; the TestPyPI job
skips files that already exist there.
- A temporary workflow, to be removed before merge, calls pypi-publish the
way backend-release does, aimed at TestPyPI, on this pull request: PyPI
verifies the job_workflow_ref claim, which names the called file, so the
run proves the trusted-publishing path through workflow_call.
- zizmor-action v0.6.3: the pinned v0.5.2 bundles zizmor 1.23, which cannot
parse the `uses: $/...` self-repository syntax and aborted the audit.
backend-release creates the GitHub release with GITHUB_TOKEN, and GitHub
never starts workflows from events that token produces, so the
`release: published` triggers on the Docker and PyPI publish workflows only
fired for releases made by hand. 0.18.0 got no Docker images for that
reason, and 0.19.0 reached PyPI by a manual run.
backend-release now calls both publish workflows after creating the release,
passing the version it tagged. Both workflows gain a `workflow_call` trigger
with a `version` input and read the tag from it or from the release event,
so a release created by hand still publishes through the release trigger.
The Docker Hub credentials are passed by name; the PyPI job authenticates
through trusted publishing, which matches the called workflow's filename and
environment, so the publisher configuration is unchanged.
Permissions in backend-release move from the workflow to the jobs: the
release job writes contents; the Docker call writes contents (the compose
file attached to the release) and packages; the PyPI call gets an OIDC token.
- The image pins DOCSGPT_HOME=/app: it ships no checkout, so the data home
no longer depends on the working directory.
- api, worker, beat and migrate print the data home and env file they
resolved, so an API and a worker started from different directories show
it.
- The worker passes -Q only when asked; a bare worker consumes every
configured queue, which honours EMBEDDINGS_QUEUE and DOCUMENT_PARSE_QUEUE.
- The worker runs through celery.start and returns its exit code; click
usage errors print usage and exit 2 instead of a traceback.
- Windows: solo pool and no embedded scheduler (celery rejects -B there),
with a pointer to the new `docsgpt beat` command, which runs the
scheduler on its own.
- prefetch_models and verify_offline parse their arguments, so --help is
help rather than a model name.
- A DOCSGPT_ENV_FILE that is not a file raises instead of booting with
defaults.
- pipx: replacing the CUDA torch inside the pipx environment needs
pip's --force-reinstall (pipx inject, with or without --force, leaves a
satisfied requirement alone), and --no-deps so torch's dependencies are not
reinstalled from the PyTorch index, which carries older copies of them.
- Upgrading: the embedded Milvus and LanceDB default paths now resolve under
the checkout (or DOCSGPT_HOME) instead of the start directory; the note
names the old location and the settings to pin.
- docsgpt api binds 127.0.0.1 by default, like gunicorn and uvicorn do;
--host 0.0.0.0 exposes it. The docs say so.
- The embedded Milvus and LanceDB defaults derive from the data home, so
they follow DOCSGPT_HOME like the faiss indexes and uploads do. A checkout
run from its root and the Docker image resolve to the same paths as before.
- The docs and the pyproject comment describe the CPU torch install as two
steps (torch and torchvision from the PyTorch CPU index first, then the
docling extra): pip picks the highest version across indexes, so
--extra-index-url only yields the CPU build while that index keeps pace
with PyPI.
- AGENTS.md separates DOCSGPT_HOME (moves the data home) from
DOCSGPT_ENV_FILE (selects the .env file); the docs example uses a password
placeholder.
pip install docsgpt (extras: docling, milvus) installs the backend with a
docsgpt command: api, worker, migrate, prefetch-models, verify-offline,
reembed. Second step of the PyPI work after the package rename.
- hatchling build; the version comes from docsgpt/version.py. The wheel is
the docsgpt package with the data it reads at runtime (prompts, model
catalogs, seed config, alembic.ini and migrations) and without the
Dockerfile, the exported requirements, the sample index and local runtime
data. The application import alias stays checkout-only. uv sync installs
the package editable now that [tool.uv] package = false is gone.
- docsgpt/cli.py: api (gunicorn + BoundedDrainUvicornWorker with the image's
flags, --reload for uvicorn), worker (Celery worker with beat embedded,
--no-beat/-Q/--concurrency/--pool, solo pool on macOS), migrate, and
argument pass-through to the maintenance scripts. --help imports no app.
- docsgpt/core/paths.py: runtime data lives in a data home (DOCSGPT_HOME,
else the checkout, else cwd); DOCSGPT_ENV_FILE overrides the env file.
Settings, the dotenv load, LocalStorage and the internal upload route use
it instead of "three directories above this file", which is site-packages
for an installed package. A checkout and the Docker image behave as before.
- [project] dependencies are compatible ranges so the package installs next
to other packages; uv.lock resolves to the same versions and the exported
requirements files are unchanged.
- package-build.yml builds and checks the wheel on PRs and installs it into
a clean venv; pypi-publish.yml publishes on a published release through
trusted publishing (environment pypi), or to TestPyPI on a manual run.
- Docs: Deploying -> Install with pip. AGENTS.md notes the package.
- The post-task reclaim skip recognises the legacy application.* embed name,
so query embeds queued by the previous release do not pay a full collect.
- The Azure compose file mounts host data on /app/{indexes,inputs,vectors},
where the process actually reads and writes; it mounted /app/application/...
before the rename and /app/docsgpt/... after it, and nothing wrote to either.
- The offline image check triggers on docsgpt/requirements*.txt again;
dependabot's pip entry points at docsgpt/.
- install_hint() and its docstring name docsgpt/requirements-<extra>.txt;
the test asserts the full path.
- application/vectors/ stays ignored: the compose files still mount it.
- The durability QA script quiets the docsgpt logger tree.
- Upgrade note: the three renamed source-sync entries start their timers
from the upgrade.
- CI installs the backend requirements from docsgpt/; the old cd into
application/ silently installed nothing.
- The root .dockerignore re-admits only application/__init__.py. An upgraded
checkout may still hold gitignored application/{inputs,indexes,vectors,.env}
from the old layout, and the directory rule shipped them into the image.
- The compose files keep the host bind mounts on application/{indexes,inputs,
vectors}, so an upgrade does not start with empty data. The move comes with
the packaging work, together with an upgrade note.
- The alias loader puts the real docsgpt spec back on the shared module object
after import (the import machinery stamped the alias spec on it, which made
importlib.reload rename the module and skip re-execution) and delegates
get_code/get_source/get_filename to the target loader, so
python -m application.<name> runs.
- Each legacy application.* task name is registered as its own task object,
a subclass carrying the old name. Registering the same object under two
keys made Celery's tracer log every run under whichever name it built last.
- The redbeat key prefix stays redbeat:docsgpt:; the three schedule_syncs
entries get stable names instead. redbeat tracks its static entries and
deletes the ones that vanish from beat_schedule at start-up, and rewrites the
task path of named entries in place, so neither a prefix bump nor a cleanup
pass is needed (checked against redbeat 2.4.2 with a seeded Redis).
The backend import package is now docsgpt, the name it will carry on PyPI;
application was far too generic to install into anyone's site-packages.
git mv plus a mechanical rewrite of every import, dotted string and path
reference: 734 Python files, the compose files, Dockerfile, workflows, docs,
setup scripts, devcontainer, k8s manifests, vscode config, pytest and coverage
config, .gitignore. Behaviour is unchanged.
Kept for one release:
- A top-level application package whose meta-path finder resolves
application.x.y to the already-imported docsgpt.x.y object, so old imports
and entry points (celery -A application.app.celery,
uvicorn application.asgi:asgi_app) keep working with a FutureWarning.
- Celery registers every application.* task name as an alias of its
docsgpt.* task on start-up, so messages queued by the previous release still
run. The redbeat key prefix moves to redbeat:docsgpt:v2: so schedule entries
the previous release wrote are left unread instead of firing twice.
The backend image builds from the repository root (docker build -f
docsgpt/Dockerfile .) so it can ship the alias package; a root .dockerignore
allow-lists docsgpt/ and application/ and keeps caches, local data, .env
files, the sample index files and the Dockerfile out. Compose and the image
workflows point at the new context.
Alex