94 Commits
Author SHA1 Message Date
arc53-machine 4b08e94be7 Name every person on agent pages by one rule
people_named_to decides whom the edit page and the share dialog name:
the reader, the holder's owner, anyone sharing a team with the reader and,
for the owner, whoever sponsored something on the holder. A resource's
owner, named as whom to ask or whose credentials a tool uses, also needs
the reader to see that resource. Sponsors, runs_as, contact and account
in resource_states and sponsors in sponsor_details all follow it; anyone
else is left unnamed and the pages say someone else.

A tool's run details (note, credential_mode, account, writes) come from
one function and only for a tool that runs. The share dialog keys each
person's own account on the per_user_account note, as the notice does,
and the unused noteKey helper and its string are gone.
2026-09-29 18:12:54 +01:00
arc53-machine d04f76e97a Say which resources are left out of runs and which can't run until fixed 2026-09-29 17:52:12 +01:00
arc53-machine db9eb2b1c3 Correct what the agent sharing docs say about accounts, sponsors and switches
The table adds tools with saved credentials, says API and widget users
get the owner's account on a tool each person connects, that a sponsored
item runs as the owner again once the owner can use it, and that a
teammate's name shows only to people who share a team with them. The
editor switches are named per resource type, public-link users are said
to be asked only for writes that need approval, the allowlist is said to
cover sponsored and workflow node tools, and the badge names and the new
resource_states fields match the app.
2026-09-29 17:50:42 +01:00
arc53-machine da75845fcf Document sharing agents and whose access what they use runs with
A new "Sharing agents and what they use" section covers viewers and
editors, whose access each tool, source and prompt runs with as the share
dialog labels it, sponsors, stopped resources, what API, widget and
public-link users can't do without the write allowlist, the wiki switch
and research steps. The sharing rules now mention the editor switches and
member-mode tools, the guardrails page no longer says editors can't
change guardrails, the connector guide uses the new tool share labels,
and related pages link to the section.
2026-09-29 17:35:17 +01:00
arc53-machine 9775dd1af9 Document what happens when an agent's resource stops working 2026-09-29 17:17:13 +01:00
arc53-machine 90b385e9ea Document taking over stopped resources and how the agent's audience can grow 2026-09-29 16:55:00 +01:00
arc53-machine ff10fcca0f Say the wiki switch covers API keys and widgets, not public links
Public-link visitors edit only wikis they can edit themselves and approve
each edit. The docs also note that with authentication off every caller
is the owner's local user, so the switch has no effect there.
2026-09-29 16:37:58 +01:00
arc53-machine c66afd6b30 Document who may sponsor a resource in someone else's agent 2026-09-29 16:34:53 +01:00
arc53-machine d4b30f4838 Document who can edit a wiki from the API, widget and public links 2026-09-29 16:12:42 +01:00
arc53-machine 7b516ade82 Add a built-in GitHub connector for repository sync and read-only tools
One GitHub connection feeds both a Knowledge source and an agent tool.
Users connect with a personal access token, checked against GitHub and
named after the account, or, when an admin registers a GitHub App
(GITHUB_CLIENT_ID, GITHUB_CLIENT_SECRET, GITHUB_APP_SLUG), with Sign in
with GitHub. App tokens expire after eight hours and are refreshed before
each sync or tool call; tokens with no expiry are never treated as expired.

The catalog gains an optional second sign-in method (oauth_settings,
exposed as sign_in_methods) without changing any other connector.

Sync lists the repositories the connection can read (the token's own, or
the App installations') and ingests one with the connection's token. The
tool is GitHub's read-only MCP server (api.githubcopilot.com/mcp/readonly):
setup discovers its actions and creates it bound to the connection, and
the executor sends the connection's token only to that server.
2026-09-29 10:41:03 +01:00
arc53-machine 667d4bbab0 Encrypt connection credentials with an owner-bound AES-GCM envelope
Adds a v2 credential envelope next to the v1 tool-secret helpers:
AES-256-GCM, a master key derived once per process from
ENCRYPTION_SECRET_KEY, and a per-record key from HKDF over the owner's
id, which is also the associated data, so a blob moved onto another
user's row does not decrypt. The envelope names its key, so
ENCRYPTION_SECRET_KEY_PREVIOUS keeps old rows readable during a
rotation.

Log redaction now also covers token_info, tokens and client_info, and
the API warns at startup when the public default key is in use.
2026-09-28 17:03:02 +01:00
arc53-machine 4e1002f5e0 Document execution traces and the GenAI OpenTelemetry export
Covers what is recorded, where it is shown, the TRACES_* settings, the
gen_ai.* spans and metrics, content capture, and the limits of exporting
spans when the request ends.
2026-09-23 17:46:18 +01:00
arc53-machine f6269c483f Add execution-trace settings and declare opentelemetry-api
TRACES_* settings for the per-request trace timeline and its GenAI OTel
export. opentelemetry-api was only transitive; the tracing package
imports it directly.
2026-09-23 17:14:51 +01:00
arc53-machine f693111de1 fix(audit): derive attribution for legacy inserts instead of defaulting it
The DEFAULT 'unknown' added in the last pass stopped a previous-release
process from raising NotNullViolation mid-rollout, but it threw the
attribution away to do it. The highest-volume auth_events writers are OIDC
login and silent renewal, where the actor is simply the user, so a rollout
window would have flattened exactly the rows that were trivially recoverable.

Lifts both derivation rules into SQL functions -- auth_events_derive_actor
and auth_events_derive_target -- so the backfill and a BEFORE INSERT trigger
share one definition rather than two copies of the same CASE drifting apart.

The trigger guards on NEW.actor_id IS NULL, which identifies a legacy insert
exactly because the repository always supplies one. That distinction matters
for target_id: a current writer sets it NULL deliberately for events with no
user target, and deriving there would name the actor as their own target.

Kept rather than scheduled for removal: it is a no-op on the path the
repository takes, and it keeps the column's contract true for any writer that
bypasses it.
2026-09-22 13:22:11 +01:00
arc53-machine 25c82003d7 fix(admin): second review pass
Correctness
- /api/remote never recorded source.created, so URL, GitHub and connector
  sources had a source.deleted with no matching creation. All three creation
  paths now go through one _audit_source_created helper.
- The prompt-cache rate divided cached tokens by a whole bucket's prompt
  tokens. A bucket is a day and mixes calls whose provider reports a cache
  breakdown with calls whose provider does not, so filtering buckets in the
  client could not separate them and the rate was understated by however much
  traffic ran on a non-reporting provider. The denominator is now computed in
  SQL over the reporting rows.
- The outcome pill matched values nothing writes. Guardrails emit triggered /
  not_evaluated and the device feed emits dispatched; the map had blocked /
  denied / allowed, so a guardrail that fired rendered neutral grey -- the one
  signal the merged feed exists to surface. Fixtures were seeding the
  fictional values, so the tests passed on it too.
- Stream duration_ms timed the consumer. stream_token_usage is a generator,
  so start-to-exhaustion includes the agent loop's tool handling and the SSE
  client's pace; a slow browser recorded ~30s for a sub-second call. It now
  accumulates only the time spent inside next().

Safety
- Activity filters failed open: an unknown facet or unparseable timestamp was
  dropped, and no filter means every row, so a typo widened an audit view and
  on the export streamed the full history. Both are now a 400.
- The search term was interpolated into an ILIKE pattern, so "100%" matched
  everything and "q1_report" matched more than it should. Escaped.
- 0034 set actor_id NOT NULL with no default. A previous-release process
  inserting mid-rollout would raise, and in admin/routes.py that insert shares
  the request transaction, so a role grant beside it would roll back too.

Noise and dead code
- The per-user panel is a security panel: data-plane events file under the
  actor, so an active account's routine deletes pushed a denied login out of
  the 20-row window. It now excludes them; the Activity tab shows everything.
- device_audit_log had no created_at-leading index, so the merged feed
  sequentially scanned that branch every page (migration 0036).
- conversation.deleted_all no longer records when nothing was deleted, and
  agent.updated no longer records an empty field list.
- Dropped by_model from /admin/usage (no consumer; an extra aggregate per page
  load), the duplicate filter surface on AuthEventsRepository that nothing
  called, and the unreachable FLOW_LABELS.schedule entry.
- Type hints on record_event's conn and the remaining unannotated helpers.
2026-09-22 12:47:40 +01:00
arc53-machine f6f8af8cef docs: document the activity feed, actor/target and usage spend
Covers what the previous commits changed: the actor_id / target_id split and
the system actors, the data-plane events, the merged Activity feed and its
export, and the new usage and per-user spend endpoints.

Also corrects "there is no UI for these events yet" in the OIDC login
auditing section, which is no longer true.
2026-09-22 10:35:17 +01:00
Alex 1e14605ee7 fix(quotas): keyless agent chat bucket, keep disabled policies disabled, cached rates
- check_usage treats a request through a keyless (draft) agent as agent
  traffic, matching how its usage rows are bucketed and the headless rule
- dashboard edits carry the stored enabled flag instead of re-enabling the
  policy; disabled policies are labelled in the Quotas tab and the editor
- quota 429s send x-should-retry: false so OpenAI SDK clients do not retry
  a refusal that cannot succeed before the reset
- cached-input and cache-write rates for Anthropic, OpenRouter and Groq
  gpt-oss-120b; refresh OpenRouter deepseek-v3.2 list prices
- UsageQuota reuses usagePercent; docs note that a user override needs an
  existing user
2026-09-21 15:26:29 +01:00
Alex 4bd259fc09 fix(quotas): address review: resume claims, agent bucket rule, UI races
- A tool continuation refused for usage now releases the resume claim it
  took; before, retries got a 409 until the stale claim was reverted.
- Agent traffic is any row with an agent key or an agent id, so keyless
  agents and workflow nodes count toward the agent bucket, not direct.
- The user quota modal discards responses for a previously opened user.
- The usage meter shows every limited bucket, not only 'all'.
- Restore the class separator on the analytics stat card that a formatter
  run removed, and align the OpenRouter DeepSeek description with its rates.
2026-09-21 12:44:26 +01:00
Alex 1eacfdd3d0 docs: usage quotas
How the instance default, team allowances and user overrides resolve
(including users in several teams), the quota window, who is charged for
agent traffic, how cost budgets price models and what happens to unpriced
ones, and the admin and user API.
2026-09-21 12:11:38 +01:00
Alex b77561288d feat(pricing): per-million model rates and a cost module
Rename the unused *_cost_per_token capability fields to USD per 1M tokens,
add prompt-cache read/write rates, and ship list prices for the hosted
catalogs. The old per-token keys still load, scaled, with a warning.

docsgpt/pricing.py turns a call's token bins into a USD cost. Models with
no declared rate cost $0 unless QUOTA_UNPRICED_RATE_PER_MILLION is set.
2026-09-21 11:38:22 +01:00
Alex 1c1bc2538f Merge pull request #2812 from arc53-machine/feat/personal-access-tokens
feat: personal access tokens (scoped API tokens for CLI and CI/CD)
2026-09-21 10:55:33 +01:00
arc53-machine 7ddb5f6400 fix(pat): align replay scopes, keep 404/405, retire expired names, document the PAT_ENABLED switch
The ASGI message events route now accepts the same scopes as its Flask
sibling (conversations:read or chat:run) through a shared constant. A token
request that fails routing gets Flask's 404/405 instead of a 403. Creating a
token retires an expired token that still held the name. allowed_ids uses
is_pat instead of a bare literal. PAT_ENABLED is documented as the master
switch it is: turning it off stops every existing token from authenticating.
The docs explain that sources are matched by name (oldest wins) and point CI
flows at sources upload --replace.
2026-09-20 12:12:42 +01:00
arc53-machine 22ecc0aee3 feat(pat): add personal access token storage and settings
A personal_access_tokens table (migration 0032) holds scoped user-level API
credentials. Only the SHA-256 of the secret is stored, like device session
tokens. Lookups exclude revoked and expired tokens and the tokens of
deactivated users. PAT_* settings cover the feature switch, default and
maximum lifetime, the operator opt-in for non-expiring tokens and the
per-user cap.
2026-09-19 23:39:58 +01:00
Alex a83e1dc0af feat(graphrag): seed the walk from what entities are, and rank with passages and vector hits
Graph retrieval tied plain vector search at best and never beat it. Measured
across five corpora, the bottleneck was seeding, not the graph: the walk
started from nodes whose embeddings were computed from bare entity names, and
a whole question shares almost nothing with a name like "Quill".

Extraction now embeds each node from "name (type): description" and each
relationship as the fact it asserts ("Alder streams_to Quill: ..."), stored on
a new nullable graph_edges.fact_embedding column that ensure_vector_schema adds
in place. Entity names are canonicalised (case, punctuation, word breaks and a
cautious plural) so "VECTOR_STORE" and "vector stores" land on one node. Extraction calls run
concurrently (GRAPHRAG_EXTRACTION_WORKERS, default 8) while embedding and graph
writes stay serial on the task thread, so ordering and idempotency are
unchanged; that measured 8.4x faster with identical output.

Retrieval gains per-source options, stored under retrieval.graph and read live
at query time:

- seed_strategy: start from matching entities (default) or matching
  relationships, which can reach an entity the question never names;
- passage_nodes (on): walk the source's passages alongside entities, with
  PageRank damping 0.5 instead of 0.85;
- blend_vector (on): fuse the graph ranking with the source's vector ranking
  by reciprocal rank.

The defaults are the measured-best configuration. Through GraphRAGRetriever,
the new seeding moved recall@4 from 0.41 to 0.68 on a multi-hop corpus and
from 0.50 to 1.00 on the docs corpus, and regressed none of the corpora
measured. Existing graphs keep name-only embeddings until rebuilt.
2026-09-19 14:07:41 +01:00
Alex bc333f157c Merge pull request #2803 from arc53/feat/install-6-dev
feat: docsgpt dev, doctor and restart for the development loop
2026-09-17 14:48:25 +01:00
Alex c59741f604 Merge pull request #2800 from arc53/feat/install-5-native
feat: docsgpt up --native
2026-09-17 14:48:03 +01:00
Alex 9ca1eb8f34 Merge pull request #2799 from arc53/feat/install-4-backup
docsgpt backup and restore
2026-09-17 14:47:40 +01:00
arc53-machine a47a2c34ec docs(settings): render field constraints as code spans in the reference
The docs site failed to build: a bare "<= 1" in the prose of the
generated page is parsed by MDX as the start of a JSX tag ("Unexpected
character '=' before name"). Constraints are rendered as code spans now,
where MDX leaves them alone, and a test rejects any bare <, { or } outside
a code span so a future description cannot reintroduce the failure.
Verified with a local next build of the docs site.
2026-09-17 13:05:01 +01:00
arc53-machine 5578039c19 refactor(settings): treat unset spellings of every optional string as None
Review follow-up. The per-group secret validators normalised a hand-picked
list of API keys, which left other optional credentials and overrides
(OPEN_ROUTER_API_KEY, S3 and Daytona keys, ELASTIC_PASSWORD, the OIDC
trio, connector client ids, MICROSOFT_AUTHORITY, MCP_OAUTH_REDIRECT_URI)
holding the literal "None" or "" a .env file spells "unset" with, so
truthiness checks and fallbacks downstream saw a value. One rule on the
group base replaces those lists: every Optional[str] field maps "", "None"
and whitespace to None and strips real values. Plain str fields are left
alone. The OIDC required-settings check therefore also rejects those
spellings.

EMBEDDINGS_POOLING is Literal["cls", "mean"] with case-insensitive
parsing; its consumer silently ignored anything else.

Bounds added where the consumer rejects or misbehaves on the value:
SCHEDULE_RUN_OUTPUT_RETENTION_DAYS and MESSAGE_EVENTS_RETENTION_DAYS (the
cleanup repositories raise on <= 0), EMBEDDINGS_DELEGATE_TIMEOUT, the
remote-device idle/pairing/invocation TTLs and CELERY_VISIBILITY_TIMEOUT
(> 0), REMOTE_DEVICE_CMD_QUEUE_TTL_SECONDS (> 605, the documented drain
deadline), GRAPHRAG_MAX_CHUNKS_FOR_EXTRACTION (>= 0; negative would slice
the pending list from the end).

The generated reference now renders generic type arguments
(dict[str, int] rather than dict).
2026-09-17 11:37:57 +01:00
Alex a20c83f468 feat: a development loop in one command
`docsgpt up --native` installs services meant to outlive the shell. Development
wants the opposite, and until now it meant three terminals from the guide:
uvicorn, celery, and vite.

`docsgpt dev` runs this checkout's API and worker as children of one terminal,
both restarting when a file is saved, their output interleaved and labelled, and
Ctrl-C stopping them together. `--ui` adds the Vite dev server, `--mock-llm`
runs the bundled mock model so no API key is needed, and `--no-worker` leaves
the worker to your editor's debugger. Celery has no reloader of its own, so the
worker is wrapped in watchfiles when it is installed, and runs plain when it is
not.

Alongside it, the commands a dev loop keeps reaching for:

- `docsgpt doctor` checks what usually breaks a new setup: PostgreSQL answering
  and its schema matching this version, Redis answering, a model provider being
  configured, and the port being free.
- `docsgpt restart [api|worker]` bounces services without rewriting settings or
  rerunning migrations, which `down` plus `up` did.
- `docsgpt logs -f` follows a native install instead of telling you to run
  `tail -f` yourself.
- `docsgpt env set` applies itself to a running native install rather than
  asking you to run `docsgpt up` again to change one value.

Two bugs found on the way, both older than this change:

- `docsgpt api --reload` watched the working directory, which in a checkout is
  178,425 files: .venv, node_modules, and the indexes/ and inputs/ the app
  writes to while ingesting, so the server restarted itself mid-request. It
  watches the package now — 1,217 files.
- The VS Code "Flask Debugger" ran `flask run`, which serves only the WSGI app:
  /mcp, the SSE streams and artifact downloads 404 under it. The guide warned
  about this in prose while the debug config did it anyway. It runs uvicorn on
  the ASGI app now, like production.
2026-09-17 11:32:50 +01:00
arc53-machine f882ef49a7 refactor: read settings directly instead of getattr with a second default
About 85 call sites read a setting as getattr(settings, "NAME", fallback),
each carrying its own copy of the default. Every one of those names is a
field with a default on the model, so the fallback could never apply to
the real settings object; it only masked drift. Two had drifted:

- OPENAI_PROMPT_CACHE_KEY defaults to True on the model but the reader
  fell back to False, and two test stubs relied on that.
- SharePoint's MICROSOFT_AUTHORITY fallback to
  https://login.microsoftonline.com/<tenant> never fired, because the
  attribute always exists (as None), so MSAL got authority=None. The
  connector now derives the tenant authority when the setting is unset,
  as its test always assumed.

Four places read EMBEDDINGS_KEY straight from os.environ, skipping the
"None"/"" normalisation the model applies; they read the setting now.
Test stubs that replaced a module's settings with a SimpleNamespace list
every setting the code under test reads.
2026-09-17 11:14:34 +01:00
arc53-machine d70644743b refactor(settings): deprecate SAGEMAKER_* and drop two unused settings
SAGEMAKER_REGION, SAGEMAKER_ACCESS_KEY and SAGEMAKER_SECRET_KEY survive
only as a fallback for the S3_* credentials. They carry
Field(deprecated=...) now, so any read emits a DeprecationWarning naming
the replacement and the generated reference shows the notice. The S3
store is the one sanctioned reader; it silences that warning locally
because it already logs its own operator-facing one when the fallback
is actually used.

DEFAULT_MAX_HISTORY was referenced nowhere. RETRIEVERS_ENABLED was read by
no code at all, while two docs pages described it as an enforced
allow-list; both the setting and those claims are removed.
2026-09-17 11:11:00 +01:00
arc53-machine a8dab8864d refactor(settings): validate cross-field rules in the model
The "AUTH_TYPE=oidc requires OIDC_ISSUER, OIDC_CLIENT_ID and
OIDC_FRONTEND_URL" check lived in app.py, so it only ran when the Flask
app was imported; a worker or script with the same misconfiguration
started fine. It is now a model validator on the auth group and runs
wherever Settings is loaded, with the same message.

DEPLOYMENT_TYPE, which app.py read straight from the environment to
decide whether a missing JWT_SECRET_KEY is fatal, is a documented
setting on the server group now, so it shows up in the reference like
every other variable the app reads.
2026-09-17 11:09:09 +01:00
arc53-machine 95d0799494 refactor(settings): tighten types on closed choices, containers and bounds
Enum-like settings whose allowed values were only listed in a comment are
now Literal types, so a typo fails at startup with a message naming the
allowed values instead of falling through to a default with a warning
(or, for VECTOR_STORE, failing on first use):

  AUTH_TYPE, VECTOR_STORE, STORAGE_TYPE, URL_STRATEGY, OCR_BACKEND,
  OCR_ENGINE, SANDBOX_BACKEND, DOC_PARSER_ENGINE, TTS_PROVIDER, STT_PROVIDER

Each keeps a before-validator that strips and lower-cases the value, since
the registries that consume them already lower-cased at the use site, and
AUTH_TYPE maps the "None"/"none"/"" spellings a .env file carries to None
(it was the string "None" before, which only worked because nothing
compared against it). An empty TTS/STT provider still means "off".
LLM_PROVIDER stays a plain str because providers are plugin-extensible.

Containers are typed (dict[str, int], list[str], dict[str, Any]) instead
of bare dict/list, six fields that were Optional with a non-None default
are plain, and integer settings whose description already states a range
carry it as a constraint (ge=0 for "0 disables", ge=1 for counts that
cannot be zero, 0 < threshold <= 1).
2026-09-17 11:08:50 +01:00
arc53-machine 5a5226ebe0 docs(settings): generate the settings reference from the definitions
The hand-maintained settings page documented 95 of 258 settings and
.env-template 42, and both drifted as fields were added. The field
descriptions now live on the model, so the reference is rendered from it:

  python -m docsgpt.core.settings.reference --write

writes docs/content/Deploying/Settings-Reference.mdx, one section per
settings group with each field's type, default, constraints, aliases and
description. --check reports a stale page, and tests/core/test_settings.py
fails when the checked-in page no longer matches the definitions, so a
new setting cannot land undocumented.

test_settings.py also pins the composition contract: every group field is
a flat Settings attribute, no field is defined twice, every field has a
description, and the secret-normalising validator of every group is
applied (the case that a shared method name would silently drop).

The App Configuration page points at the reference instead of at
settings.py, and the reference is listed in the Deploying navigation.
2026-09-17 11:04:11 +01:00
Alex f9f52e99f0 fix: make a native install take effect on systemd and stay out of Docker's way
From review of #2800:

- systemd `enable --now` starts nothing when the unit is already active, so a
  second `up --native` kept the old ExecStart and left the API on its previous
  port. start enables and then restarts, as the launchd path already did by
  booting the job out first.
- An explicit `home` now wins over XDG_CONFIG_HOME, which is what callers pass
  it for.
- The ExecStart program must be a real executable: when `docsgpt` is not on
  PATH, sys.argv[0] is accepted only if it can be run, and otherwise the
  failure is raised before any unit is written.
- `up --native` over a directory holding a Docker install now refuses and says
  how to proceed, instead of starting native services beside containers that
  down, status and uninstall would no longer see.
- Docs: without a terminal only --postgres-uri is required, and the Windows
  fallback names `docsgpt beat`, which the worker cannot embed there.

SystemdServices was the least covered part of the module and cannot be run on
this machine, so it now has tests for install, start, stop, remove, is_running
and a failing systemctl.
2026-09-16 22:35:14 +01:00
Alex e3c5d0511c Merge feat/install-4-backup into feat/install-5-native 2026-09-16 21:56:21 +01:00
Alex 447ae72fe2 fix: harden docsgpt backup and docsgpt restore
From review of #2799:

- The archive is created 0600 rather than at the process umask: it holds the
  install's data, and --with-settings puts .env and its secrets in it.
- restore validates everything the manifest declares before the stack is
  stopped, so a damaged archive fails while DocsGPT is still running rather
  than after `compose down` has taken it away.
- Only the volumes a backup is made of are restored. A hand-made manifest can
  no longer point import_volume at postgres_data, whose contents it empties.
- psql runs with ON_ERROR_STOP=on, so a restore that fails halfway cannot
  start DocsGPT again and call it a success.
- import_volume unpacks into the container's own filesystem first and clears
  the live volume only once the tar has come out whole, so a corrupt one
  leaves the volume as it was.
- The backend and the worker stop while the archive is made and start again
  even if the dump fails, so the dump and the volume tars describe the same
  moment instead of drifting apart as ingestion writes.
2026-09-16 21:55:33 +01:00
Alex f23a32d9c5 feat: docsgpt up --native
Run DocsGPT without Docker: the API and the worker each become a service
on the machine itself, a launchd agent on macOS and a systemd user unit
on Linux, pointed at a PostgreSQL and a Redis that already run.

`docsgpt up --native --postgres-uri ... --redis-url ...` writes the same
.env a Docker install uses, applies the migrations and starts both
services. status, logs, down and uninstall work on a native install the
same way they do on a Docker one, and never touch the database or Redis:
they were the user's to begin with.

One Redis URL covers the broker, the result backend and the cache on
three consecutive databases, starting at the one the URL names, so a
Redis that already holds something else can be shared.

Windows has neither service manager, so native mode refuses it and says
what to do instead.
2026-09-16 21:51:43 +01:00
Alex fe68fec69e feat: docsgpt backup and docsgpt restore
`docsgpt backup` writes one archive holding a pg_dump of the database, a tar
of each data volume and a manifest of what it came from; `docsgpt restore`
puts it back over an install. The settings file is left out unless
--with-settings asks for it, since it holds the install's secrets, and a
backup taken with a newer DocsGPT is refused without --force.

The volume tars go through the image the install already runs, so a backup
pulls nothing extra, and compose calls can now redirect stdout and stdin so
the dump never passes through this process.
2026-09-16 21:33:33 +01:00
Alex 49823a6859 fix: installer review follow-ups
- install.sh saves the get.docker.com and uv installers to a file and runs
  them only after the download finished, so a cut-off transfer runs nothing.
- Neither installer prints DOCSGPT_PACKAGE, which may be a URL with
  credentials.
- The CI step assigns the wheel path before exporting it, so a missing wheel
  fails instead of installing from PyPI.
- Docker-Deploying shows one code block per platform; Quickstart names the
  /opt/docsgpt home used for root on Linux.
2026-09-16 01:10:11 +01:00
Alex 4f0bf2cca8 feat: one-command installers for macOS, Linux and Windows
deployment/install.sh (curl | bash) and install.ps1 (irm | iex) check for
Docker, install uv when it is missing or older than 0.8 (pinned 0.12.15 via
Astral's installer), install or upgrade the docsgpt package with
`uv tool install`, and hand the terminal to `docsgpt up` with any arguments.
On Linux without Docker the shell installer offers get.docker.com. Both run
entirely inside a function, so a download cut short runs nothing.

Releases attach both scripts next to the Compose file, which is where
docs.ac/install and docs.ac/install.ps1 will point. installer-lint.yml runs
shellcheck and the PowerShell parser; docker-image-verify.yml now installs
through install.sh. README, Quickstart, Docker-Deploying and the changelog
lead with the one-liner.
2026-09-16 01:10:11 +01:00
Alex 68d9772f2e docs: docsgpt up and the new data home; CI runs up against the built image
Docker-Deploying gains a `docsgpt up` section, Pip-Install and Upgrading
describe the ~/.docsgpt/server data home, and the changelog covers both.

docker-image-verify.yml installs the wheel and runs `docsgpt up`, `status`,
a second `up` that must keep the secrets, and `uninstall --purge` against
the image it built. The standalone Compose file maps host.docker.internal
to the host gateway, so a model server on a Linux host is reachable the way
`docsgpt up` suggests.
2026-09-15 23:00:23 +01:00
Alex 95fd4bdefe feat: serve the UI from the backend image, one-port standalone stack
The backend image builds the web UI with scripts/build_frontend.sh and
serves it through docsgpt/ui.py, so the standalone Compose file drops the
frontend container. UI and API share port 7091, published on 127.0.0.1
unless DOCSGPT_BIND says otherwise. POSTGRES_PASSWORD is configurable, and
an optional https profile puts Caddy in front of a public domain.

docker-image-verify.yml starts the standalone stack on the image it built
and checks the API, the UI, /config.js and a client-side route on one port.
2026-09-15 22:41:53 +01:00
Alex 02f2fa198a fix: disabled STT also covers live finish; clarify model cache location 2026-09-15 19:28:19 +01:00
Alex 7da46c2bea feat: air-gapped deployment guide, no implicit downloads
- Ship tiktoken's cl100k_base inside the package and build the encoding
  from it, so token counting never downloads anything.
- Default EMBEDDINGS_CACHE_DIR to <data home>/models instead of FastEmbed's
  temp dir, and read tokenizer.json and repo metadata from that cache, so
  a model downloads once and survives reboots.
- TTS_PROVIDER=none and STT_PROVIDER=none switch the speech features off:
  the endpoints return 404, audio files fail to ingest with a clear
  message, /api/config reports tts_available/stt_available, and the UI
  hides the Speak and microphone buttons.
- Drop the Google Fonts Roboto import from the web UI.
- prefetch-models fills the cache the app reads; verify-offline checks the
  packaged encoding.
- Docs: new Air-Gapped Deployment guide, settings and cache notes.
2026-09-15 17:54:24 +01:00
Alex 54b540ea42 feat(api): serve long-lived streams on the event loop
Move GET /api/events, the remote-device command stream and artifact
downloads from Flask to Starlette routes mounted ahead of the Flask
catch-all. On Flask each held an a2wsgi threadpool slot for as long as
its response stayed open, and because uvicorn drops writes after a
client disconnects, a closed tab never released it.

- asgi_auth: one JWT/OIDC gate for Starlette routes; the chat reconnect
  reader uses it too
- ClosingStreamingResponse closes the body iterator and releases the
  SSE slot or file handle even when the client leaves before the first
  frame
- AsyncTopic liveness probe replaces the sync client's socket_timeout
  guard against half-open pub/sub sockets
- ASYNC_REDIS_MAX_CONNECTIONS sizes the async Redis pool; every open
  stream holds a connection and redis-py defaults to 100
2026-09-14 08:33:46 +01:00
Alex 43f493b926 feat(package): ship the web UI in the wheel and serve it from the API
pip install docsgpt now brings the web UI with it: `docsgpt api` serves the
API and the UI on one port.

- scripts/build_frontend.sh builds the frontend into docsgpt/static
  (gitignored) the way the frontend image does: .env.development as the
  production baseline, and index.html loading /config.js ahead of the
  bundle. hatch admits the directory into the wheel and the sdist through
  `artifacts`; the package workflows run the script before `uv build` and
  fail if the wheel lacks the UI. The backend image keeps ignoring it.
- docsgpt/ui.py serves the build in front of Flask: files as they are,
  hashed assets immutable, Flask's own path prefixes (taken from its URL map,
  so new blueprints need no registration) passed through, every other GET
  rendered as index.html for the client-side router. /config.js is generated
  per request with VITE_API_HOST and VITE_BASE_URL set to the page's origin,
  VITE_* environment variables winning. SERVE_UI=false leaves the API alone.
- docsgpt api configures gunicorn in code (gunicorn.app.base.Application)
  instead of rewriting sys.argv, so the SIGUSR2 re-exec that gunicorn uses
  for zero-downtime upgrades runs the docsgpt console script again and
  works; verified with a live handover.
- Docs: the pip page says the UI is included, that DOCSGPT_HOME and
  DOCSGPT_ENV_FILE are process environment variables rather than .env
  entries, and the settings page describes SERVE_UI.
2026-09-09 11:09:09 +01:00
Alex 3036ece3ff fix(cli): review pass on the docsgpt command
- The image pins DOCSGPT_HOME=/app: it ships no checkout, so the data home
  no longer depends on the working directory.
- api, worker, beat and migrate print the data home and env file they
  resolved, so an API and a worker started from different directories show
  it.
- The worker passes -Q only when asked; a bare worker consumes every
  configured queue, which honours EMBEDDINGS_QUEUE and DOCUMENT_PARSE_QUEUE.
- The worker runs through celery.start and returns its exit code; click
  usage errors print usage and exit 2 instead of a traceback.
- Windows: solo pool and no embedded scheduler (celery rejects -B there),
  with a pointer to the new `docsgpt beat` command, which runs the
  scheduler on its own.
- prefetch_models and verify_offline parse their arguments, so --help is
  help rather than a model name.
- A DOCSGPT_ENV_FILE that is not a file raises instead of booting with
  defaults.
2026-09-07 17:53:01 +01:00
Alex f6f0d49452 docs(package): pipx CPU torch swap that works, and the local vector-store path note
- pipx: replacing the CUDA torch inside the pipx environment needs
  pip's --force-reinstall (pipx inject, with or without --force, leaves a
  satisfied requirement alone), and --no-deps so torch's dependencies are not
  reinstalled from the PyTorch index, which carries older copies of them.
- Upgrading: the embedded Milvus and LanceDB default paths now resolve under
  the checkout (or DOCSGPT_HOME) instead of the start directory; the note
  names the old location and the settings to pin.
2026-09-07 16:09:52 +01:00