Pg 4 (#2390)

* feat: postgres tests * feat: mongo cutoff * feat: mongo cutoff * feat: adjust docs and compose files * fix: mini code mongo removals * fix: tests and k8s mongo stuff * feat: test fixes * fix: ruff * fix: vale * Potential fix for pull request finding 'CodeQL / Clear-text logging of sensitive information' Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com> * fix: mini suggestions * vale lint fix 2 * fix: codeql columns thing * fix: test mongo * fix: tests coverage * feat: better tests 4 * feat: more tests * feat: decent coverage * fix: ruff fixes * fix: remove mongo mock * feat: enhance workflow engine and API routes; add document retrieval and source handling * feat: e2e tests * fix: mcp, mongo and more * fix: mini codeql warning * fix: agent chunk view * fix: mini issues * fix: more pg fixes * feat: postgres prep on start * feat: qa tests * fix: mini improvements * fix: tests --------- Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com> Co-authored-by: Siddhant Rai <siddhant.rai.5686@gmail.com>
2026-10-14 06:17:23 +00:00 · 2026-04-18 13:13:57 +01:00
1 parent 7f6360b4ff
commit 81b6ee5daa
@@ -1,114 +1,151 @@
 ---
 title: PostgreSQL for User Data
-description: Set up PostgreSQL as the user-data store for DocsGPT and migrate from MongoDB at your own pace.
+description: PostgreSQL is the user-data store for DocsGPT. Covers auto-bootstrap, production hardening, and the one-shot migration from legacy MongoDB deployments.
 ---

 import { Callout } from 'nextra/components'

 # PostgreSQL for User Data

-DocsGPT is progressively moving user data (conversations, agents, prompts,
-preferences, etc.) from MongoDB to PostgreSQL, one collection at a time.
-Each collection is guarded by a feature flag so you can opt in and roll
-back instantly. MongoDB stays the source of truth until you cut over
-reads; vector stores (`VECTOR_STORE=pgvector`, `faiss`, `qdrant`, `mongodb`, …)
-are unaffected.
+DocsGPT stores conversations, agents, prompts, sources, attachments,
+workflows, logs, and token usage in **PostgreSQL**. MongoDB is no longer
+required.

 <Callout type="info" emoji="ℹ️">
-  Which collections are available today is in the [Status](#status)
-  table below. That table is the only part of this page that changes
-  release to release.
+  Vector stores are independent — `VECTOR_STORE` can still be `pgvector`,
+  `faiss`, `qdrant`, `milvus`, `elasticsearch`, or `mongodb`.
 </Callout>

-## Setup
+## Quickstart

-1. **Run Postgres 13+.** Native install, Docker, or managed (Neon, RDS,
-   Supabase, Cloud SQL…) — all work. You'll need the `pgcrypto` and
-   `citext` extensions, both standard contrib modules available
-   everywhere.
+Three common paths. Each assumes Postgres 13+ and the default env vars
+`AUTO_MIGRATE=true` / `AUTO_CREATE_DB=true` (both ship enabled).

-2. **Create a database and role** (skip if your managed provider gave
-   you these):
+### Docker Compose

-   ```sql
-   CREATE ROLE docsgpt LOGIN PASSWORD 'docsgpt';
-   CREATE DATABASE docsgpt OWNER docsgpt;
-   ```
+The bundled compose file ships a `postgres` service. App boot handles the
+rest — no sidecar, no init job.

-3. **Set `POSTGRES_URI` in `.env`.** Any standard Postgres URI works —
-   DocsGPT normalizes it internally.
+```bash
+cd deployment && docker compose up
+```

-   ```bash
-   POSTGRES_URI=postgresql://docsgpt:docsgpt@localhost:5432/docsgpt
-   # Append ?sslmode=require for managed providers that enforce SSL.
-   ```
+### Managed Postgres (Neon, RDS, Supabase, Cloud SQL)

-4. **Apply the schema** (idempotent — safe to re-run):
+Point `POSTGRES_URI` at the provider-given URI. The app applies the
+schema on first boot.

-   ```bash
-   python scripts/db/init_postgres.py
-   ```
+```bash
+export POSTGRES_URI="postgresql://user:pass@host/docsgpt?sslmode=require"
+flask --app application/app.py run --host=0.0.0.0 --port=7091
+```

-## Migrating data
+### Bare-metal Postgres

-Two global flags, no per-collection knobs — every collection marked ✅
-in the [Status](#status) table is handled automatically.
+Run Postgres locally and point `POSTGRES_URI` at the default superuser.
+First boot creates both the database and the schema.

-1. **Enable dual-write.** Writes go to both Mongo and Postgres; Mongo
-   remains source of truth. Set the flag in `.env` and restart:
+```bash
+export POSTGRES_URI="postgresql://postgres@localhost/docsgpt"
+flask --app application/app.py run --host=0.0.0.0 --port=7091
+```

-   ```bash
-   USE_POSTGRES=true
-   ```
+Prefer a dedicated non-superuser role? Create it once as superuser — the
+app never creates roles.

-2. **Backfill existing data.** Idempotent — re-run any time to re-sync
-   drifted rows. Without arguments, backfills every registered table;
-   pass `--tables` to limit.
+```sql
+CREATE ROLE docsgpt LOGIN PASSWORD 'docsgpt' CREATEDB;
+-- Then: POSTGRES_URI=postgresql://docsgpt:docsgpt@localhost/docsgpt
+```

-   ```bash
-   python scripts/db/backfill.py --dry-run           # preview everything
-   python scripts/db/backfill.py                      # real run, everything
-   python scripts/db/backfill.py --tables users      # only specific tables
-   ```
+## How auto-bootstrap works

-3. **Cut over reads** once you trust the Postgres state:
+Two env vars control startup behavior. Both default to `true` in the
+app and are idempotent.

-   ```bash
-   READ_POSTGRES=true
-   ```
+| Setting | Effect | Requires |
+| --- | --- | --- |
+| `AUTO_CREATE_DB` | If the target database is missing, connects to the server's `postgres` maintenance DB and issues `CREATE DATABASE`. | `CREATEDB` privilege (or superuser) |
+| `AUTO_MIGRATE` | Runs `alembic upgrade head` against the target database. | Table-owner or superuser on the target DB |

-   Rollback is instant: unset `READ_POSTGRES` and restart. Dual-write
-   keeps Postgres up to date so you can flip back and forth.
+Concurrent workers serialize through `alembic_version`, so rolling
+restarts are safe. If the role lacks the required privilege, startup
+fails fast with a clear error rather than silently skipping.
+
+<Callout type="info" emoji="ℹ️">
+  Convenient in dev. In production, disable both and run migrations as
+  an explicit step — see [Production hardening](#production-hardening).
+</Callout>
+
+## Production hardening
+
+Set both flags to `false` in prod and run migrations as a gated,
+auditable step before rolling out the app.
+
+```env
+AUTO_MIGRATE=false
+AUTO_CREATE_DB=false
+```
+
+Run migrations from your CI/CD pipeline, a Kubernetes `Job`, or an
+init-container ahead of the app rollout:
+
+```bash
+python scripts/db/init_postgres.py
+# equivalently:
+alembic -c application/alembic.ini upgrade head
+```
+
+The reasoning: the app's runtime role shouldn't carry DDL privileges,
+migrations should gate each rollout, and an explicit step is
+auditable — implicit first-boot bootstrap is fine for dev but muddies
+prod deploys.

 <Callout type="warning" emoji="⚠️">
-  Don't decommission MongoDB until every collection you use is fully
-  cut over. During the migration window, Mongo is still required.
+  Migrations are not reversible by the app. Always back up production
+  Postgres before running `alembic upgrade head` on a new release.
 </Callout>

-## Status
+## Migrating from MongoDB

-_Last updated: 2026-04-10_
+One-shot, offline, app stopped. The app itself will create the
+Postgres schema when it boots — you only need to run the data copy.

-| Collection | Status |
-|---|---|
-| `users` | ✅ Phase 1 |
-| `prompts`, `user_tools`, `feedback`, `stack_logs`, `user_logs`, `token_usage` | ⏳ Phase 1 |
-| `agents`, `sources`, `attachments`, `memories`, `todos`, `notes`, `connector_sessions`, `agent_folders` | ⏳ Phase 2 |
-| `conversations`, `pending_tool_state`, `workflows` | ⏳ Phase 3 |
+```bash
+pip install -r application/requirements.txt
+pip install 'pymongo>=4.6'

-Schemas for **every** row above already exist after `init_postgres.py`
-runs. What's landing progressively is the application-level dual-write
-wiring and the backfill logic for each collection. Once a collection
-is ✅, enabling `USE_POSTGRES=true` and running `python scripts/db/backfill.py`
-picks it up automatically — no per-collection config change.
+export POSTGRES_URI="postgresql://docsgpt:docsgpt@localhost:5432/docsgpt"
+export MONGO_URI="mongodb://user:pass@host:27017/docsgpt"
+
+python scripts/db/backfill.py --dry-run    # preview
+python scripts/db/backfill.py              # real run
+# or: python scripts/db/backfill.py --tables users,agents
+```
+
+Then unset `MONGO_URI` and start the backend — nothing consults Mongo
+in the default path anymore. The backfill is idempotent (per-table
+`ON CONFLICT` upserts, event-log tables deduped via `mongo_id`), so
+re-running is safe and re-syncs any drifted rows. Keep Mongo online
+until you've verified Postgres is complete; decommission afterwards
+unless you still use it as a vector store.
+
+<Callout type="warning" emoji="⚠️">
+  No dual-write window and no runtime flag — on the current version,
+  Postgres is the only user-data store the backend reads or writes.
+</Callout>

 ## Troubleshooting

- **`relation "..." does not exist`** — run `python scripts/db/init_postgres.py`.
- **`FATAL: role "docsgpt" does not exist`** — run the `CREATE ROLE` /
-  `CREATE DATABASE` statements from step 2 as a Postgres superuser.
+- **`relation "..." does not exist`** — schema not applied. Either let
+  the app bootstrap it (`AUTO_MIGRATE=true`) or run
+  `python scripts/db/init_postgres.py`.
+- **`permission denied to create database`** — the role lacks
+  `CREATEDB`. As superuser: `ALTER ROLE <name> CREATEDB;`. Or create
+  the database manually and set `AUTO_CREATE_DB=false`.
+- **`role "docsgpt" does not exist`** — roles are never auto-created.
+  As superuser: `CREATE ROLE docsgpt LOGIN PASSWORD '...';`.
 - **SSL errors on a managed provider** — append `?sslmode=require` to
  `POSTGRES_URI`.
- **Dual-write warnings in the logs** — expected to be non-fatal. Mongo
-  is source of truth, so the user-facing request succeeds. Re-run the
-  backfill to re-sync whichever rows drifted.
+- **`ModuleNotFoundError: pymongo`** — `pip install 'pymongo>=4.6'`
+  (only needed for the one-shot Mongo backfill).