tiennm99/DocsGPT
1
0 Fork 0
mirror of https://github.com/tiennm99/DocsGPT.git synced 2026-10-14 10:17:55 +00:00
Code Issues Packages Projects Releases Wiki Activity

Pg 4 (#2390)

Browse Source 
* feat: postgres tests

* feat: mongo cutoff

* feat: mongo cutoff

* feat: adjust docs and compose files

* fix: mini code mongo removals

* fix: tests and k8s mongo stuff

* feat: test fixes

* fix: ruff

* fix: vale

* Potential fix for pull request finding 'CodeQL / Clear-text logging of sensitive information'

Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>

* fix: mini suggestions

* vale lint fix 2

* fix: codeql columns thing

* fix: test mongo

* fix: tests coverage

* feat: better tests 4

* feat: more tests

* feat: decent coverage

* fix: ruff fixes

* fix: remove mongo mock

* feat: enhance workflow engine and API routes; add document retrieval and source handling

* feat: e2e tests

* fix: mcp, mongo and more

* fix: mini codeql warning

* fix: agent chunk view

* fix: mini issues

* fix: more pg fixes

* feat: postgres prep on start

* feat: qa tests

* fix: mini improvements

* fix: tests

---------

Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>
Co-authored-by: Siddhant Rai <siddhant.rai.5686@gmail.com>
This commit is contained in:
AlexCopilot Autofix powered by AISiddhant Rai
 authored and GitHub committed 2026-04-18 13:13:57 +01:00
1 parent 7f6360b4ff
commit 81b6ee5daa
312 files changed
+763725 -25159
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

docs/content/Agents/basics.mdx
+2 -2
View File
@@ -115,10 +115,10 @@ Once an agent is created, you can:
## Seeding Premade Agents from YAML
You can bootstrap a fresh DocsGPT deployment with a curated set of agents by seeding them directly into MongoDB.
You can bootstrap a fresh DocsGPT deployment with a curated set of agents by seeding them directly into the user-data store (Postgres).
1. **Customize the configuration** – edit `application/seed/config/premade_agents.yaml` (or copy from `application/seed/config/agents_template.yaml`) to describe the agents you want to provision. Each entry lets you define prompts, tools, and optional data sources.
2. **Ensure dependencies are running** – MongoDB must be reachable using the credentials in `.env`, and a Celery worker should be available if any agent sources need to be ingested via `ingest_remote`.
2. **Ensure dependencies are running** – Postgres must be reachable using `POSTGRES_URI` from `.env` (schema applied via `python scripts/db/init_postgres.py`), and a Celery worker should be available if any agent sources need to be ingested via `ingest_remote`.
3. **Execute the seeder** – run `python -m application.seed.commands init`. Add `--force` when you need to reseed an existing environment.
The seeder keeps templates under the `system` user so they appear in the UI for anyone to clone or customize. Environment variable placeholders such as `${MY_TOKEN}` inside tool configs are resolved during the seeding process.
docs/content/Deploying/Development-Environment.mdx
+15 -4
View File
@@ -3,17 +3,19 @@ title: Setting Up a Development Environment
description: Guide to setting up a development environment for DocsGPT, including backend and frontend setup.
---
import { Callout } from 'nextra/components'
# Setting Up a Development Environment
This guide will walk you through setting up a development environment for DocsGPT. This setup allows you to modify and test the application's backend and frontend components.
## 1. Spin Up MongoDB and Redis
## 1. Spin Up Postgres and Redis
For development purposes, you can quickly start MongoDB and Redis containers, which are the primary database and caching systems used by DocsGPT. We provide a dedicated Docker Compose file, `docker-compose-dev.yaml`, located in the `deployment` directory, that includes only these essential services.
For development purposes, you can quickly start Postgres and Redis containers. Postgres is the user-data store for DocsGPT (conversations, agents, prompts, sources, attachments, workflows, logs, and token usage), and Redis is used as the cache and Celery broker. We provide a dedicated Docker Compose file, `docker-compose-dev.yaml`, located in the `deployment` directory, that includes only these essential services. The backend applies the Alembic schema automatically on first boot (`AUTO_MIGRATE=true` / `AUTO_CREATE_DB=true` ship enabled), so no separate migration step is required. You can still run `python scripts/db/init_postgres.py` explicitly if you prefer.
You can find the `docker-compose-dev.yaml` file [here](https://github.com/arc53/DocsGPT/blob/main/deployment/docker-compose-dev.yaml).
**Steps to start MongoDB and Redis:**
**Steps to start Postgres and Redis:**
1. Navigate to the root directory of your DocsGPT repository in your terminal.
@@ -24,7 +26,16 @@ You can find the `docker-compose-dev.yaml` file [here](https://github.com/arc53/
docker compose -f deployment/docker-compose-dev.yaml up -d
```
These commands will start MongoDB and Redis in detached mode, running in the background.
These commands will start Postgres and Redis in detached mode, running in the background. When the Flask backend boots against the fresh Postgres instance, it will automatically create the database (if missing) and apply the current Alembic schema.
<Callout type="info" emoji="ℹ️">
MongoDB is no longer required for a default DocsGPT install. If you
specifically want to use MongoDB Atlas as the vector store
(`VECTOR_STORE=mongodb`), start it on the side via
`deployment/docker-compose.mongo.yaml`. For migrating an existing
Mongo-based install to Postgres, see
[PostgreSQL for User Data](/Deploying/Postgres-Migration).
</Callout>
## 2. Run the Backend
docs/content/Deploying/DocsGPT-Settings.mdx
+43
View File
@@ -3,6 +3,8 @@ title: DocsGPT Settings
description: Configure your DocsGPT application by understanding the basic settings.
---
import { Callout } from 'nextra/components'
# DocsGPT Settings
DocsGPT is highly configurable, allowing you to tailor it to your specific needs and preferences. You can control various aspects of the application, from choosing the Large Language Model (LLM) provider to selecting embedding models and vector stores.
@@ -239,6 +241,47 @@ SAGEMAKER_REGION=us-east-1
Your IAM user needs these permissions on the bucket: `s3:PutObject`, `s3:GetObject`, `s3:DeleteObject`, `s3:ListBucket`, `s3:HeadObject`.
## User-Data Storage (Postgres)
DocsGPT stores user data — conversations, agents, prompts, sources, attachments, workflows, logs, and token usage — in **PostgreSQL**. The backend connects via a single setting:
| Setting | Description | Default |
| --- | --- | --- |
| `POSTGRES_URI` | SQLAlchemy-compatible Postgres URI. Any standard `postgresql://` form works — DocsGPT normalizes it internally to the `psycopg` v3 dialect. | — |
| `AUTO_CREATE_DB` | On startup, connect to the server's `postgres` maintenance DB and issue `CREATE DATABASE` if the target is missing. Requires `CREATEDB` or superuser. No-op when the database already exists. Disable in production. | `true` |
| `AUTO_MIGRATE` | On startup, run `alembic upgrade head` against the target database. Idempotent and serialized across workers via `alembic_version`. Disable in production in favor of an explicit migration step. | `true` |
Example:
```env
POSTGRES_URI=postgresql://docsgpt:docsgpt@localhost:5432/docsgpt
# Append ?sslmode=require for managed providers that enforce SSL.
```
With the defaults, the app applies the schema automatically on first
boot. To run it explicitly instead (e.g., in CI/CD or a k8s `Job`):
```bash
python scripts/db/init_postgres.py
```
The default Docker Compose file bundles a `postgres` service, and the
app auto-bootstraps the database on boot, so containerized deployments
need no manual migration step. See
[PostgreSQL for User Data](/Deploying/Postgres-Migration#production-hardening)
for the recommended production flow (both flags `false`, migrations
gated by CI/CD).
<Callout type="info" emoji="ℹ️">
`MONGO_URI` is **opt-in**. It is only consulted when you select the
MongoDB Atlas vector-store backend (`VECTOR_STORE=mongodb`) or when
running the one-shot `scripts/db/backfill.py` migration from a legacy
Mongo-based install. Installing the optional Mongo client libraries
requires `pip install 'pymongo>=4.6'`. See
[PostgreSQL for User Data](/Deploying/Postgres-Migration) for the
migration path.
</Callout>
## Exploring More Settings
These are just the basic settings to get you started. The `settings.py` file contains many more advanced options that you can explore to further customize DocsGPT, such as:
docs/content/Deploying/Postgres-Migration.mdx
+111 -74
View File
@@ -1,114 +1,151 @@
---
title: PostgreSQL for User Data
description: Set up PostgreSQL as the user-data store for DocsGPT and migrate from MongoDB at your own pace.
description: PostgreSQL is the user-data store for DocsGPT. Covers auto-bootstrap, production hardening, and the one-shot migration from legacy MongoDB deployments.
---
import { Callout } from 'nextra/components'
# PostgreSQL for User Data
DocsGPT is progressively moving user data (conversations, agents, prompts,
preferences, etc.) from MongoDB to PostgreSQL, one collection at a time.
Each collection is guarded by a feature flag so you can opt in and roll
back instantly. MongoDB stays the source of truth until you cut over
reads; vector stores (`VECTOR_STORE=pgvector`, `faiss`, `qdrant`, `mongodb`, …)
are unaffected.
DocsGPT stores conversations, agents, prompts, sources, attachments,
workflows, logs, and token usage in **PostgreSQL**. MongoDB is no longer
required.
<Callout type="info" emoji="ℹ️">
Which collections are available today is in the [Status](#status)
table below. That table is the only part of this page that changes
release to release.
Vector stores are independent — `VECTOR_STORE` can still be `pgvector`,
`faiss`, `qdrant`, `milvus`, `elasticsearch`, or `mongodb`.
</Callout>
## Setup
## Quickstart
1. **Run Postgres 13+.** Native install, Docker, or managed (Neon, RDS,
Supabase, Cloud SQL…) — all work. You'll need the `pgcrypto` and
`citext` extensions, both standard contrib modules available
everywhere.
Three common paths. Each assumes Postgres 13+ and the default env vars
`AUTO_MIGRATE=true` / `AUTO_CREATE_DB=true` (both ship enabled).
2. **Create a database and role** (skip if your managed provider gave
you these):
### Docker Compose
```sql
CREATE ROLE docsgpt LOGIN PASSWORD 'docsgpt';
CREATE DATABASE docsgpt OWNER docsgpt;
```
The bundled compose file ships a `postgres` service. App boot handles the
rest — no sidecar, no init job.
3. **Set `POSTGRES_URI` in `.env`.** Any standard Postgres URI works —
DocsGPT normalizes it internally.
```bash
cd deployment && docker compose up
```
```bash
POSTGRES_URI=postgresql://docsgpt:docsgpt@localhost:5432/docsgpt
# Append ?sslmode=require for managed providers that enforce SSL.
```
### Managed Postgres (Neon, RDS, Supabase, Cloud SQL)
4. **Apply the schema** (idempotent — safe to re-run):
Point `POSTGRES_URI` at the provider-given URI. The app applies the
schema on first boot.
```bash
python scripts/db/init_postgres.py
```
```bash
export POSTGRES_URI="postgresql://user:pass@host/docsgpt?sslmode=require"
flask --app application/app.py run --host=0.0.0.0 --port=7091
```
## Migrating data
### Bare-metal Postgres
Two global flags, no per-collection knobs — every collection marked ✅
in the [Status](#status) table is handled automatically.
Run Postgres locally and point `POSTGRES_URI` at the default superuser.
First boot creates both the database and the schema.
1. **Enable dual-write.** Writes go to both Mongo and Postgres; Mongo
remains source of truth. Set the flag in `.env` and restart:
```bash
export POSTGRES_URI="postgresql://postgres@localhost/docsgpt"
flask --app application/app.py run --host=0.0.0.0 --port=7091
```
```bash
USE_POSTGRES=true
```
Prefer a dedicated non-superuser role? Create it once as superuser — the
app never creates roles.
2. **Backfill existing data.** Idempotent — re-run any time to re-sync
drifted rows. Without arguments, backfills every registered table;
pass `--tables` to limit.
```sql
CREATE ROLE docsgpt LOGIN PASSWORD 'docsgpt' CREATEDB;
-- Then: POSTGRES_URI=postgresql://docsgpt:docsgpt@localhost/docsgpt
```
```bash
python scripts/db/backfill.py --dry-run # preview everything
python scripts/db/backfill.py # real run, everything
python scripts/db/backfill.py --tables users # only specific tables
```
## How auto-bootstrap works
3. **Cut over reads** once you trust the Postgres state:
Two env vars control startup behavior. Both default to `true` in the
app and are idempotent.
```bash
READ_POSTGRES=true
```
| Setting | Effect | Requires |
| --- | --- | --- |
| `AUTO_CREATE_DB` | If the target database is missing, connects to the server's `postgres` maintenance DB and issues `CREATE DATABASE`. | `CREATEDB` privilege (or superuser) |
| `AUTO_MIGRATE` | Runs `alembic upgrade head` against the target database. | Table-owner or superuser on the target DB |
Rollback is instant: unset `READ_POSTGRES` and restart. Dual-write
keeps Postgres up to date so you can flip back and forth.
Concurrent workers serialize through `alembic_version`, so rolling
restarts are safe. If the role lacks the required privilege, startup
fails fast with a clear error rather than silently skipping.
<Callout type="info" emoji="ℹ️">
Convenient in dev. In production, disable both and run migrations as
an explicit step — see [Production hardening](#production-hardening).
</Callout>
## Production hardening
Set both flags to `false` in prod and run migrations as a gated,
auditable step before rolling out the app.
```env
AUTO_MIGRATE=false
AUTO_CREATE_DB=false
```
Run migrations from your CI/CD pipeline, a Kubernetes `Job`, or an
init-container ahead of the app rollout:
```bash
python scripts/db/init_postgres.py
# equivalently:
alembic -c application/alembic.ini upgrade head
```
The reasoning: the app's runtime role shouldn't carry DDL privileges,
migrations should gate each rollout, and an explicit step is
auditable — implicit first-boot bootstrap is fine for dev but muddies
prod deploys.
<Callout type="warning" emoji="⚠️">
Don't decommission MongoDB until every collection you use is fully
cut over. During the migration window, Mongo is still required.
Migrations are not reversible by the app. Always back up production
Postgres before running `alembic upgrade head` on a new release.
</Callout>
## Status
## Migrating from MongoDB
_Last updated: 2026-04-10_
One-shot, offline, app stopped. The app itself will create the
Postgres schema when it boots — you only need to run the data copy.
| Collection | Status |
|---|---|
| `users` | ✅ Phase 1 |
| `prompts`, `user_tools`, `feedback`, `stack_logs`, `user_logs`, `token_usage` | ⏳ Phase 1 |
| `agents`, `sources`, `attachments`, `memories`, `todos`, `notes`, `connector_sessions`, `agent_folders` | ⏳ Phase 2 |
| `conversations`, `pending_tool_state`, `workflows` | ⏳ Phase 3 |
```bash
pip install -r application/requirements.txt
pip install 'pymongo>=4.6'
Schemas for **every** row above already exist after `init_postgres.py`
runs. What's landing progressively is the application-level dual-write
wiring and the backfill logic for each collection. Once a collection
is ✅, enabling `USE_POSTGRES=true` and running `python scripts/db/backfill.py`
picks it up automatically — no per-collection config change.
export POSTGRES_URI="postgresql://docsgpt:docsgpt@localhost:5432/docsgpt"
export MONGO_URI="mongodb://user:pass@host:27017/docsgpt"
python scripts/db/backfill.py --dry-run # preview
python scripts/db/backfill.py # real run
# or: python scripts/db/backfill.py --tables users,agents
```
Then unset `MONGO_URI` and start the backend — nothing consults Mongo
in the default path anymore. The backfill is idempotent (per-table
`ON CONFLICT` upserts, event-log tables deduped via `mongo_id`), so
re-running is safe and re-syncs any drifted rows. Keep Mongo online
until you've verified Postgres is complete; decommission afterwards
unless you still use it as a vector store.
<Callout type="warning" emoji="⚠️">
No dual-write window and no runtime flag — on the current version,
Postgres is the only user-data store the backend reads or writes.
</Callout>
## Troubleshooting
- **`relation "..." does not exist`** — run `python scripts/db/init_postgres.py`.
- **`FATAL: role "docsgpt" does not exist`** — run the `CREATE ROLE` /
`CREATE DATABASE` statements from step 2 as a Postgres superuser.
- **`relation "..." does not exist`** — schema not applied. Either let
the app bootstrap it (`AUTO_MIGRATE=true`) or run
`python scripts/db/init_postgres.py`.
- **`permission denied to create database`** — the role lacks
`CREATEDB`. As superuser: `ALTER ROLE <name> CREATEDB;`. Or create
the database manually and set `AUTO_CREATE_DB=false`.
- **`role "docsgpt" does not exist`** — roles are never auto-created.
As superuser: `CREATE ROLE docsgpt LOGIN PASSWORD '...';`.
- **SSL errors on a managed provider** — append `?sslmode=require` to
`POSTGRES_URI`.
- **Dual-write warnings in the logs** — expected to be non-fatal. Mongo
is source of truth, so the user-facing request succeeds. Re-run the
backfill to re-sync whichever rows drifted.
- **`ModuleNotFoundError: pymongo`** — `pip install 'pymongo>=4.6'`
(only needed for the one-shot Mongo backfill).
docs/content/Guides/Architecture.mdx
+8 -7
View File
@@ -15,7 +15,7 @@ This diagram provides a bird's-eye view of the DocsGPT architecture, illustratin
flowchart LR
User["User"] --> Frontend["Frontend (React/Vite)"]
Frontend --> Backend["Backend API (Flask)"]
Backend --> LLM["LLM Integration Layer"] & VectorStore["Vector Stores"] & TaskQueue["Task Queue (Celery)"] & Databases["Databases (MongoDB, Redis)"]
Backend --> LLM["LLM Integration Layer"] & VectorStore["Vector Stores"] & TaskQueue["Task Queue (Celery)"] & Databases["Databases (Postgres, Redis)"]
LLM -- Cloud APIs / Local Engines --> InferenceEngine["Inference Engine"]
VectorStore -- Document Embeddings --> Indexes[("Indexes")]
TaskQueue -- Asynchronous Tasks --> DocumentIngestion["Document Ingestion"]
@@ -86,10 +86,11 @@ flowchart LR
* Improves application responsiveness by offloading heavy tasks.
* Enhances scalability and reliability through distributed task processing.
### 7. Databases (MongoDB, Redis)
### 7. Databases (Postgres, Redis)
* **Technology:** MongoDB and Redis.
* **Responsibility:** Databases are used for persistent data storage and caching. MongoDB stores structured data such as conversations, documents, user settings, and API keys. Redis is used as a cache, as well as a message broker for Celery.
* **Technology:** PostgreSQL and Redis.
* **Responsibility:** Databases are used for persistent data storage and caching. PostgreSQL stores structured user data such as conversations, agents, prompts, sources, attachments, workflows, logs, token usage, user settings, and API keys. Redis is used as a cache and as the message broker/result backend for Celery.
* **Note:** MongoDB is no longer used for user data. It remains an **optional** backend for the vector store (`VECTOR_STORE=mongodb`, i.e. Mongo Atlas Vector Search) and as the source database for the one-shot `scripts/db/backfill.py` migration from legacy installs.
## Request Flow Diagram
@@ -135,7 +136,7 @@ graph LR
RedisPod[Redis Pod]
end
subgraph Node 3
MongoDBPod[MongoDB Pod]
PostgresPod[Postgres Pod]
VectorStorePod[Vector Store Pod]
end
end
@@ -145,12 +146,12 @@ graph LR
docsgpt-api-service --> BackendAPIPod
BackendAPIPod --> CeleryWorkerPod
BackendAPIPod --> RedisPod
BackendAPIPod --> MongoDBPod
BackendAPIPod --> PostgresPod
BackendAPIPod --> VectorStorePod
CeleryWorkerPod --> RedisPod
BackendAPIPod --> InferenceEngine[(Inference Engine)]
VectorStorePod --> Indexes[(Indexes)]
MongoDBPod --> Data[(Data)]
PostgresPod --> Data[(Data)]
RedisPod --> Cache[(Cache)]
end
User[User] --> LoadBalancer
docs/content/Guides/ocr.mdx
+1 -1
View File
@@ -31,7 +31,7 @@ DOCLING_OCR_ATTACHMENTS_ENABLED=false
### Attachment flow (Chat-only file context)
1. Files are uploaded through `/api/store_attachment`.
2. Celery task `attachment_worker` parses and stores the attachment in MongoDB (`attachments` collection).
2. Celery task `attachment_worker` parses and stores the attachment in Postgres (`attachments` table).
3. OCR in this path is controlled by `DOCLING_OCR_ATTACHMENTS_ENABLED`.
4. Attachments are not vectorized and are not added to the source index.
5. During answer generation, selected attachment IDs are loaded and passed directly to the LLM pipeline.