* feat: postgres tests

* feat: mongo cutoff

* feat: mongo cutoff

* feat: adjust docs and compose files

* fix: mini code mongo removals

* fix: tests and k8s mongo stuff

* feat: test fixes

* fix: ruff

* fix: vale

* Potential fix for pull request finding 'CodeQL / Clear-text logging of sensitive information'

Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>

* fix: mini suggestions

* vale lint fix 2

* fix: codeql columns thing

* fix: test mongo

* fix: tests coverage

* feat: better tests 4

* feat: more tests

* feat: decent coverage

* fix: ruff fixes

* fix: remove mongo mock

* feat: enhance workflow engine and API routes; add document retrieval and source handling

* feat: e2e tests

* fix: mcp, mongo and more

* fix: mini codeql warning

* fix: agent chunk view

* fix: mini issues

* fix: more pg fixes

* feat: postgres prep on start

* feat: qa tests

* fix: mini improvements

* fix: tests

---------

Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>
Co-authored-by: Siddhant Rai <siddhant.rai.5686@gmail.com>
This commit is contained in:
authored and GitHub committed 2026-04-18 13:13:57 +01:00
1 parent 7f6360b4ff
commit 81b6ee5daa
312 files changed
+763725 -25159

No files matched your search

+8 -7
View File
@@ -15,7 +15,7 @@ This diagram provides a bird's-eye view of the DocsGPT architecture, illustratin
flowchart LR
User["User"] --> Frontend["Frontend (React/Vite)"]
Frontend --> Backend["Backend API (Flask)"]
Backend --> LLM["LLM Integration Layer"] & VectorStore["Vector Stores"] & TaskQueue["Task Queue (Celery)"] & Databases["Databases (MongoDB, Redis)"]
Backend --> LLM["LLM Integration Layer"] & VectorStore["Vector Stores"] & TaskQueue["Task Queue (Celery)"] & Databases["Databases (Postgres, Redis)"]
LLM -- Cloud APIs / Local Engines --> InferenceEngine["Inference Engine"]
VectorStore -- Document Embeddings --> Indexes[("Indexes")]
TaskQueue -- Asynchronous Tasks --> DocumentIngestion["Document Ingestion"]
@@ -86,10 +86,11 @@ flowchart LR
* Improves application responsiveness by offloading heavy tasks.
* Enhances scalability and reliability through distributed task processing.
### 7. Databases (MongoDB, Redis)
### 7. Databases (Postgres, Redis)
* **Technology:** MongoDB and Redis.
* **Responsibility:** Databases are used for persistent data storage and caching. MongoDB stores structured data such as conversations, documents, user settings, and API keys. Redis is used as a cache, as well as a message broker for Celery.
* **Technology:** PostgreSQL and Redis.
* **Responsibility:** Databases are used for persistent data storage and caching. PostgreSQL stores structured user data such as conversations, agents, prompts, sources, attachments, workflows, logs, token usage, user settings, and API keys. Redis is used as a cache and as the message broker/result backend for Celery.
* **Note:** MongoDB is no longer used for user data. It remains an **optional** backend for the vector store (`VECTOR_STORE=mongodb`, i.e. Mongo Atlas Vector Search) and as the source database for the one-shot `scripts/db/backfill.py` migration from legacy installs.
## Request Flow Diagram
@@ -135,7 +136,7 @@ graph LR
RedisPod[Redis Pod]
end
subgraph Node 3
MongoDBPod[MongoDB Pod]
PostgresPod[Postgres Pod]
VectorStorePod[Vector Store Pod]
end
end
@@ -145,12 +146,12 @@ graph LR
docsgpt-api-service --> BackendAPIPod
BackendAPIPod --> CeleryWorkerPod
BackendAPIPod --> RedisPod
BackendAPIPod --> MongoDBPod
BackendAPIPod --> PostgresPod
BackendAPIPod --> VectorStorePod
CeleryWorkerPod --> RedisPod
BackendAPIPod --> InferenceEngine[(Inference Engine)]
VectorStorePod --> Indexes[(Indexes)]
MongoDBPod --> Data[(Data)]
PostgresPod --> Data[(Data)]
RedisPod --> Cache[(Cache)]
end
User[User] --> LoadBalancer
+1 -1
View File
@@ -31,7 +31,7 @@ DOCLING_OCR_ATTACHMENTS_ENABLED=false
### Attachment flow (Chat-only file context)
1. Files are uploaded through `/api/store_attachment`.
2. Celery task `attachment_worker` parses and stores the attachment in MongoDB (`attachments` collection).
2. Celery task `attachment_worker` parses and stores the attachment in Postgres (`attachments` table).
3. OCR in this path is controlled by `DOCLING_OCR_ATTACHMENTS_ENABLED`.
4. Attachments are not vectorized and are not added to the source index.
5. During answer generation, selected attachment IDs are loaded and passed directly to the LLM pipeline.