mirror of
https://github.com/tiennm99/DocsGPT.git
synced 2026-10-15 02:16:08 +00:00
Pg 4 (#2390)
* feat: postgres tests * feat: mongo cutoff * feat: mongo cutoff * feat: adjust docs and compose files * fix: mini code mongo removals * fix: tests and k8s mongo stuff * feat: test fixes * fix: ruff * fix: vale * Potential fix for pull request finding 'CodeQL / Clear-text logging of sensitive information' Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com> * fix: mini suggestions * vale lint fix 2 * fix: codeql columns thing * fix: test mongo * fix: tests coverage * feat: better tests 4 * feat: more tests * feat: decent coverage * fix: ruff fixes * fix: remove mongo mock * feat: enhance workflow engine and API routes; add document retrieval and source handling * feat: e2e tests * fix: mcp, mongo and more * fix: mini codeql warning * fix: agent chunk view * fix: mini issues * fix: more pg fixes * feat: postgres prep on start * feat: qa tests * fix: mini improvements * fix: tests --------- Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com> Co-authored-by: Siddhant Rai <siddhant.rai.5686@gmail.com>
This commit is contained in:
312 files changed
+763725
-25159
No files matched your search
@@ -15,7 +15,7 @@ This diagram provides a bird's-eye view of the DocsGPT architecture, illustratin
|
||||
flowchart LR
|
||||
User["User"] --> Frontend["Frontend (React/Vite)"]
|
||||
Frontend --> Backend["Backend API (Flask)"]
|
||||
Backend --> LLM["LLM Integration Layer"] & VectorStore["Vector Stores"] & TaskQueue["Task Queue (Celery)"] & Databases["Databases (MongoDB, Redis)"]
|
||||
Backend --> LLM["LLM Integration Layer"] & VectorStore["Vector Stores"] & TaskQueue["Task Queue (Celery)"] & Databases["Databases (Postgres, Redis)"]
|
||||
LLM -- Cloud APIs / Local Engines --> InferenceEngine["Inference Engine"]
|
||||
VectorStore -- Document Embeddings --> Indexes[("Indexes")]
|
||||
TaskQueue -- Asynchronous Tasks --> DocumentIngestion["Document Ingestion"]
|
||||
@@ -86,10 +86,11 @@ flowchart LR
|
||||
* Improves application responsiveness by offloading heavy tasks.
|
||||
* Enhances scalability and reliability through distributed task processing.
|
||||
|
||||
### 7. Databases (MongoDB, Redis)
|
||||
### 7. Databases (Postgres, Redis)
|
||||
|
||||
* **Technology:** MongoDB and Redis.
|
||||
* **Responsibility:** Databases are used for persistent data storage and caching. MongoDB stores structured data such as conversations, documents, user settings, and API keys. Redis is used as a cache, as well as a message broker for Celery.
|
||||
* **Technology:** PostgreSQL and Redis.
|
||||
* **Responsibility:** Databases are used for persistent data storage and caching. PostgreSQL stores structured user data such as conversations, agents, prompts, sources, attachments, workflows, logs, token usage, user settings, and API keys. Redis is used as a cache and as the message broker/result backend for Celery.
|
||||
* **Note:** MongoDB is no longer used for user data. It remains an **optional** backend for the vector store (`VECTOR_STORE=mongodb`, i.e. Mongo Atlas Vector Search) and as the source database for the one-shot `scripts/db/backfill.py` migration from legacy installs.
|
||||
|
||||
## Request Flow Diagram
|
||||
|
||||
@@ -135,7 +136,7 @@ graph LR
|
||||
RedisPod[Redis Pod]
|
||||
end
|
||||
subgraph Node 3
|
||||
MongoDBPod[MongoDB Pod]
|
||||
PostgresPod[Postgres Pod]
|
||||
VectorStorePod[Vector Store Pod]
|
||||
end
|
||||
end
|
||||
@@ -145,12 +146,12 @@ graph LR
|
||||
docsgpt-api-service --> BackendAPIPod
|
||||
BackendAPIPod --> CeleryWorkerPod
|
||||
BackendAPIPod --> RedisPod
|
||||
BackendAPIPod --> MongoDBPod
|
||||
BackendAPIPod --> PostgresPod
|
||||
BackendAPIPod --> VectorStorePod
|
||||
CeleryWorkerPod --> RedisPod
|
||||
BackendAPIPod --> InferenceEngine[(Inference Engine)]
|
||||
VectorStorePod --> Indexes[(Indexes)]
|
||||
MongoDBPod --> Data[(Data)]
|
||||
PostgresPod --> Data[(Data)]
|
||||
RedisPod --> Cache[(Cache)]
|
||||
end
|
||||
User[User] --> LoadBalancer
|
||||
|
||||
@@ -31,7 +31,7 @@ DOCLING_OCR_ATTACHMENTS_ENABLED=false
|
||||
### Attachment flow (Chat-only file context)
|
||||
|
||||
1. Files are uploaded through `/api/store_attachment`.
|
||||
2. Celery task `attachment_worker` parses and stores the attachment in MongoDB (`attachments` collection).
|
||||
2. Celery task `attachment_worker` parses and stores the attachment in Postgres (`attachments` table).
|
||||
3. OCR in this path is controlled by `DOCLING_OCR_ATTACHMENTS_ENABLED`.
|
||||
4. Attachments are not vectorized and are not added to the source index.
|
||||
5. During answer generation, selected attachment IDs are loaded and passed directly to the LLM pipeline.
|
||||
|
||||
Reference in new issue
Block a user