mirror of
https://github.com/tiennm99/DocsGPT.git
synced 2026-10-16 02:15:08 +00:00
refactor: rename the application package to docsgpt
The backend import package is now docsgpt, the name it will carry on PyPI; application was far too generic to install into anyone's site-packages. git mv plus a mechanical rewrite of every import, dotted string and path reference: 734 Python files, the compose files, Dockerfile, workflows, docs, setup scripts, devcontainer, k8s manifests, vscode config, pytest and coverage config, .gitignore. Behaviour is unchanged. Kept for one release: - A top-level application package whose meta-path finder resolves application.x.y to the already-imported docsgpt.x.y object, so old imports and entry points (celery -A application.app.celery, uvicorn application.asgi:asgi_app) keep working with a FutureWarning. - Celery registers every application.* task name as an alias of its docsgpt.* task on start-up, so messages queued by the previous release still run. The redbeat key prefix moves to redbeat:docsgpt:v2: so schedule entries the previous release wrote are left unread instead of firing twice. The backend image builds from the repository root (docker build -f docsgpt/Dockerfile .) so it can ship the alias package; a root .dockerignore allow-lists docsgpt/ and application/ and keeps caches, local data, .env files, the sample index files and the Dockerfile out. Compose and the image workflows point at the new context.
This commit is contained in:
1 parent
3e7f1f91b4
commit
574f96341e
984 files changed
+9847
-9682
No files matched your search
@@ -53,7 +53,7 @@ To run the DocsGPT backend locally, you'll need to set up a Python environment a
|
||||
|
||||
* **Option 1: Using a `.env` file (Recommended):**
|
||||
* If you haven't already, create a file named `.env` in the **root directory** of your DocsGPT project.
|
||||
* Modify the `.env` file to adjust settings as needed. You can find a comprehensive list of configurable options in [`application/core/settings.py`](https://github.com/arc53/DocsGPT/blob/main/application/core/settings.py).
|
||||
* Modify the `.env` file to adjust settings as needed. You can find a comprehensive list of configurable options in [`docsgpt/core/settings.py`](https://github.com/arc53/DocsGPT/blob/main/docsgpt/core/settings.py).
|
||||
|
||||
* **Option 2: Exporting Environment Variables:**
|
||||
* Alternatively, you can export environment variables directly in your terminal. However, using a `.env` file is generally more organized for development.
|
||||
@@ -83,7 +83,7 @@ To run the DocsGPT backend locally, you'll need to set up a Python environment a
|
||||
For an offline or air-gapped machine, fetch it ahead of time instead:
|
||||
|
||||
```bash
|
||||
python -m application.scripts.prefetch_models
|
||||
python -m docsgpt.scripts.prefetch_models
|
||||
```
|
||||
|
||||
4. **Install Backend Dependencies:**
|
||||
@@ -91,7 +91,7 @@ To run the DocsGPT backend locally, you'll need to set up a Python environment a
|
||||
Navigate to the root of your DocsGPT repository and install the required Python packages:
|
||||
|
||||
```bash
|
||||
pip install -r application/requirements.txt
|
||||
pip install -r docsgpt/requirements.txt
|
||||
```
|
||||
|
||||
Dependencies are declared in `pyproject.toml` and locked in `uv.lock`; the
|
||||
@@ -103,8 +103,8 @@ To run the DocsGPT backend locally, you'll need to set up a Python environment a
|
||||
feature (each file is the core set plus the extra):
|
||||
|
||||
```bash
|
||||
pip install -r application/requirements-docling.txt # docling parser engine: OCR backend, read_document structured output
|
||||
pip install -r application/requirements-milvus.txt # VECTOR_STORE=milvus
|
||||
pip install -r docsgpt/requirements-docling.txt # docling parser engine: OCR backend, read_document structured output
|
||||
pip install -r docsgpt/requirements-milvus.txt # VECTOR_STORE=milvus
|
||||
# or with uv: uv sync --extra docling --extra milvus
|
||||
```
|
||||
|
||||
@@ -122,25 +122,25 @@ To run the DocsGPT backend locally, you'll need to set up a Python environment a
|
||||
For local development, run the ASGI composition under uvicorn. It serves the **whole** application, hot-reloads on source changes, and matches the production runtime:
|
||||
|
||||
```bash
|
||||
uvicorn application.asgi:asgi_app --host 0.0.0.0 --port 7091 --reload
|
||||
uvicorn docsgpt.asgi:asgi_app --host 0.0.0.0 --port 7091 --reload
|
||||
```
|
||||
|
||||
This makes the backend accessible on `http://localhost:7091`. Production uses `gunicorn -k uvicorn_worker.UvicornWorker` against the same `application.asgi:asgi_app` target.
|
||||
This makes the backend accessible on `http://localhost:7091`. Production uses `gunicorn -k uvicorn_worker.UvicornWorker` against the same `docsgpt.asgi:asgi_app` target.
|
||||
|
||||
A plain Flask run is a faster inner loop (quick startup, the Werkzeug interactive debugger):
|
||||
|
||||
```bash
|
||||
flask --app application/app.py run --host=0.0.0.0 --port=7091
|
||||
flask --app docsgpt/app.py run --host=0.0.0.0 --port=7091
|
||||
```
|
||||
|
||||
But it serves **only** the WSGI Flask app and omits the routes mounted on the ASGI shell in `application/asgi.py`: the `/mcp` FastMCP endpoint and the native-async SSE reconnect reader `GET /api/messages/<id>/events`. Under `flask run` those paths return 404 — chat still works (`POST /stream` is a Flask route), but a stream interrupted by a disconnect won't auto-resume on reconnect. Use `flask run` only when you don't need those routes.
|
||||
But it serves **only** the WSGI Flask app and omits the routes mounted on the ASGI shell in `docsgpt/asgi.py`: the `/mcp` FastMCP endpoint and the native-async SSE reconnect reader `GET /api/messages/<id>/events`. Under `flask run` those paths return 404 — chat still works (`POST /stream` is a Flask route), but a stream interrupted by a disconnect won't auto-resume on reconnect. Use `flask run` only when you don't need those routes.
|
||||
|
||||
6. **Start the Celery Worker:**
|
||||
|
||||
Open a new terminal window (and activate your virtual environment if you used one). Start the Celery worker to handle background tasks:
|
||||
|
||||
```bash
|
||||
celery -A application.app.celery worker -l INFO
|
||||
celery -A docsgpt.app.celery worker -l INFO
|
||||
```
|
||||
|
||||
This command will start the Celery worker, which processes tasks such as document parsing and vector embedding.
|
||||
@@ -148,7 +148,7 @@ To run the DocsGPT backend locally, you'll need to set up a Python environment a
|
||||
**macOS note:** Due to a threading issue, start Celery with the solo pool:
|
||||
|
||||
```bash
|
||||
python -m celery -A application.app.celery worker -l INFO --pool=solo
|
||||
python -m celery -A docsgpt.app.celery worker -l INFO --pool=solo
|
||||
```
|
||||
|
||||
**Running in Debugger (VSCode):**
|
||||
|
||||
@@ -69,8 +69,8 @@ checkout.
|
||||
## Using the Source Checkout
|
||||
|
||||
With a clone of the repository, `deployment/docker-compose-hub.yaml` runs the
|
||||
same pre-built images while keeping your data in `application/indexes`,
|
||||
`application/inputs` and `application/vectors`, and `deployment/docker-compose.yaml`
|
||||
same pre-built images while keeping your data in `docsgpt/indexes`,
|
||||
`docsgpt/inputs` and `docsgpt/vectors`, and `deployment/docker-compose.yaml`
|
||||
builds the images from your working tree (for local changes, or a build with
|
||||
extra packages: `EXTRAS=docling` in `.env`).
|
||||
|
||||
|
||||
@@ -29,11 +29,11 @@ LLM_NAME=gpt-4o
|
||||
|
||||
### 2. Configuration via `settings.py` file (Advanced)
|
||||
|
||||
For more advanced configurations or if you prefer to manage settings directly in code, you can modify the `settings.py` file. This file is located in the `application/core` directory of your DocsGPT project.
|
||||
For more advanced configurations or if you prefer to manage settings directly in code, you can modify the `settings.py` file. This file is located in the `docsgpt/core` directory of your DocsGPT project.
|
||||
|
||||
While modifying `settings.py` offers more flexibility, it's generally recommended to use the `.env` file for basic settings and reserve `settings.py` for more complex adjustments or when you need to configure settings programmatically.
|
||||
|
||||
**Location of `settings.py`:** `application/core/settings.py`
|
||||
**Location of `settings.py`:** `docsgpt/core/settings.py`
|
||||
|
||||
## Basic Settings Explained
|
||||
|
||||
@@ -61,7 +61,7 @@ Here are some of the most fundamental settings you'll likely want to configure:
|
||||
|
||||
- **Default value:** leave it unset and DocsGPT picks for you at first boot, recording the choice so it never changes underneath you: a fresh install is pinned to `ibm-granite/granite-embedding-311m-multilingual-r2` (multilingual, 32k context, same 768 dimensions), and an install that already has sources is pinned to `huggingface_sentence-transformers/all-mpnet-base-v2` so its index stays readable. Setting it here overrides that pin.
|
||||
- **Other options:** Any FastEmbed built-in model, or any Hugging Face repository shipping an ONNX export. See [Embeddings](/Models/embeddings).
|
||||
- **Changing it on an existing index requires re-embedding** — same-width models swap without any error and silently degrade retrieval. Run `python -m application.scripts.reembed`.
|
||||
- **Changing it on an existing index requires re-embedding** — same-width models swap without any error and silently degrade retrieval. Run `python -m docsgpt.scripts.reembed`.
|
||||
|
||||
- **`API_KEY`**: Required for most cloud-based LLM providers. This is your authentication key to access the LLM provider's API. You'll need to obtain this key from your chosen provider's platform.
|
||||
|
||||
@@ -133,7 +133,7 @@ models:
|
||||
|
||||
After restart, those models appear in `/api/models` and are selectable
|
||||
in the UI. A working template lives at
|
||||
`application/core/models/examples/mistral.yaml.example`.
|
||||
`docsgpt/core/models/examples/mistral.yaml.example`.
|
||||
|
||||
**What you can do:**
|
||||
|
||||
@@ -148,8 +148,8 @@ in the UI. A working template lives at
|
||||
|
||||
**What you cannot do via `MODELS_CONFIG_DIR`:** add a brand-new
|
||||
non-OpenAI provider. That requires a Python plugin under
|
||||
`application/llm/providers/`. See
|
||||
`application/core/models/README.md` for the full schema reference.
|
||||
`docsgpt/llm/providers/`. See
|
||||
`docsgpt/core/models/README.md` for the full schema reference.
|
||||
|
||||
### Docker
|
||||
|
||||
@@ -207,7 +207,7 @@ for the engines and flows.
|
||||
|
||||
| Setting | Default | Description |
|
||||
| --- | --- | --- |
|
||||
| `DOC_PARSER_ENGINE` | `anydoc` | Parser engine: `anydoc` (fast Rust converter, no ML models) or `docling` (layout/table models, structured output). Files anydoc cannot read fall back to Docling when installed. Docling is an optional extra: `pip install -r application/requirements-docling.txt`, or Docker builds with `--build-arg INSTALL_DOCLING=true`. |
|
||||
| `DOC_PARSER_ENGINE` | `anydoc` | Parser engine: `anydoc` (fast Rust converter, no ML models) or `docling` (layout/table models, structured output). Files anydoc cannot read fall back to Docling when installed. Docling is an optional extra: `pip install -r docsgpt/requirements-docling.txt`, or Docker builds with `--build-arg INSTALL_DOCLING=true`. |
|
||||
| `OCR_ENABLED` | `false` | OCR for source ingestion. Alias: `DOCLING_OCR_ENABLED`. |
|
||||
| `OCR_ATTACHMENTS_ENABLED` | `false` | OCR for chat attachments. Alias: `DOCLING_OCR_ATTACHMENTS_ENABLED`. |
|
||||
| `OCR_BACKEND` | `auto` | Who performs OCR: `auto` (Docling when installed, else native), `native` (pypdfium2 + Pillow rendering into tesseract or DeepSeek-OCR; no docling needed), or `docling` (layout-model hybrid OCR). See the [OCR guide](/Guides/ocr#ocr-backends). |
|
||||
@@ -469,7 +469,7 @@ See [Embeddings](/Models/embeddings) for full guidance.
|
||||
|
||||
| Setting | Default | Description |
|
||||
| --- | --- | --- |
|
||||
| `EMBEDDINGS_NAME` | `huggingface_sentence-transformers/all-mpnet-base-v2` | The embedding model. New installs use `ibm-granite/granite-embedding-311m-multilingual-r2`. Changing it on a populated index requires `application.scripts.reembed`. |
|
||||
| `EMBEDDINGS_NAME` | `huggingface_sentence-transformers/all-mpnet-base-v2` | The embedding model. New installs use `ibm-granite/granite-embedding-311m-multilingual-r2`. Changing it on a populated index requires `docsgpt.scripts.reembed`. |
|
||||
| `EMBEDDINGS_BASE_URL` | unset | Base URL of a remote OpenAI-compatible embeddings server. Setting it routes all embedding calls there. |
|
||||
| `EMBEDDINGS_THREADS` | unset (Docker image: `4`) | Threads one local FastEmbed/onnxruntime session may use. onnxruntime otherwise sizes its pool to the host's core count, which a CPU-limited container still reports, so the image pins it like `OMP_NUM_THREADS`. Raise it on a large dedicated worker. |
|
||||
| `EMBEDDINGS_KEY` | unset | Optional bearer token for the remote embeddings server. |
|
||||
@@ -550,4 +550,4 @@ These are just the basic settings to get you started. The `settings.py` file con
|
||||
- Cache settings (`CACHE_REDIS_URL`)
|
||||
- And many more!
|
||||
|
||||
For a complete list of available settings and their descriptions, refer to the `settings.py` file in `application/core`. Remember to restart your Docker containers after making changes to your `.env` file or `settings.py` for the changes to take effect.
|
||||
For a complete list of available settings and their descriptions, refer to the `settings.py` file in `docsgpt/core`. Remember to restart your Docker containers after making changes to your `.env` file or `settings.py` for the changes to take effect.
|
||||
@@ -8,7 +8,7 @@ import { Callout } from 'nextra/components'
|
||||
# Observability
|
||||
|
||||
DocsGPT bundles the OpenTelemetry SDK and auto-instrumentation packages
|
||||
in `application/requirements.txt` — they install with the rest of the
|
||||
in `docsgpt/requirements.txt` — they install with the rest of the
|
||||
backend deps. Telemetry is **off by default**; opt in by prefixing the
|
||||
launch command with `opentelemetry-instrument` and setting OTLP env
|
||||
vars.
|
||||
@@ -42,12 +42,12 @@ services:
|
||||
backend:
|
||||
command: >
|
||||
opentelemetry-instrument gunicorn -w 1 -k uvicorn_worker.UvicornWorker
|
||||
--bind 0.0.0.0:7091 --config application/gunicorn_conf.py
|
||||
application.asgi:asgi_app
|
||||
--bind 0.0.0.0:7091 --config docsgpt/gunicorn_conf.py
|
||||
docsgpt.asgi:asgi_app
|
||||
environment:
|
||||
- OTEL_SERVICE_NAME=docsgpt-backend
|
||||
worker:
|
||||
command: opentelemetry-instrument celery -A application.app.celery worker -l INFO -B
|
||||
command: opentelemetry-instrument celery -A docsgpt.app.celery worker -l INFO -B
|
||||
environment:
|
||||
- OTEL_SERVICE_NAME=docsgpt-celery-worker
|
||||
```
|
||||
@@ -56,14 +56,14 @@ For local dev, prepend `dotenv run --` so the `OTEL_*` vars from `.env`
|
||||
reach `opentelemetry-instrument` before it boots the SDK:
|
||||
|
||||
```bash
|
||||
dotenv run -- opentelemetry-instrument flask --app application/app.py run --port=7091
|
||||
dotenv run -- opentelemetry-instrument celery -A application.app.celery worker -l INFO --pool=solo
|
||||
dotenv run -- opentelemetry-instrument flask --app docsgpt/app.py run --port=7091
|
||||
dotenv run -- opentelemetry-instrument celery -A docsgpt.app.celery worker -l INFO --pool=solo
|
||||
```
|
||||
|
||||
|
||||
<Callout type="info" emoji="ℹ️">
|
||||
Logs are exported in-process when `OTEL_LOGS_EXPORTER=otlp` is set —
|
||||
`application/core/logging_config.py` detects the flag and preserves
|
||||
`docsgpt/core/logging_config.py` detects the flag and preserves
|
||||
the OTEL log handler. Without it, `logging` writes only to stdout.
|
||||
</Callout>
|
||||
|
||||
|
||||
@@ -37,7 +37,7 @@ schema on first boot.
|
||||
|
||||
```bash
|
||||
export POSTGRES_URI="postgresql://user:pass@host/docsgpt?sslmode=require"
|
||||
uvicorn application.asgi:asgi_app --host 0.0.0.0 --port 7091
|
||||
uvicorn docsgpt.asgi:asgi_app --host 0.0.0.0 --port 7091
|
||||
```
|
||||
|
||||
### Bare-metal Postgres
|
||||
@@ -47,7 +47,7 @@ First boot creates both the database and the schema.
|
||||
|
||||
```bash
|
||||
export POSTGRES_URI="postgresql://postgres@localhost/docsgpt"
|
||||
uvicorn application.asgi:asgi_app --host 0.0.0.0 --port 7091
|
||||
uvicorn docsgpt.asgi:asgi_app --host 0.0.0.0 --port 7091
|
||||
```
|
||||
|
||||
Prefer a dedicated non-superuser role? Create it once as superuser — the
|
||||
@@ -93,7 +93,7 @@ init-container ahead of the app rollout:
|
||||
```bash
|
||||
python scripts/db/init_postgres.py
|
||||
# equivalently:
|
||||
alembic -c application/alembic.ini upgrade head
|
||||
alembic -c docsgpt/alembic.ini upgrade head
|
||||
```
|
||||
|
||||
The reasoning: the app's runtime role shouldn't carry DDL privileges,
|
||||
@@ -112,7 +112,7 @@ One-shot, offline, app stopped. The app itself will create the
|
||||
Postgres schema when it boots — you only need to run the data copy.
|
||||
|
||||
```bash
|
||||
pip install -r application/requirements.txt
|
||||
pip install -r docsgpt/requirements.txt
|
||||
pip install 'pymongo>=4.6'
|
||||
|
||||
export POSTGRES_URI="postgresql://docsgpt:docsgpt@localhost:5432/docsgpt"
|
||||
|
||||
Reference in new issue
Block a user