mirror of
https://github.com/tiennm99/DocsGPT.git
synced 2026-10-12 16:15:14 +00:00
feat: air-gapped deployment guide, no implicit downloads
- Ship tiktoken's cl100k_base inside the package and build the encoding from it, so token counting never downloads anything. - Default EMBEDDINGS_CACHE_DIR to <data home>/models instead of FastEmbed's temp dir, and read tokenizer.json and repo metadata from that cache, so a model downloads once and survives reboots. - TTS_PROVIDER=none and STT_PROVIDER=none switch the speech features off: the endpoints return 404, audio files fail to ingest with a clear message, /api/config reports tts_available/stt_available, and the UI hides the Speak and microphone buttons. - Drop the Google Fonts Roboto import from the web UI. - prefetch-models fills the cache the app reads; verify-offline checks the packaged encoding. - Docs: new Air-Gapped Deployment guide, settings and cache notes.
This commit is contained in:
Alex committed
1 parent
ca69d1ea29
commit
7da46c2bea
38 files changed
+100892
-95
No files matched your search
@@ -0,0 +1,197 @@
|
||||
---
|
||||
title: Air-Gapped Deployment
|
||||
description: Run DocsGPT on a network with no internet access.
|
||||
---
|
||||
|
||||
import { Callout } from 'nextra/components'
|
||||
|
||||
# Air-Gapped Deployment
|
||||
|
||||
DocsGPT runs without internet access once three things are inside your network: the software, the embedding models, and an LLM you host yourself. This guide covers bringing them in, the settings that switch off features that need the internet, blocking outbound traffic, and checking that nothing gets out.
|
||||
|
||||
## What needs the internet
|
||||
|
||||
With a self-hosted LLM, chat, document ingestion and retrieval work offline. These parts of DocsGPT contact external services by default and need a setting:
|
||||
|
||||
| Feature | What it contacts | Air-gapped setting |
|
||||
| --- | --- | --- |
|
||||
| Default LLM (`LLM_PROVIDER=docsgpt`) | DocsGPT's hosted model API | Point DocsGPT at your own model server |
|
||||
| Version check | `gptcloud.arc53.com`, when the worker starts and periodically after | `VERSION_CHECK=0` |
|
||||
| Text-to-speech (Speak button) | Google (`google_tts`) or ElevenLabs | `TTS_PROVIDER=none` |
|
||||
| Speech-to-text (microphone, audio files) | OpenAI (`openai`) | `STT_PROVIDER=none` |
|
||||
| `read_webpage` chat tool | Any URL the model chooses | Leave it out of `DEFAULT_CHAT_TOOLS` |
|
||||
| Embedding models | Hugging Face, the first time a model is used | Bundled in the Docker image; `docsgpt prefetch-models` for pip |
|
||||
|
||||
Features whose purpose is to fetch outside content cannot work offline: URL, crawler, sitemap, GitHub and Reddit sources, the Google Drive, SharePoint and Confluence connectors, web search and other internet-facing tools, and hosted LLM or embedding providers.
|
||||
|
||||
<Callout type="info">
|
||||
Keep the default document parser. `DOC_PARSER_ENGINE=anydoc` needs no models, and OCR is off unless you turn it on. Use the default backend image rather than the `-docling` variant, and don't install the `docling` extra.
|
||||
</Callout>
|
||||
|
||||
## 1. Bring the software in
|
||||
|
||||
### Docker
|
||||
|
||||
The backend image already contains everything the default configuration loads at run time: both default embedding models (granite for new installs, mpnet for upgrades), their tokenizers and the token-counting encoding. A container downloads nothing on first use.
|
||||
|
||||
On a machine with internet access, pull the images and save them to one file:
|
||||
|
||||
```bash
|
||||
TAG=latest # or a release, e.g. 0.19.0
|
||||
docker pull arc53/docsgpt:$TAG
|
||||
docker pull arc53/docsgpt-fe:$TAG
|
||||
docker pull redis:6-alpine
|
||||
docker pull postgres:16-alpine
|
||||
docker save -o docsgpt-images.tar \
|
||||
arc53/docsgpt:$TAG arc53/docsgpt-fe:$TAG redis:6-alpine postgres:16-alpine
|
||||
```
|
||||
|
||||
Copy `docsgpt-images.tar` and the [standalone Compose file](/Deploying/Docker-Deploying#quickest-setup-pre-built-images-no-checkout) into the air-gapped network, then load the images (or push them to your internal registry):
|
||||
|
||||
```bash
|
||||
docker load -i docsgpt-images.tar
|
||||
```
|
||||
|
||||
Set `DOCSGPT_IMAGE_TAG` to the same tag when you start the stack. Bring your model server's image and weights in the same way, following its own documentation.
|
||||
|
||||
To use an embedding model other than the two defaults, build your own backend image with it baked in: `docker build -f docsgpt/Dockerfile --build-arg EMBEDDINGS_PREFETCH=<model> .`
|
||||
|
||||
### pip
|
||||
|
||||
On a connected machine with the same operating system, CPU architecture and Python version as the target, download the packages and the models:
|
||||
|
||||
```bash
|
||||
python -m venv .venv && source .venv/bin/activate
|
||||
pip download docsgpt -d wheelhouse
|
||||
pip install --no-index --find-links wheelhouse docsgpt
|
||||
DOCSGPT_HOME=./docsgpt-home docsgpt prefetch-models
|
||||
```
|
||||
|
||||
`prefetch-models` stores both default embedding models and their tokenizers under `models/` in the [data home](/Deploying/Pip-Install). Pass model names to fetch a different set.
|
||||
|
||||
Copy `wheelhouse/` and `docsgpt-home/models/` to the target, install from the wheelhouse, and put `models/` in the target's data home (or point `EMBEDDINGS_CACHE_DIR` at it):
|
||||
|
||||
```bash
|
||||
pip install --no-index --find-links wheelhouse docsgpt
|
||||
```
|
||||
|
||||
## 2. Point DocsGPT at your own LLM
|
||||
|
||||
Run an OpenAI-compatible model server inside the network, such as vLLM, Ollama, a llama.cpp server or Text Generation Inference, and set:
|
||||
|
||||
```env
|
||||
LLM_PROVIDER=openai
|
||||
OPENAI_BASE_URL=http://llm.internal:8000/v1
|
||||
LLM_NAME=your-model-name
|
||||
API_KEY=None
|
||||
```
|
||||
|
||||
Setting `OPENAI_BASE_URL` also removes DocsGPT's hosted model from the model list. See [Local inference engines](/Models/local-inference) for the base URL of each server.
|
||||
|
||||
## 3. Switch off the features that call out
|
||||
|
||||
Add these to `.env`:
|
||||
|
||||
```env
|
||||
VERSION_CHECK=0
|
||||
TTS_PROVIDER=none
|
||||
STT_PROVIDER=none
|
||||
DEFAULT_CHAT_TOOLS=["memory","scheduler"]
|
||||
HF_HUB_OFFLINE=1
|
||||
```
|
||||
|
||||
- `VERSION_CHECK=0` stops the worker's anonymous version check.
|
||||
- `TTS_PROVIDER=none` hides the Speak button on answers.
|
||||
- `STT_PROVIDER=none` hides the microphone button, and audio files fail to ingest with a message naming the setting. If your model server offers an OpenAI-compatible `/v1/audio/transcriptions` endpoint, keep `STT_PROVIDER=openai` instead: speech-to-text uses `OPENAI_BASE_URL` too.
|
||||
- `DEFAULT_CHAT_TOOLS` defaults to `memory`, `read_webpage` and `scheduler`; this list drops `read_webpage`.
|
||||
- `HF_HUB_OFFLINE=1` makes the Hugging Face libraries read models from the local cache only. A model that was never fetched fails at once with an error, instead of waiting on a connection that never opens. The libraries read it from the process environment: Docker Compose's `env_file` provides that, and for pip installs export it in the environment that starts `docsgpt api` and `docsgpt worker`.
|
||||
|
||||
Leave the document parsing and OCR settings at their defaults.
|
||||
|
||||
## 4. Block outbound traffic in the network
|
||||
|
||||
DocsGPT doesn't block outbound connections itself. Enforce that in the network, where the same rule also covers Postgres, Redis, your model server and anything you add later.
|
||||
|
||||
**Hosts with no route out.** The simplest setup: servers whose firewall allows only internal address ranges and your internal DNS.
|
||||
|
||||
**Docker.** Attach the services to a network created with `internal: true`, which has no route out of the Docker host. A container on an internal network can't publish ports, so give only a reverse proxy (nginx, Caddy, Traefik) a second, regular network and serve the UI and API through it:
|
||||
|
||||
```yaml
|
||||
networks:
|
||||
airgap:
|
||||
internal: true
|
||||
edge: {}
|
||||
|
||||
services:
|
||||
backend:
|
||||
networks: [airgap]
|
||||
worker:
|
||||
networks: [airgap]
|
||||
frontend:
|
||||
networks: [airgap]
|
||||
postgres:
|
||||
networks: [airgap]
|
||||
redis:
|
||||
networks: [airgap]
|
||||
proxy:
|
||||
image: nginx:stable
|
||||
networks: [airgap, edge]
|
||||
ports: ["443:443"]
|
||||
```
|
||||
|
||||
Docker's published ports bypass host firewalls such as `ufw`. If you rely on host rules instead of internal networks, filter container traffic in the `DOCKER-USER` iptables chain.
|
||||
|
||||
**Kubernetes.** With a network plugin that enforces NetworkPolicy, deny egress for the namespace except to its own pods and cluster DNS:
|
||||
|
||||
```yaml
|
||||
apiVersion: networking.k8s.io/v1
|
||||
kind: NetworkPolicy
|
||||
metadata:
|
||||
name: deny-external-egress
|
||||
namespace: docsgpt
|
||||
spec:
|
||||
podSelector: {}
|
||||
policyTypes: [Egress]
|
||||
egress:
|
||||
- to:
|
||||
- podSelector: {}
|
||||
- to:
|
||||
- namespaceSelector: {}
|
||||
podSelector:
|
||||
matchLabels:
|
||||
k8s-app: kube-dns
|
||||
ports:
|
||||
- protocol: UDP
|
||||
port: 53
|
||||
- protocol: TCP
|
||||
port: 53
|
||||
```
|
||||
|
||||
Add a rule for your model server if it runs in another namespace.
|
||||
|
||||
**Egress proxy.** When some traffic must leave the network, for example to a hosted LLM, send it through a forward proxy that allows only those hosts. The Python HTTP libraries DocsGPT uses honour the standard variables; exclude internal services in `NO_PROXY`:
|
||||
|
||||
```env
|
||||
HTTPS_PROXY=http://proxy.internal:3128
|
||||
HTTP_PROXY=http://proxy.internal:3128
|
||||
NO_PROXY=localhost,127.0.0.1,backend,llm.internal
|
||||
```
|
||||
|
||||
- Features that fetch user-supplied URLs connect to the resolved IP address, so a hostname allowlist can't match them. Keep those features off.
|
||||
- If the proxy inspects TLS, add its CA certificate with `SSL_CERT_FILE` and `REQUESTS_CA_BUNDLE`. For model downloads through such a proxy, also set `HF_HUB_DISABLE_XET=1`: Hugging Face's transfer client ignores `SSL_CERT_FILE`.
|
||||
|
||||
## 5. Verify
|
||||
|
||||
Check that the models load with networking disabled:
|
||||
|
||||
```bash
|
||||
docker run --rm --network none arc53/docsgpt:$TAG python -m docsgpt.scripts.verify_offline
|
||||
```
|
||||
|
||||
On a pip install, run `docsgpt verify-offline` on the air-gapped host. Both check token counting, each default embedding model's tokenizer, and an embedding; pass a model name to check a different model.
|
||||
|
||||
Then use the app: upload a document and ask a question about it. Your firewall or proxy logs should show no connection attempts from DocsGPT to outside hosts. The web UI loads its fonts and scripts from the DocsGPT server only.
|
||||
|
||||
<Callout type="info">
|
||||
The [chat widget](/Extensions/chat-widget) loads its font from Google Fonts and, unless you pass `buttonIcon`, its launcher icon from a CDN. Set `apiHost` to your own DocsGPT API; the default is DocsGPT's cloud.
|
||||
</Callout>
|
||||
@@ -78,7 +78,7 @@ To run the DocsGPT backend locally, you'll need to set up a Python environment a
|
||||
|
||||
3. **Embedding Model (no action needed):**
|
||||
|
||||
The embedding model is downloaded automatically the first time you ingest a document, and cached for subsequent runs. Set `EMBEDDINGS_CACHE_DIR` to control where.
|
||||
The embedding model is downloaded automatically the first time you ingest a document, and cached under `models/` in the repository root for subsequent runs. Set `EMBEDDINGS_CACHE_DIR` to use another directory.
|
||||
|
||||
For an offline or air-gapped machine, fetch it ahead of time instead:
|
||||
|
||||
|
||||
@@ -67,7 +67,9 @@ Here are some of the most fundamental settings you'll likely want to configure:
|
||||
|
||||
- **`OPENAI_BASE_URL`**: Specifically used when `LLM_PROVIDER` is set to `openai` but you are connecting to a local inference engine (like Ollama, Llama.cpp, etc.) that exposes an OpenAI-compatible API. This setting tells DocsGPT where to find your local LLM server.
|
||||
|
||||
- **`STT_PROVIDER`**: Selects the speech-to-text provider used for microphone transcription in chat and for audio file ingestion through the parser pipeline.
|
||||
- **`STT_PROVIDER`**: Selects the speech-to-text provider used for microphone transcription in chat and for audio file ingestion through the parser pipeline. Set it to `none` to switch speech-to-text off: the microphone button is hidden, and audio files fail to ingest with a message naming the setting.
|
||||
|
||||
- **`TTS_PROVIDER`**: Selects the text-to-speech provider behind the Speak button on answers: `google_tts` (default) or `elevenlabs`. Both are hosted services. Set it to `none` to switch text-to-speech off and hide the button.
|
||||
|
||||
- **`SERVE_UI`**: Whether the API process serves the web UI shipped in the Python package (`pip install docsgpt`). Defaults to `true`; set `false` when a separate frontend, such as the Docker frontend image, serves the UI. Has no effect in the backend Docker image, which carries no UI build.
|
||||
|
||||
@@ -238,7 +240,7 @@ The settings below control speech-to-text behaviour for both voice input and aud
|
||||
|
||||
| Setting | Purpose | Typical values |
|
||||
| --- | --- | --- |
|
||||
| `STT_PROVIDER` | Speech-to-text backend provider. | `openai`, `faster_whisper` |
|
||||
| `STT_PROVIDER` | Speech-to-text backend provider, or `none` to switch speech-to-text off. | `openai`, `faster_whisper`, `none` |
|
||||
| `OPENAI_STT_MODEL` | OpenAI transcription model used when `STT_PROVIDER=openai`. | `gpt-4o-mini-transcribe` |
|
||||
| `STT_LANGUAGE` | Optional language hint passed to the provider. Leave unset for auto-detection when supported. | `en`, `es`, unset |
|
||||
| `STT_MAX_FILE_SIZE_MB` | Maximum file size accepted by the synchronous `/api/stt` endpoint. | `50` |
|
||||
|
||||
@@ -49,7 +49,7 @@ pipx runpip docsgpt install --force-reinstall --no-deps --index-url https://down
|
||||
|
||||
## Configure
|
||||
|
||||
DocsGPT keeps its runtime files in a **data home**: the `.env` file it reads settings from, uploaded files under `inputs/` and vector indexes under `indexes/`. The data home is the directory you run the commands from, or the directory `DOCSGPT_HOME` points to. `DOCSGPT_ENV_FILE` points at a `.env` kept somewhere else. Both variables must be set in the process environment, not in `.env`: they decide where `.env` is read from.
|
||||
DocsGPT keeps its runtime files in a **data home**: the `.env` file it reads settings from, uploaded files under `inputs/`, vector indexes under `indexes/` and downloaded embedding models under `models/`. The data home is the directory you run the commands from, or the directory `DOCSGPT_HOME` points to. `DOCSGPT_ENV_FILE` points at a `.env` kept somewhere else. Both variables must be set in the process environment, not in `.env`: they decide where `.env` is read from.
|
||||
|
||||
Create a `.env` in the data home. The minimum for a hosted LLM:
|
||||
|
||||
@@ -87,7 +87,7 @@ The worker is not optional: query embedding runs on it, so search fails without
|
||||
Other commands:
|
||||
|
||||
- `docsgpt api --reload`: a development server with auto-reload.
|
||||
- `docsgpt prefetch-models`: download the embedding, tokenizer and parser models ahead of time, for machines that go offline.
|
||||
- `docsgpt prefetch-models`: download the embedding models and their tokenizers ahead of time, for machines that go offline (see [Air-Gapped Deployment](/Deploying/Air-Gapped)).
|
||||
- `docsgpt verify-offline`: check that a prepared install starts with networking off.
|
||||
- `docsgpt reembed`: re-embed every index after changing `EMBEDDINGS_NAME` (see [Upgrading](/upgrading)).
|
||||
|
||||
|
||||
@@ -27,6 +27,10 @@ export default {
|
||||
"title": "☸️ Deploying on Kubernetes",
|
||||
"href": "/Deploying/Kubernetes-Deploying"
|
||||
},
|
||||
"Air-Gapped": {
|
||||
"title": "🔒 Air-Gapped Deployment",
|
||||
"href": "/Deploying/Air-Gapped"
|
||||
},
|
||||
"Hosting-the-app": {
|
||||
"title": "☁️ Hosting DocsGPT",
|
||||
"href": "/Deploying/Hosting-the-app"
|
||||
|
||||
@@ -37,7 +37,7 @@ Set `EMBEDDINGS_NAME` in your `.env` to a registry name or a Hugging Face reposi
|
||||
EMBEDDINGS_NAME=ibm-granite/granite-embedding-311m-multilingual-r2
|
||||
```
|
||||
|
||||
The model is downloaded on first use and cached; set `EMBEDDINGS_CACHE_DIR` to control where. There is no `model/` folder to populate by hand, and a filesystem path is not accepted as a model name.
|
||||
The model is downloaded on first use and cached, together with its tokenizer, under `models/` in the data home; set `EMBEDDINGS_CACHE_DIR` to use another directory. There is no folder to populate by hand, and a filesystem path is not accepted as a model name.
|
||||
|
||||
DocsGPT knows the pooling, vector width and context window of the models in its registry (`all-mpnet-base-v2`, `granite-embedding-311m-multilingual-r2`, `granite-embedding-97m-multilingual-r2`). For any other repository it reads those from the repository's own `1_Pooling/config.json` and `modules.json`. If a repository declares neither, mean pooling with L2 normalization is assumed and a warning is logged — pin the real values with `EMBEDDINGS_POOLING` (`cls` or `mean`) and `EMBEDDINGS_NORMALIZE`.
|
||||
|
||||
|
||||
Reference in new issue
Block a user