From 03cc60d481822334b8a3b9ad0123a3dc0626e123 Mon Sep 17 00:00:00 2001 From: Alex Date: Sat, 5 Sep 2026 22:15:31 +0100 Subject: [PATCH] build(docker): pin onnxruntime threads in the image OMP_NUM_THREADS=4 capped torch in the old image, but FastEmbed runs on onnxruntime, which ignores it and sizes its pool to the host's core count. A CPU-limited container still reports every host core, so embedding ran with 64 threads on a 4-CPU sandbox and took up to 12x longer per file. The image now sets EMBEDDINGS_THREADS=4 next to the existing caps; the settings guide documents the knob. --- application/Dockerfile | 6 +++++- docs/content/Deploying/DocsGPT-Settings.mdx | 1 + 2 files changed, 6 insertions(+), 1 deletion(-) diff --git a/application/Dockerfile b/application/Dockerfile index 5e9fdd57..4fd045fd 100644 --- a/application/Dockerfile +++ b/application/Dockerfile @@ -143,10 +143,14 @@ RUN mkdir -p /app/application/inputs/local /app/inputs /app/indexes /app/vectors ENV FLASK_APP=app.py +# Thread caps. onnxruntime (FastEmbed) ignores OMP_NUM_THREADS and sizes its +# pool to the host's core count, which a CPU-limited container still reports; +# EMBEDDINGS_THREADS pins it the way OMP_NUM_THREADS pinned torch before. ENV MALLOC_ARENA_MAX=2 \ OMP_NUM_THREADS=4 \ MKL_NUM_THREADS=4 \ - OPENBLAS_NUM_THREADS=4 + OPENBLAS_NUM_THREADS=4 \ + EMBEDDINGS_THREADS=4 EXPOSE 7091 diff --git a/docs/content/Deploying/DocsGPT-Settings.mdx b/docs/content/Deploying/DocsGPT-Settings.mdx index 3779e7aa..9f5c460d 100644 --- a/docs/content/Deploying/DocsGPT-Settings.mdx +++ b/docs/content/Deploying/DocsGPT-Settings.mdx @@ -471,6 +471,7 @@ See [Embeddings](/Models/embeddings) for full guidance. | --- | --- | --- | | `EMBEDDINGS_NAME` | `huggingface_sentence-transformers/all-mpnet-base-v2` | The embedding model. New installs use `ibm-granite/granite-embedding-311m-multilingual-r2`. Changing it on a populated index requires `application.scripts.reembed`. | | `EMBEDDINGS_BASE_URL` | unset | Base URL of a remote OpenAI-compatible embeddings server. Setting it routes all embedding calls there. | +| `EMBEDDINGS_THREADS` | unset (Docker image: `4`) | Threads one local FastEmbed/onnxruntime session may use. onnxruntime otherwise sizes its pool to the host's core count, which a CPU-limited container still reports, so the image pins it like `OMP_NUM_THREADS`. Raise it on a large dedicated worker. | | `EMBEDDINGS_KEY` | unset | Optional bearer token for the remote embeddings server. | | `EMBEDDINGS_MAX_INPUT_TOKENS` | unset | Truncate each remote embedding input to N tokens (guards servers that reject oversized inputs). | | `EMBEDDINGS_DELEGATE_TO_WORKER` | `true` | Embed queries on the Celery worker instead of loading a model in the API. Requires a worker consuming `EMBEDDINGS_QUEUE`; set `false` to run the API standalone. Ignored when `EMBEDDINGS_BASE_URL` is set. |