mirror of
https://github.com/tiennm99/composes.git
synced 2026-10-11 03:13:16 +00:00
feat(gitea-mirror): split gitea-mirror out of the gitea stack into its own service
This commit is contained in:
tiennm99 committed
1 parent
5caa497d7f
commit
0a7e8f9783
11 files changed
+97
-61
No files matched your search
@@ -1,12 +1,12 @@
|
||||
---
|
||||
name: gitea-mirror-maintenance
|
||||
description: Detect and clean up failed, broken, or empty Gitea mirror repositories in the Coolify-deployed gitea + gitea-mirror stack, using tea and the gitea-mirror API. Use when the user asks to check mirror health, find failed or empty repos, investigate why a mirror did not sync or clone, delete broken mirror repos, delete archived copies of the user's own deleted repos, clean up duplicates left by renamed, transferred or re-cased GitHub repos, reclaim disk space from partial clones, re-mirror repos that failed, or run routine mirror upkeep. Not for Gitea setup, upgrades, or deployment problems — those belong to `gitea/compose.yml`.
|
||||
description: Detect and clean up failed, broken, or empty Gitea mirror repositories in the Coolify-deployed gitea + gitea-mirror stack, using tea and the gitea-mirror API. Use when the user asks to check mirror health, find failed or empty repos, investigate why a mirror did not sync or clone, delete broken mirror repos, delete archived copies of the user's own deleted repos, clean up duplicates left by renamed, transferred or re-cased GitHub repos, reclaim disk space from partial clones, re-mirror repos that failed, or run routine mirror upkeep. Not for Gitea or gitea-mirror setup, upgrades, or deployment problems — those belong to `gitea/compose.yml` and `gitea-mirror/compose.yml`.
|
||||
---
|
||||
|
||||
# Gitea Mirror Maintenance
|
||||
|
||||
Maintain the `gitea` + `gitea-mirror` stack deployed by
|
||||
`gitea/compose.yml` on Coolify: find mirror repositories whose pull failed, classify
|
||||
Maintain the `gitea` and `gitea-mirror` stacks deployed by
|
||||
`gitea/compose.yml` and `gitea-mirror/compose.yml` on Coolify: find mirror repositories whose pull failed, classify
|
||||
each failure, then clean up only what is safe to delete.
|
||||
|
||||
**Scope.** Mirror health auditing and cleanup only. Not Gitea first-run setup,
|
||||
|
||||
@@ -80,7 +80,8 @@ Each links to its own README for variables, ports, and storage.
|
||||
| [code-server](code-server/README.md) | VS Code in the browser, as a remote dev box |
|
||||
| [couchbase](couchbase/README.md) | Couchbase Server |
|
||||
| [diun](diun/README.md) | Image-update notifier, reading the Docker API through a read-only proxy |
|
||||
| [gitea](gitea/README.md) | Gitea + PostgreSQL + gitea-mirror, mirroring GitHub repos |
|
||||
| [gitea](gitea/README.md) | Gitea backed by PostgreSQL |
|
||||
| [gitea-mirror](gitea-mirror/README.md) | gitea-mirror, mirroring GitHub repos into Gitea |
|
||||
| [goclaw](goclaw/README.md) | Multi-tenant AI agent gateway, with pgvector PostgreSQL |
|
||||
| [hermes](hermes/README.md) | Hermes Agent with its built-in web dashboard |
|
||||
| [litellm](litellm/README.md) | LiteLLM proxy in front of many LLM providers, with PostgreSQL and Redis |
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
# gitea-mirror troubleshooting
|
||||
# gitea troubleshooting
|
||||
|
||||
Applies to `gitea/compose.yml`.
|
||||
|
||||
|
||||
@@ -0,0 +1,4 @@
|
||||
BETTER_AUTH_SECRET=
|
||||
ENCRYPTION_SECRET=
|
||||
GITEA_MIRROR_URL=https://gitea-mirror.example.com
|
||||
GITEA_CLONE_TIMEOUT=3600
|
||||
@@ -0,0 +1,3 @@
|
||||
gitea-mirror/
|
||||
.env
|
||||
purge.py
|
||||
@@ -0,0 +1,57 @@
|
||||
# gitea-mirror
|
||||
|
||||
[gitea-mirror](https://github.com/RayLabsHQ/gitea-mirror) mirroring GitHub
|
||||
repositories into a Gitea instance.
|
||||
|
||||
## Services
|
||||
|
||||
| Service | Image | Internal port | Domain |
|
||||
| --- | --- | --- | --- |
|
||||
| `gitea-mirror` | `ghcr.io/raylabshq/gitea-mirror:latest` | 4321 | `GITEA_MIRROR_URL` |
|
||||
|
||||
In Coolify, give `gitea-mirror` a domain on port 4321 matching
|
||||
`GITEA_MIRROR_URL`. The image ships its own health check.
|
||||
|
||||
## Variables
|
||||
|
||||
| Variable | Feeds | Notes |
|
||||
| --- | --- | --- |
|
||||
| `BETTER_AUTH_SECRET` | gitea-mirror | Signs sessions and encrypts its login keys. Generate with `openssl rand -base64 32`. |
|
||||
| `ENCRYPTION_SECRET` | gitea-mirror | Encrypts the stored GitHub and Gitea tokens. Generate with `openssl rand -base64 48`. |
|
||||
| `GITEA_MIRROR_URL` | `BETTER_AUTH_URL`, `PUBLIC_BETTER_AUTH_URL`, `BETTER_AUTH_TRUSTED_ORIGINS` | Public URL of the mirror UI, no trailing slash. |
|
||||
| `GITEA_CLONE_TIMEOUT` | `BUN_CONFIG_HTTP_IDLE_TIMEOUT` | Seconds to wait on Gitea's migrate API. Set it to the target Gitea's clone timeout. Defaults to `3600`; at most `14340`. |
|
||||
|
||||
Behind a reverse proxy, gitea-mirror rejects sign-in with "invalid origin"
|
||||
unless all three Better Auth variables hold the external URL, so one variable
|
||||
feeds them all.
|
||||
|
||||
Both secrets are set explicitly rather than left to the image, which would
|
||||
otherwise generate its own into `gitea-mirror-data`. Data encrypted under one
|
||||
secret is unreadable under another, so moving the data to a new deployment
|
||||
means carrying the secrets with it. Never change either on an existing
|
||||
install.
|
||||
|
||||
## Choices
|
||||
|
||||
- **Waits as long as Gitea clones.** Gitea's migrate API sends nothing until
|
||||
the clone ends, and Bun's `fetch` drops a connection idle for 5 minutes.
|
||||
gitea-mirror then marks the repository failed while Gitea keeps cloning; on
|
||||
retry it finds the half-made repository and marks it mirrored. If that
|
||||
clone later fails, Gitea keeps an empty repository with no mirror record,
|
||||
which never syncs. `BUN_CONFIG_HTTP_IDLE_TIMEOUT` takes Gitea's clone
|
||||
timeout so the request outlives the clone. Bun caps it at 239 minutes, so a
|
||||
larger value stops helping there.
|
||||
- **`gitea-mirror:latest`** with `pull_policy: always`: upstream publishes no
|
||||
major tag, so every redeploy takes the newest release.
|
||||
|
||||
## Usage
|
||||
|
||||
Sign up at `GITEA_MIRROR_URL`, then in its settings enter the Gitea instance's
|
||||
public URL and an access token, plus the GitHub token to mirror from. These
|
||||
are stored in `gitea-mirror-data`, not in the environment.
|
||||
|
||||
## Storage
|
||||
|
||||
| Volume | Holds |
|
||||
| --- | --- |
|
||||
| `gitea-mirror-data` | Mirror job database, settings and generated secrets |
|
||||
@@ -0,0 +1,17 @@
|
||||
services:
|
||||
gitea-mirror:
|
||||
image: ghcr.io/raylabshq/gitea-mirror:latest
|
||||
restart: unless-stopped
|
||||
pull_policy: always
|
||||
environment:
|
||||
BETTER_AUTH_SECRET: ${BETTER_AUTH_SECRET:?required}
|
||||
ENCRYPTION_SECRET: ${ENCRYPTION_SECRET:?required}
|
||||
BETTER_AUTH_URL: ${GITEA_MIRROR_URL:?required}
|
||||
PUBLIC_BETTER_AUTH_URL: ${GITEA_MIRROR_URL:?required}
|
||||
BETTER_AUTH_TRUSTED_ORIGINS: ${GITEA_MIRROR_URL:?required}
|
||||
BUN_CONFIG_HTTP_IDLE_TIMEOUT: ${GITEA_CLONE_TIMEOUT:-3600}
|
||||
volumes:
|
||||
- gitea-mirror-data:/app/data
|
||||
|
||||
volumes:
|
||||
gitea-mirror-data:
|
||||
@@ -1,6 +1,3 @@
|
||||
POSTGRES_PASSWORD=gitea
|
||||
GITEA_ROOT_URL=https://gitea.example.com/
|
||||
GITEA_CLONE_TIMEOUT=3600
|
||||
BETTER_AUTH_SECRET=
|
||||
ENCRYPTION_SECRET=
|
||||
GITEA_MIRROR_URL=https://gitea-mirror.example.com
|
||||
@@ -1,4 +1,3 @@
|
||||
gitea/
|
||||
gitea-mirror/
|
||||
.env
|
||||
purge.py
|
||||
+10
-37
@@ -1,8 +1,6 @@
|
||||
# gitea
|
||||
|
||||
Self-hosted [Gitea](https://about.gitea.com/) backed by PostgreSQL, with
|
||||
[gitea-mirror](https://github.com/RayLabsHQ/gitea-mirror) mirroring GitHub
|
||||
repositories into it.
|
||||
Self-hosted [Gitea](https://about.gitea.com/) backed by PostgreSQL.
|
||||
|
||||
## Services
|
||||
|
||||
@@ -10,13 +8,11 @@ repositories into it.
|
||||
| --- | --- | --- | --- |
|
||||
| `db` | `postgres:16-alpine` | 5432 | none |
|
||||
| `gitea` | `gitea/gitea:28` | 3000 | `GITEA_ROOT_URL` |
|
||||
| `gitea-mirror` | `ghcr.io/raylabshq/gitea-mirror:latest` | 4321 | `GITEA_MIRROR_URL` |
|
||||
|
||||
In Coolify, give `gitea` and `gitea-mirror` each a domain on their internal
|
||||
port, matching the two URL variables.
|
||||
In Coolify, give `gitea` a domain on port 3000 matching `GITEA_ROOT_URL`.
|
||||
|
||||
`gitea` waits for `db` to pass its health check. `gitea` checks
|
||||
`/api/healthz`; the `gitea-mirror` image ships its own health check.
|
||||
`gitea` waits for `db` to pass its health check, and checks `/api/healthz`
|
||||
itself.
|
||||
|
||||
## Variables
|
||||
|
||||
@@ -24,20 +20,7 @@ port, matching the two URL variables.
|
||||
| --- | --- | --- |
|
||||
| `POSTGRES_PASSWORD` | `db`, `gitea` | Defaults to `gitea`. |
|
||||
| `GITEA_ROOT_URL` | Gitea `server.ROOT_URL` | Public URL, with trailing slash. Gitea builds clone URLs and redirects from it. |
|
||||
| `GITEA_CLONE_TIMEOUT` | Gitea `git.timeout` `MIGRATE` and `MIRROR`; gitea-mirror `BUN_CONFIG_HTTP_IDLE_TIMEOUT` | Seconds a mirror's first clone or a later fetch may run. Defaults to `3600`; at most `14340`. |
|
||||
| `BETTER_AUTH_SECRET` | gitea-mirror | Signs sessions and encrypts its login keys. Generate with `openssl rand -base64 32`. |
|
||||
| `ENCRYPTION_SECRET` | gitea-mirror | Encrypts the stored GitHub and Gitea tokens. Generate with `openssl rand -base64 48`. |
|
||||
| `GITEA_MIRROR_URL` | `BETTER_AUTH_URL`, `PUBLIC_BETTER_AUTH_URL`, `BETTER_AUTH_TRUSTED_ORIGINS` | Public URL of the mirror UI, no trailing slash. |
|
||||
|
||||
Behind a reverse proxy, gitea-mirror rejects sign-in with "invalid origin"
|
||||
unless all three Better Auth variables hold the external URL, so one variable
|
||||
feeds them all.
|
||||
|
||||
Both secrets are set explicitly rather than left to the image, which would
|
||||
otherwise generate its own into `gitea-mirror-data`. Data encrypted under one
|
||||
secret is unreadable under another, so moving the data to a new deployment
|
||||
means carrying the secrets with it. Never change either on an existing
|
||||
install.
|
||||
| `GITEA_CLONE_TIMEOUT` | Gitea `git.timeout` `MIGRATE` and `MIRROR` | Seconds a mirror's first clone or a later fetch may run. Defaults to `3600`. |
|
||||
|
||||
## Choices
|
||||
|
||||
@@ -45,27 +28,18 @@ install.
|
||||
disabled and the UI offers HTTPS clone URLs only.
|
||||
- **One-hour clone timeout.** Gitea's defaults (600 s to migrate, 300 s to
|
||||
fetch) cut off multi-gigabyte repositories mid-clone, leaving empty mirrors
|
||||
that still hold gigabytes of unreachable packfiles.
|
||||
- **gitea-mirror waits as long as Gitea clones.** Gitea's migrate API sends
|
||||
nothing until the clone ends, and Bun's `fetch` drops a connection idle for
|
||||
5 minutes. gitea-mirror then marks the repository failed while Gitea keeps
|
||||
cloning; on retry it finds the half-made repository and marks it mirrored.
|
||||
If that clone later fails, Gitea keeps an empty repository with no mirror
|
||||
record, which never syncs. `BUN_CONFIG_HTTP_IDLE_TIMEOUT` takes the same
|
||||
value as the clone timeout so the request outlives the clone. Bun caps it
|
||||
at 239 minutes, so a larger `GITEA_CLONE_TIMEOUT` stops helping there.
|
||||
that still hold gigabytes of unreachable packfiles. A client that calls the
|
||||
migrate API gets no response until the clone ends, so it must wait at least
|
||||
this long.
|
||||
- **`gitea/gitea:28`.** Gitea publishes major tags; the major pin takes
|
||||
updates without a surprise major upgrade.
|
||||
- **`gitea-mirror:latest`** with `pull_policy: always`: upstream publishes no
|
||||
major tag, so every redeploy takes the newest release.
|
||||
- **`postgres:16-alpine`** stays on 16: a new Postgres major cannot read the
|
||||
existing data directory without a dump and restore.
|
||||
|
||||
## Usage
|
||||
|
||||
Complete Gitea's first-run setup at `GITEA_ROOT_URL`, create an access token,
|
||||
then configure mirroring at `GITEA_MIRROR_URL`. In gitea-mirror, set the Gitea
|
||||
URL to `http://gitea:3000` so it talks to Gitea over the internal network.
|
||||
Complete Gitea's first-run setup at `GITEA_ROOT_URL`, then create an access
|
||||
token for any tool that mirrors into it.
|
||||
|
||||
## Storage
|
||||
|
||||
@@ -73,4 +47,3 @@ URL to `http://gitea:3000` so it talks to Gitea over the internal network.
|
||||
| --- | --- |
|
||||
| `db-data` | PostgreSQL data |
|
||||
| `gitea-data` | Repositories, Gitea config and state |
|
||||
| `gitea-mirror-data` | Mirror job database and generated secrets |
|
||||
@@ -39,21 +39,6 @@ services:
|
||||
retries: 5
|
||||
start_period: 30s
|
||||
|
||||
gitea-mirror:
|
||||
image: ghcr.io/raylabshq/gitea-mirror:latest
|
||||
restart: unless-stopped
|
||||
pull_policy: always
|
||||
environment:
|
||||
BETTER_AUTH_SECRET: ${BETTER_AUTH_SECRET:?required}
|
||||
ENCRYPTION_SECRET: ${ENCRYPTION_SECRET:?required}
|
||||
BETTER_AUTH_URL: ${GITEA_MIRROR_URL:?required}
|
||||
PUBLIC_BETTER_AUTH_URL: ${GITEA_MIRROR_URL:?required}
|
||||
BETTER_AUTH_TRUSTED_ORIGINS: ${GITEA_MIRROR_URL:?required}
|
||||
BUN_CONFIG_HTTP_IDLE_TIMEOUT: ${GITEA_CLONE_TIMEOUT:-3600}
|
||||
volumes:
|
||||
- gitea-mirror-data:/app/data
|
||||
|
||||
volumes:
|
||||
db-data:
|
||||
gitea-data:
|
||||
gitea-mirror-data:
|
||||
Reference in new issue
Block a user