diff --git a/.claude/skills/gitea-mirror-maintenance/SKILL.md b/.claude/skills/gitea-mirror-maintenance/SKILL.md index d7452a8..c8d4087 100644 --- a/.claude/skills/gitea-mirror-maintenance/SKILL.md +++ b/.claude/skills/gitea-mirror-maintenance/SKILL.md @@ -1,12 +1,12 @@ --- name: gitea-mirror-maintenance -description: Detect and clean up failed, broken, or empty Gitea mirror repositories in the Coolify-deployed gitea + gitea-mirror stack, using tea and the gitea-mirror API. Use when the user asks to check mirror health, find failed or empty repos, investigate why a mirror did not sync or clone, delete broken mirror repos, delete archived copies of the user's own deleted repos, clean up duplicates left by renamed, transferred or re-cased GitHub repos, reclaim disk space from partial clones, re-mirror repos that failed, or run routine mirror upkeep. Not for Gitea setup, upgrades, or deployment problems — those belong to `gitea/compose.yml`. +description: Detect and clean up failed, broken, or empty Gitea mirror repositories in the Coolify-deployed gitea + gitea-mirror stack, using tea and the gitea-mirror API. Use when the user asks to check mirror health, find failed or empty repos, investigate why a mirror did not sync or clone, delete broken mirror repos, delete archived copies of the user's own deleted repos, clean up duplicates left by renamed, transferred or re-cased GitHub repos, reclaim disk space from partial clones, re-mirror repos that failed, or run routine mirror upkeep. Not for Gitea or gitea-mirror setup, upgrades, or deployment problems — those belong to `gitea/compose.yml` and `gitea-mirror/compose.yml`. --- # Gitea Mirror Maintenance -Maintain the `gitea` + `gitea-mirror` stack deployed by -`gitea/compose.yml` on Coolify: find mirror repositories whose pull failed, classify +Maintain the `gitea` and `gitea-mirror` stacks deployed by +`gitea/compose.yml` and `gitea-mirror/compose.yml` on Coolify: find mirror repositories whose pull failed, classify each failure, then clean up only what is safe to delete. **Scope.** Mirror health auditing and cleanup only. Not Gitea first-run setup, diff --git a/README.md b/README.md index 9251410..eaba52b 100644 --- a/README.md +++ b/README.md @@ -80,7 +80,8 @@ Each links to its own README for variables, ports, and storage. | [code-server](code-server/README.md) | VS Code in the browser, as a remote dev box | | [couchbase](couchbase/README.md) | Couchbase Server | | [diun](diun/README.md) | Image-update notifier, reading the Docker API through a read-only proxy | -| [gitea](gitea/README.md) | Gitea + PostgreSQL + gitea-mirror, mirroring GitHub repos | +| [gitea](gitea/README.md) | Gitea backed by PostgreSQL | +| [gitea-mirror](gitea-mirror/README.md) | gitea-mirror, mirroring GitHub repos into Gitea | | [goclaw](goclaw/README.md) | Multi-tenant AI agent gateway, with pgvector PostgreSQL | | [hermes](hermes/README.md) | Hermes Agent with its built-in web dashboard | | [litellm](litellm/README.md) | LiteLLM proxy in front of many LLM providers, with PostgreSQL and Redis | diff --git a/docs/gitea/troubleshooting.md b/docs/gitea/troubleshooting.md index f9d05a7..0447e9a 100644 --- a/docs/gitea/troubleshooting.md +++ b/docs/gitea/troubleshooting.md @@ -1,4 +1,4 @@ -# gitea-mirror troubleshooting +# gitea troubleshooting Applies to `gitea/compose.yml`. diff --git a/gitea-mirror/.env.example b/gitea-mirror/.env.example new file mode 100644 index 0000000..a8ac83b --- /dev/null +++ b/gitea-mirror/.env.example @@ -0,0 +1,4 @@ +BETTER_AUTH_SECRET= +ENCRYPTION_SECRET= +GITEA_MIRROR_URL=https://gitea-mirror.example.com +GITEA_CLONE_TIMEOUT=3600 diff --git a/gitea-mirror/.gitignore b/gitea-mirror/.gitignore new file mode 100644 index 0000000..835601f --- /dev/null +++ b/gitea-mirror/.gitignore @@ -0,0 +1,3 @@ +gitea-mirror/ +.env +purge.py diff --git a/gitea-mirror/README.md b/gitea-mirror/README.md new file mode 100644 index 0000000..830c87d --- /dev/null +++ b/gitea-mirror/README.md @@ -0,0 +1,57 @@ +# gitea-mirror + +[gitea-mirror](https://github.com/RayLabsHQ/gitea-mirror) mirroring GitHub +repositories into a Gitea instance. + +## Services + +| Service | Image | Internal port | Domain | +| --- | --- | --- | --- | +| `gitea-mirror` | `ghcr.io/raylabshq/gitea-mirror:latest` | 4321 | `GITEA_MIRROR_URL` | + +In Coolify, give `gitea-mirror` a domain on port 4321 matching +`GITEA_MIRROR_URL`. The image ships its own health check. + +## Variables + +| Variable | Feeds | Notes | +| --- | --- | --- | +| `BETTER_AUTH_SECRET` | gitea-mirror | Signs sessions and encrypts its login keys. Generate with `openssl rand -base64 32`. | +| `ENCRYPTION_SECRET` | gitea-mirror | Encrypts the stored GitHub and Gitea tokens. Generate with `openssl rand -base64 48`. | +| `GITEA_MIRROR_URL` | `BETTER_AUTH_URL`, `PUBLIC_BETTER_AUTH_URL`, `BETTER_AUTH_TRUSTED_ORIGINS` | Public URL of the mirror UI, no trailing slash. | +| `GITEA_CLONE_TIMEOUT` | `BUN_CONFIG_HTTP_IDLE_TIMEOUT` | Seconds to wait on Gitea's migrate API. Set it to the target Gitea's clone timeout. Defaults to `3600`; at most `14340`. | + +Behind a reverse proxy, gitea-mirror rejects sign-in with "invalid origin" +unless all three Better Auth variables hold the external URL, so one variable +feeds them all. + +Both secrets are set explicitly rather than left to the image, which would +otherwise generate its own into `gitea-mirror-data`. Data encrypted under one +secret is unreadable under another, so moving the data to a new deployment +means carrying the secrets with it. Never change either on an existing +install. + +## Choices + +- **Waits as long as Gitea clones.** Gitea's migrate API sends nothing until + the clone ends, and Bun's `fetch` drops a connection idle for 5 minutes. + gitea-mirror then marks the repository failed while Gitea keeps cloning; on + retry it finds the half-made repository and marks it mirrored. If that + clone later fails, Gitea keeps an empty repository with no mirror record, + which never syncs. `BUN_CONFIG_HTTP_IDLE_TIMEOUT` takes Gitea's clone + timeout so the request outlives the clone. Bun caps it at 239 minutes, so a + larger value stops helping there. +- **`gitea-mirror:latest`** with `pull_policy: always`: upstream publishes no + major tag, so every redeploy takes the newest release. + +## Usage + +Sign up at `GITEA_MIRROR_URL`, then in its settings enter the Gitea instance's +public URL and an access token, plus the GitHub token to mirror from. These +are stored in `gitea-mirror-data`, not in the environment. + +## Storage + +| Volume | Holds | +| --- | --- | +| `gitea-mirror-data` | Mirror job database, settings and generated secrets | diff --git a/gitea-mirror/compose.yml b/gitea-mirror/compose.yml new file mode 100644 index 0000000..6d8f08f --- /dev/null +++ b/gitea-mirror/compose.yml @@ -0,0 +1,17 @@ +services: + gitea-mirror: + image: ghcr.io/raylabshq/gitea-mirror:latest + restart: unless-stopped + pull_policy: always + environment: + BETTER_AUTH_SECRET: ${BETTER_AUTH_SECRET:?required} + ENCRYPTION_SECRET: ${ENCRYPTION_SECRET:?required} + BETTER_AUTH_URL: ${GITEA_MIRROR_URL:?required} + PUBLIC_BETTER_AUTH_URL: ${GITEA_MIRROR_URL:?required} + BETTER_AUTH_TRUSTED_ORIGINS: ${GITEA_MIRROR_URL:?required} + BUN_CONFIG_HTTP_IDLE_TIMEOUT: ${GITEA_CLONE_TIMEOUT:-3600} + volumes: + - gitea-mirror-data:/app/data + +volumes: + gitea-mirror-data: diff --git a/gitea/.env.example b/gitea/.env.example index f182af6..e445db7 100644 --- a/gitea/.env.example +++ b/gitea/.env.example @@ -1,6 +1,3 @@ POSTGRES_PASSWORD=gitea GITEA_ROOT_URL=https://gitea.example.com/ GITEA_CLONE_TIMEOUT=3600 -BETTER_AUTH_SECRET= -ENCRYPTION_SECRET= -GITEA_MIRROR_URL=https://gitea-mirror.example.com diff --git a/gitea/.gitignore b/gitea/.gitignore index d4ebb49..e9af3b3 100644 --- a/gitea/.gitignore +++ b/gitea/.gitignore @@ -1,4 +1,3 @@ gitea/ -gitea-mirror/ .env purge.py diff --git a/gitea/README.md b/gitea/README.md index 5d5ab84..6f9e820 100644 --- a/gitea/README.md +++ b/gitea/README.md @@ -1,8 +1,6 @@ # gitea -Self-hosted [Gitea](https://about.gitea.com/) backed by PostgreSQL, with -[gitea-mirror](https://github.com/RayLabsHQ/gitea-mirror) mirroring GitHub -repositories into it. +Self-hosted [Gitea](https://about.gitea.com/) backed by PostgreSQL. ## Services @@ -10,13 +8,11 @@ repositories into it. | --- | --- | --- | --- | | `db` | `postgres:16-alpine` | 5432 | none | | `gitea` | `gitea/gitea:28` | 3000 | `GITEA_ROOT_URL` | -| `gitea-mirror` | `ghcr.io/raylabshq/gitea-mirror:latest` | 4321 | `GITEA_MIRROR_URL` | -In Coolify, give `gitea` and `gitea-mirror` each a domain on their internal -port, matching the two URL variables. +In Coolify, give `gitea` a domain on port 3000 matching `GITEA_ROOT_URL`. -`gitea` waits for `db` to pass its health check. `gitea` checks -`/api/healthz`; the `gitea-mirror` image ships its own health check. +`gitea` waits for `db` to pass its health check, and checks `/api/healthz` +itself. ## Variables @@ -24,20 +20,7 @@ port, matching the two URL variables. | --- | --- | --- | | `POSTGRES_PASSWORD` | `db`, `gitea` | Defaults to `gitea`. | | `GITEA_ROOT_URL` | Gitea `server.ROOT_URL` | Public URL, with trailing slash. Gitea builds clone URLs and redirects from it. | -| `GITEA_CLONE_TIMEOUT` | Gitea `git.timeout` `MIGRATE` and `MIRROR`; gitea-mirror `BUN_CONFIG_HTTP_IDLE_TIMEOUT` | Seconds a mirror's first clone or a later fetch may run. Defaults to `3600`; at most `14340`. | -| `BETTER_AUTH_SECRET` | gitea-mirror | Signs sessions and encrypts its login keys. Generate with `openssl rand -base64 32`. | -| `ENCRYPTION_SECRET` | gitea-mirror | Encrypts the stored GitHub and Gitea tokens. Generate with `openssl rand -base64 48`. | -| `GITEA_MIRROR_URL` | `BETTER_AUTH_URL`, `PUBLIC_BETTER_AUTH_URL`, `BETTER_AUTH_TRUSTED_ORIGINS` | Public URL of the mirror UI, no trailing slash. | - -Behind a reverse proxy, gitea-mirror rejects sign-in with "invalid origin" -unless all three Better Auth variables hold the external URL, so one variable -feeds them all. - -Both secrets are set explicitly rather than left to the image, which would -otherwise generate its own into `gitea-mirror-data`. Data encrypted under one -secret is unreadable under another, so moving the data to a new deployment -means carrying the secrets with it. Never change either on an existing -install. +| `GITEA_CLONE_TIMEOUT` | Gitea `git.timeout` `MIGRATE` and `MIRROR` | Seconds a mirror's first clone or a later fetch may run. Defaults to `3600`. | ## Choices @@ -45,27 +28,18 @@ install. disabled and the UI offers HTTPS clone URLs only. - **One-hour clone timeout.** Gitea's defaults (600 s to migrate, 300 s to fetch) cut off multi-gigabyte repositories mid-clone, leaving empty mirrors - that still hold gigabytes of unreachable packfiles. -- **gitea-mirror waits as long as Gitea clones.** Gitea's migrate API sends - nothing until the clone ends, and Bun's `fetch` drops a connection idle for - 5 minutes. gitea-mirror then marks the repository failed while Gitea keeps - cloning; on retry it finds the half-made repository and marks it mirrored. - If that clone later fails, Gitea keeps an empty repository with no mirror - record, which never syncs. `BUN_CONFIG_HTTP_IDLE_TIMEOUT` takes the same - value as the clone timeout so the request outlives the clone. Bun caps it - at 239 minutes, so a larger `GITEA_CLONE_TIMEOUT` stops helping there. + that still hold gigabytes of unreachable packfiles. A client that calls the + migrate API gets no response until the clone ends, so it must wait at least + this long. - **`gitea/gitea:28`.** Gitea publishes major tags; the major pin takes updates without a surprise major upgrade. -- **`gitea-mirror:latest`** with `pull_policy: always`: upstream publishes no - major tag, so every redeploy takes the newest release. - **`postgres:16-alpine`** stays on 16: a new Postgres major cannot read the existing data directory without a dump and restore. ## Usage -Complete Gitea's first-run setup at `GITEA_ROOT_URL`, create an access token, -then configure mirroring at `GITEA_MIRROR_URL`. In gitea-mirror, set the Gitea -URL to `http://gitea:3000` so it talks to Gitea over the internal network. +Complete Gitea's first-run setup at `GITEA_ROOT_URL`, then create an access +token for any tool that mirrors into it. ## Storage @@ -73,4 +47,3 @@ URL to `http://gitea:3000` so it talks to Gitea over the internal network. | --- | --- | | `db-data` | PostgreSQL data | | `gitea-data` | Repositories, Gitea config and state | -| `gitea-mirror-data` | Mirror job database and generated secrets | diff --git a/gitea/compose.yml b/gitea/compose.yml index e420f1e..628d80d 100644 --- a/gitea/compose.yml +++ b/gitea/compose.yml @@ -39,21 +39,6 @@ services: retries: 5 start_period: 30s - gitea-mirror: - image: ghcr.io/raylabshq/gitea-mirror:latest - restart: unless-stopped - pull_policy: always - environment: - BETTER_AUTH_SECRET: ${BETTER_AUTH_SECRET:?required} - ENCRYPTION_SECRET: ${ENCRYPTION_SECRET:?required} - BETTER_AUTH_URL: ${GITEA_MIRROR_URL:?required} - PUBLIC_BETTER_AUTH_URL: ${GITEA_MIRROR_URL:?required} - BETTER_AUTH_TRUSTED_ORIGINS: ${GITEA_MIRROR_URL:?required} - BUN_CONFIG_HTTP_IDLE_TIMEOUT: ${GITEA_CLONE_TIMEOUT:-3600} - volumes: - - gitea-mirror-data:/app/data - volumes: db-data: gitea-data: - gitea-mirror-data: