docs: move reference material from the README into docs

The README now covers what keepalive is, a quick start, the adapter table,
and links. Reference material moves to docs/: configuration (every option,
per-adapter keys, validation), deployment (Compose, Coolify, Docker, source),
how it works (writes, retries, timeouts, logging), known issues (PostgreSQL
public-schema permissions and Aiven's ACL key-pattern tilde, by log line),
and adding an adapter (now including ConfigKeys registration).

.gitignore also excludes config.yaml, matching .dockerignore.
This commit is contained in:
tiennm99 committed 2026-10-09 12:37:52 +07:00
1 parent 901b4f7b03
commit e445b5cbc5
7 files changed
+332 -118

No files matched your search

+1
View File
@@ -6,3 +6,4 @@ ca.pem
*.test
*.out
config.yml
config.yaml
+37 -118
View File
@@ -1,150 +1,69 @@
# keepalive
Pluggable Go daemon that periodically touches external services to prevent idle shutdowns, pauses, or cold starts.
**Stop free-tier databases from pausing.** keepalive is a tiny Go daemon that writes to each of your hosted datastores on a schedule, so idle-shutdown policies never trigger.
The current adapters perform cheap datastore writes for Redis Cloud, Valkey, Aiven, Neon, Supabase, MongoDB Atlas, Couchbase Capella, and similar hosted services.
- **One config, many services.** Keep Redis, PostgreSQL, MySQL, MongoDB, and Couchbase instances alive from a single process.
- **Cheapest possible write.** Each tick increments one counter; nothing else in your database is touched.
- **Self-healing.** Failed services retry every minute and recreate their counter if it disappears, without affecting the others.
- **Small and safe.** A static, distroless, non-root image with no ports, and connection strings are kept out of logs.
Successor to the `*-keepalive` family: one binary, one image, five datastore adapters. Valkey and other Redis-compatible stores use the `redis` adapter.
Works with Redis Cloud, Upstash, Aiven (PostgreSQL, MySQL, Valkey), Supabase, YugabyteDB, CockroachDB, MongoDB Atlas, Couchbase Capella, and any self-hosted Redis-, Valkey-, PostgreSQL-, MySQL-, MongoDB-, or Couchbase-compatible server.
## Configuration
## Quick start
By default, keepalive reads the first config file it finds: `config.yml`, `config.yaml`, `/config.yml`, then `/config.yaml`. One deployment can keep any number of services alive.
Write a `config.yml`:
```yaml
# Default interval for every service.
interval: 1m
counter_key: counter
interval: 1h
services:
- adapter: redis
config:
url: redis://default@redis-a.example.com:6379
namespace: keepalive
# Valkey, Dragonfly, KeyDB and other Redis-compatible stores use the redis
# adapter with a redis:// or rediss:// (TLS) URL.
- name: valkey-a
- name: upstash
adapter: redis
# One service can override the global interval and counter key.
interval: 30s
counter_key: valkey-counter
config:
url: rediss://default@valkey-a.example.com:6379
namespace: keepalive
url: rediss://default:PASSWORD@example.upstash.io:6379
- adapter: postgresql
- name: supabase
adapter: postgresql
config:
url: postgresql://user:pass@postgres-a.example.com:5432/keepalive?sslmode=require
url: postgresql://USER:PASSWORD@aws-0-region.pooler.supabase.com:5432/postgres?sslmode=require
- adapter: mysql
- name: atlas
adapter: mongodb
config:
dsn: user:pass@tcp(mysql-a.example.com:3306)/keepalive
- adapter: mongodb
config:
uri: mongodb+srv://user:pass@mongo-a.example.com
uri: mongodb+srv://USER:PASSWORD@cluster0.example.mongodb.net
database: keepalive
collection: counter
- adapter: couchbase
config:
connection_string: couchbases://couchbase-a.example.com
username: user
password: pass
bucket_name: keepalive
scope_name: _default
collection_name: _default
```
`name` is optional. When omitted, keepalive generates a name from `adapter` and the connection host, such as `redis-redis-a-example-com`. Duplicate generated names get the first free suffix, like `redis-redis-a-example-com-2`.
`interval` at the root sets the default schedule for every service and defaults to `1m`.
`interval` inside a service overrides that default only for that service.
In the example above, every service runs every `1m` except `valkey-a`, which runs every `30s`.
Interval values use Go duration syntax, for example `30s`, `5m`, `1h`, `1h30m`, or `1.5h`. Plain integers are treated as seconds, so `90` means `90s`.
`counter_key` at the root sets the default counter key for every service and defaults to `counter`.
`counter_key` inside a service overrides that default only for that service.
In the example above, every service writes `counter` except `valkey-a`, which writes `keepalive:valkey-counter`.
## Supported adapters
| `adapter` | Driver | `config` keys |
| ------------ | ----------------------------------- | ------------- |
| `redis` | `github.com/redis/go-redis/v9` | `url` (`redis://` or `rediss://`), optional `namespace` |
| `postgresql` | `github.com/lib/pq` | `url` |
| `mysql` | `github.com/go-sql-driver/mysql` | `dsn` |
| `mongodb` | `go.mongodb.org/mongo-driver/v2` | `uri`, `database`, `collection` |
| `couchbase` | `github.com/couchbase/gocb/v2` | `connection_string`, `username`, `password`, `bucket_name`, `scope_name`, `collection_name`, optional `ready_timeout`, optional `bucket_ram_quota_mb` |
## Quick start (Compose)
Run it:
```bash
cp config.example.yml config.yml
git clone https://github.com/tiennm99/keepalive && cd keepalive
# put your config.yml here (see config.example.yml for every adapter)
docker compose up -d --build
docker compose logs -f
```
`compose.yml` also deploys on Coolify with the Docker Compose build pack (compose file `/compose.yml`). Coolify turns the `./config.yml` bind mount into an editable file storage; paste your config there. keepalive is a background worker with no port, so leave the service without a domain. Keep the real `config.yml` out of git, since it holds datastore credentials. Create `config.yml` before the first start: if the file is missing, Docker mounts an empty directory in its place and keepalive exits with an error. The container runs as user `65532`, so the file must be readable by that user (for example mode `0644`).
Each service logs a line like `[upstash] counter: 42` on every tick.
## Quick start (Docker)
## Adapters
```bash
docker build -t keepalive:local .
docker run -d --name keepalive --restart unless-stopped \
-v "$PWD/config.yml:/config.yml:ro" \
keepalive:local
```
| `adapter` | Use it for |
| ------------------------- | ----------------------------------------------------------------- |
| `redis` | Redis, Valkey, Dragonfly, KeyDB, Garnet, Upstash, Redis Cloud |
| `postgresql` / `postgres` | PostgreSQL, Supabase, Aiven, YugabyteDB, CockroachDB |
| `mysql` | MySQL, MariaDB, Aiven for MySQL, TiDB |
| `mongodb` / `mongo` | MongoDB Atlas, Azure DocumentDB, FerretDB, Cosmos DB for MongoDB |
| `couchbase` | Couchbase Capella and self-hosted Couchbase |
If you prefer to mount the config into a working directory instead of the container root, set the container working directory and mount the file there:
## Documentation
```bash
docker run -d --name keepalive --restart unless-stopped \
--workdir /workspace \
-v "$PWD/config.yml:/workspace/config.yml:ro" \
keepalive:local
```
## Quick start (local)
```bash
git clone https://github.com/tiennm99/keepalive
cd keepalive
cp config.example.yml config.yml
go run .
```
## How it works
On startup each adapter initializes the minimum resource it owns, then every tick performs the cheapest write that proves the cluster is alive. `counter_key` selects the key/doc ID and defaults to `counter`.
- **Redis** (also Valkey, Dragonfly, KeyDB, Garnet, Upstash) — initialize with `SETNX key 0`, then `INCR key`. When `namespace` is empty, the key is `counter`; when `namespace: keepalive`, the key is `keepalive:counter`.
- **PostgreSQL** — `CREATE TABLE IF NOT EXISTS keepalive`, seed `key`, then `UPDATE ... RETURNING`
- **MySQL** — `CREATE TABLE IF NOT EXISTS keepalive`, seed `key`, then `UPDATE` + `SELECT`
- **MongoDB** — upsert `{_id: key, count: 0}` on connect, then `FindOneAndUpdate({_id: key}, {$inc: {count: 1}}, upsert)`
- **Couchbase** — optionally create the bucket when `bucket_ram_quota_mb` is set, create configured scope/collection when missing, insert `key = 0` if missing, then an atomic binary `INCREMENT key`
Each configured service starts independently and writes once right after connecting, then once per `interval`. A connect attempt, including initialization, gives up after 1 minute; PostgreSQL URLs without `connect_timeout` get `connect_timeout=30`. When a connect or a tick fails, the service logs the error, closes its connection, and reconnects after 1 minute, which also re-runs initialization (for example, recreating a dropped table). Other services in the same deployment keep running. An unknown `adapter`, a missing required `config` key, or `counter_key` placed inside `config` stops keepalive at startup. Unknown keys elsewhere in the file, such as a misspelled `interval`, only log a warning.
For hosted Couchbase/Capella clusters, `ready_timeout` defaults to `30s`. If Couchbase reports `CONNECTION_ERROR`, check the connection string, bucket name, database user permissions, and Capella allowed IP/network access.
## Adding a new adapter
1. Create `adapter/<name>.go`.
2. Implement the `Adapter` interface in `adapter/adapter.go` (`Connect`, `Increment`, `Close`).
3. Register the factory in `init()`:
```go
func init() { Registry["<name>"] = func(cfg Config) (Adapter, error) { return &myAdapter{}, nil } }
```
4. Add an `import _ "your driver"` if needed, and the adapter config keys to `config.example.yml` and the table above.
## Migrated from
This repo replaces six single-datastore repos (`redis-keepalive`, `valkey-keepalive`,
`postgresql-keepalive`, `mysql-keepalive`, `mongodb-keepalive`, `couchbase-keepalive`).
Their full histories were absorbed into this repository — browse earlier commits to find
each implementation under its own subfolder (`redis/`, `valkey/`, `postgresql/`, `mysql/`,
`mongodb/`, `couchbase/`).
- [Configuration](docs/configuration.md): every option, per-adapter keys, and validation rules.
- [Deployment](docs/deployment.md): Docker Compose, Coolify, plain Docker, and running from source.
- [How it works](docs/how-it-works.md): what each adapter writes, retries, timeouts, and shutdown.
- [Known issues](docs/known-issues.md): permission errors and other setup problems, by log line.
- [Adding an adapter](docs/adding-an-adapter.md): plug in a new datastore.
## License
Apache-2.0 — see [LICENSE](LICENSE).
Apache-2.0. See [LICENSE](LICENSE).
+41
View File
@@ -0,0 +1,41 @@
# Adding an adapter
An adapter connects to one datastore, increments a counter on every tick, and releases its resources on close. Adapters live in the `adapter` package, one file each.
1. **Create `adapter/<name>.go`** and implement the `Adapter` interface from [`adapter/adapter.go`](../adapter/adapter.go):
```go
type Adapter interface {
Connect(ctx context.Context) error // open the client and create the counter if missing
Increment(ctx context.Context) (int64, error) // add 1 and return the new value
Close(ctx context.Context) error // release resources; called after every session
}
```
2. **Register the factory and its config keys** in `init()`. The factory only reads config and must not do network I/O, because keepalive calls it at startup to validate the file.
```go
func init() {
ConfigKeys["<name>"] = []string{"url"}
Registry["<name>"] = func(cfg Config) (Adapter, error) {
url, err := cfg.Required("url")
if err != nil {
return nil, err
}
return &myAdapter{url: url, key: cfg.Optional("counter_key", "counter")}, nil
}
}
```
`ConfigKeys` drives the unknown-key warnings; a test fails if an adapter is registered without it. `counter_key` is supplied by the config loader and is not listed.
3. **Follow the conventions** that keep the runner's guarantees:
- Honour `ctx` in `Connect` and `Increment`; the runner bounds both.
- Store the client on the adapter only after `Connect` succeeds, so `Close` never closes a failed client twice.
- Make `Connect` idempotent: it re-runs after every failure and should recreate a missing counter.
- Implement `ConnectTimeout() time.Duration` only if setup legitimately needs longer than 1 minute.
- Keep the driver pure Go (`CGO_ENABLED=0`).
4. **Document it:** add the adapter to [`config.example.yml`](../config.example.yml), its keys to [Configuration](configuration.md), its writes to [How it works](how-it-works.md), and a row to the README's adapter table.
5. **Test it:** `go vet ./...` and `go test -race ./...`.
+106
View File
@@ -0,0 +1,106 @@
# Configuration
keepalive reads one YAML file: the first that exists of `config.yml`, `config.yaml`, `/config.yml`, then `/config.yaml`. The app reads no environment variables. [`config.example.yml`](../config.example.yml) shows every adapter.
The file holds datastore credentials, so keep it out of git (`.gitignore` and `.dockerignore` already exclude `config.yml` and `config.yaml`).
## Structure
```yaml
interval: 1m # optional, default 1m
counter_key: counter # optional, default counter
services:
- name: cache-a # optional
adapter: redis # required
interval: 30s # optional, overrides the root interval
counter_key: hits # optional, overrides the root counter_key
config: # adapter-specific keys, see below
url: rediss://default:PASSWORD@cache-a.example.com:6379
namespace: keepalive
```
| Key | Where | Meaning |
| ------------- | ----------------- | ------- |
| `interval` | root, service | Time between writes. Go duration syntax (`30s`, `5m`, `1h30m`, `1.5h`); a plain integer means seconds (`90` = `90s`). Must be greater than zero. Default `1m`. |
| `counter_key` | root, service | Key, row, or document ID the adapter increments. Default `counter`. Set it at the root or on a service, never inside `config`. |
| `name` | service | Log prefix for the service. Must be unique. When omitted, keepalive builds one from the adapter and host, such as `redis-cache-a-example-com`, adding `-2`, `-3`, and so on when it is taken. |
| `adapter` | service | One of the adapters below. |
| `config` | service | Connection settings for that adapter. |
Pick an interval well inside the provider's idle window. The default `1m` suits everything; hourly or daily writes are enough for policies measured in days.
## Adapter keys
### `redis`
Covers every server that speaks the Redis protocol: Redis, Valkey, Dragonfly, KeyDB, Garnet, and hosted variants.
| Key | Required | Meaning |
| ----------- | -------- | ------- |
| `url` | yes | `redis://` or `rediss://` (TLS) URL, with optional user, password, `/db`, and [go-redis query options](https://pkg.go.dev/github.com/redis/go-redis/v9#ParseURL). `valkey://` is not accepted; use `redis://` or `rediss://`. |
| `namespace` | no | Key prefix. With `namespace: keepalive`, the key is `keepalive:counter`. |
Cluster and Sentinel endpoints are not supported; point `url` at a single node or a proxy endpoint.
### `postgresql` (alias `postgres`)
| Key | Required | Meaning |
| ----- | -------- | ------- |
| `url` | yes | `postgres://` or `postgresql://` URL, or a key=value DSN (`host=... user=... dbname=...`). Without `connect_timeout`, keepalive adds `connect_timeout=30`. |
The user must be able to create a table in the database's `public` schema; see [Known issues](known-issues.md#postgresql-permission-denied-for-schema-public).
For Supabase, use the session pooler connection string (port `5432`) with `sslmode=require`; the direct host is IPv6-only.
### `mysql`
| Key | Required | Meaning |
| ----- | -------- | ------- |
| `dsn` | yes | [go-sql-driver DSN](https://github.com/go-sql-driver/mysql#dsn-data-source-name), for example `user:pass@tcp(host:3306)/keepalive?tls=true`. |
### `mongodb` (alias `mongo`)
| Key | Required | Meaning |
| ------------ | -------- | ------- |
| `uri` | yes | `mongodb://` or `mongodb+srv://` connection string. |
| `database` | yes | Database holding the counter. |
| `collection` | yes | Collection holding the counter document. |
The driver needs MongoDB 4.2 or later. Cosmos DB and AWS DocumentDB need `retrywrites=false` in the URI.
### `couchbase`
| Key | Required | Meaning |
| --------------------- | -------- | ------- |
| `connection_string` | yes | `couchbase://` or `couchbases://` (TLS) address. |
| `username` | yes | Database user. |
| `password` | yes | Database user's password. |
| `bucket_name` | yes | Bucket holding the counter. |
| `scope_name` | yes | Scope; created when missing. Use `_default` for the default scope. |
| `collection_name` | yes | Collection; created when missing. Use `_default` for the default collection. |
| `ready_timeout` | no | How long to wait for the bucket and each setup step. Duration or seconds. Default `30s`. |
| `bucket_ram_quota_mb` | no | When set, create the bucket with this RAM quota if it does not exist. Needs a user allowed to create buckets. |
## Validation
keepalive checks the whole file at startup.
Startup fails, with the offending `services[N]` path in the error, when:
- the file is missing, is a directory, or is not valid YAML;
- `services` is empty;
- a service has no `adapter`, or names an unknown one;
- a required `config` key is missing;
- `counter_key` appears inside `config`;
- an `interval` is not a positive duration;
- two services share a `name`.
Keys the schema does not define only log a warning, and keepalive ignores them:
```
warning: /config.yml: line 1: field intervall not found in type main.appConfig; ignoring it
warning: /config.yml: services[0].config: unknown key "namspace" for adapter redis (known: url, namespace); ignoring it
```
Warnings name the key, never its value.
+62
View File
@@ -0,0 +1,62 @@
# Deployment
keepalive is a background worker: it opens no port, serves no HTTP, and needs only outbound network access to your datastores. Every option lives in the [config file](configuration.md).
The image is built from the repository's [`Dockerfile`](../Dockerfile): a static `CGO_ENABLED=0` binary on `gcr.io/distroless/static-debian12:nonroot`, running as user `65532`.
## Docker Compose
```bash
cp config.example.yml config.yml # then edit it
docker compose up -d --build
docker compose logs -f
```
[`compose.yml`](../compose.yml) mounts `./config.yml` read-only at `/config.yml` and restarts the container unless you stop it.
Before the first start:
- **Create `config.yml` first.** If it is missing, Docker mounts an empty directory in its place and keepalive exits with `/config.yml is a directory, not a config file`.
- **Make it readable by user `65532`.** A file with mode `0600` owned by your user cannot be read inside the container; `0644` works.
After editing `config.yml`, restart the service (`docker compose restart`); keepalive reads its config only at startup.
## Coolify
1. Create an application from the repository with the **Docker Compose** build pack and compose file `/compose.yml`.
2. Coolify turns the `./config.yml` bind mount into an editable file storage. Paste your config there.
3. Leave the service without a domain; keepalive has no port and no healthcheck.
4. Deploy, then restart after any config change.
## Docker
```bash
docker build -t keepalive:local .
docker run -d --name keepalive --restart unless-stopped \
-v "$PWD/config.yml:/config.yml:ro" \
keepalive:local
```
To mount the config into a working directory instead of the container root:
```bash
docker run -d --name keepalive --restart unless-stopped \
--workdir /workspace \
-v "$PWD/config.yml:/workspace/config.yml:ro" \
keepalive:local
```
## From source
Requires Go 1.26 or later.
```bash
git clone https://github.com/tiennm99/keepalive
cd keepalive
cp config.example.yml config.yml # then edit it
go run .
```
## Stopping
On `SIGTERM` or `SIGINT`, keepalive stops every service and exits within 7 seconds, inside Docker's default 10-second stop grace period.
+40
View File
@@ -0,0 +1,40 @@
# How it works
Each configured service runs in its own goroutine and goes through the same loop:
1. **Connect.** Open a client and create the minimum resource the adapter owns (a key, table row, or document). One attempt, including this setup, has 1 minute (Couchbase: `ready_timeout` plus 1 minute).
2. **Write now.** Increment the counter once right away, so every start or restart writes even when `interval` is long.
3. **Write on schedule.** Increment once per `interval`. Each write has 3 seconds and logs `[name] counter: N`.
4. **Recover.** When a connect or a write fails, log one line, close the connection, wait 1 minute, and go back to step 1. Reconnecting re-runs setup, so a dropped table, row, key, or collection is recreated.
Services are independent: one failing service never stops or slows the others.
## What each adapter writes
`key` below is the service's `counter_key` (default `counter`).
| Adapter | On connect | On each tick |
| ------------ | ---------- | ------------ |
| `redis` | `PING`, then `SETNX key 0` (key is `namespace:key` when `namespace` is set) | `INCR key` |
| `postgresql` | `CREATE TABLE IF NOT EXISTS keepalive (key TEXT PRIMARY KEY, value BIGINT)`, then insert `key` with `0` if missing | `UPDATE keepalive SET value = value + 1 WHERE key = $1 RETURNING value` |
| `mysql` | `CREATE TABLE IF NOT EXISTS keepalive`, then `INSERT IGNORE` `key` with `0` | `UPDATE` then `SELECT` the value, in one transaction |
| `mongodb` | Upsert `{_id: key, count: 0}` | `FindOneAndUpdate({_id: key}, {$inc: {count: 1}})` with upsert |
| `couchbase` | Create the bucket when `bucket_ram_quota_mb` is set, create the scope and collection when missing, insert `key = 0` if missing | Atomic binary increment of `key` |
## Timeouts
| Step | Limit |
| ----------------------- | ----- |
| Connect and setup | 1 minute; Couchbase gets `ready_timeout` plus 1 minute |
| PostgreSQL handshake | `connect_timeout=30` seconds, added to the connection string unless you set one |
| One write | 3 seconds |
| Retry after a failure | 1 minute |
| Shutdown | 7 seconds, then keepalive exits |
## Logging
keepalive logs one line per write and one line per failure, prefixed with the service name. Connection strings never reach the logs: errors from a malformed URL or DSN are replaced with a generic hint, and the generated service name uses only the host. Driver-internal logging from go-redis is silenced, since keepalive already reports each failure once.
## History
This repository replaces six single-datastore repositories (`redis-keepalive`, `valkey-keepalive`, `postgresql-keepalive`, `mysql-keepalive`, `mongodb-keepalive`, `couchbase-keepalive`). Their histories were absorbed here; earlier commits keep each implementation under its own subfolder.
+45
View File
@@ -0,0 +1,45 @@
# Known issues
Problems seen when pointing keepalive at hosted datastores, listed by the log line they produce. keepalive creates its table, key, or document on first connect, so a dedicated keepalive user needs write access plus permission to create that object. Until the cause is fixed, the service logs the error and retries every minute; no restart is needed after the fix.
## PostgreSQL: `permission denied for schema public`
```
[aiven-postgres] connect: pq: permission denied for schema public at position 2:28 (42501); retrying in 1m0s
```
**Cause.** On connect, keepalive runs `CREATE TABLE IF NOT EXISTS keepalive` in the `public` schema. Since PostgreSQL 15, only the database owner can create objects in `public` by default, so a non-owner user is refused. PostgreSQL checks this permission even when the table already exists.
**Fix.** Give keepalive its own database and make its user the owner. Run this as an admin user (`avnadmin` on Aiven):
```sql
CREATE DATABASE keepalive OWNER keepalive; -- new database
ALTER DATABASE keepalive OWNER TO keepalive; -- existing database
```
Then point `url` at that database, for example `postgres://keepalive:...@host:port/keepalive?sslmode=require`.
On Aiven for PostgreSQL, changing the owner was the fix that worked; `GRANT CREATE ON SCHEMA public TO keepalive` did not. If you try the grant anyway, note that it applies only inside the database it runs in, so a SQL editor connected to `defaultdb` does not change the database in `url`.
## Redis / Valkey: `NOPERM No permissions to access a key`
```
[aiven-valkey] connect: NOPERM No permissions to access a key; retrying in 1m0s
```
**Cause.** The user's ACL key pattern does not cover the counter key. keepalive writes `<namespace>:<counter_key>` when `namespace` is set, otherwise `<counter_key>` (default `counter`).
On Aiven for Valkey, a common trigger is typing the key pattern with a leading `~`. Aiven adds the `~` itself, so `~keepalive:*` becomes a pattern that only matches keys starting with a literal `~`, and every write is refused.
**Fix.** Allow the commands `PING`, `SETNX`, and `INCR`, and set a key pattern that covers the counter key. On Aiven, enter the keys field without the `~`:
| Field | Value |
| ---------- | ------------------------------------------------------- |
| Categories | `+@all -@admin -@dangerous` (or `+@connection +@string`) |
| Keys | `keepalive:*` (matching `namespace: keepalive`) |
## Couchbase: `CONNECTION_ERROR` or bucket not ready
**Cause.** The cluster is unreachable or the bucket cannot be opened with the configured user.
**Fix.** Check the connection string, `bucket_name`, the database user's permissions, and Capella's allowed IP/network access. If the cluster or bucket was just created, raise `ready_timeout`; keepalive allows `ready_timeout` plus 1 minute for each connect attempt.