mirror of
https://github.com/tiennm99/keepalive.git
synced 2026-10-11 12:19:02 +00:00
docs: move reference material from the README into docs
The README now covers what keepalive is, a quick start, the adapter table, and links. Reference material moves to docs/: configuration (every option, per-adapter keys, validation), deployment (Compose, Coolify, Docker, source), how it works (writes, retries, timeouts, logging), known issues (PostgreSQL public-schema permissions and Aiven's ACL key-pattern tilde, by log line), and adding an adapter (now including ConfigKeys registration). .gitignore also excludes config.yaml, matching .dockerignore.
This commit is contained in:
tiennm99 committed
1 parent
901b4f7b03
commit
e445b5cbc5
7 files changed
+332
-118
No files matched your search
@@ -0,0 +1,41 @@
|
||||
# Adding an adapter
|
||||
|
||||
An adapter connects to one datastore, increments a counter on every tick, and releases its resources on close. Adapters live in the `adapter` package, one file each.
|
||||
|
||||
1. **Create `adapter/<name>.go`** and implement the `Adapter` interface from [`adapter/adapter.go`](../adapter/adapter.go):
|
||||
|
||||
```go
|
||||
type Adapter interface {
|
||||
Connect(ctx context.Context) error // open the client and create the counter if missing
|
||||
Increment(ctx context.Context) (int64, error) // add 1 and return the new value
|
||||
Close(ctx context.Context) error // release resources; called after every session
|
||||
}
|
||||
```
|
||||
|
||||
2. **Register the factory and its config keys** in `init()`. The factory only reads config and must not do network I/O, because keepalive calls it at startup to validate the file.
|
||||
|
||||
```go
|
||||
func init() {
|
||||
ConfigKeys["<name>"] = []string{"url"}
|
||||
Registry["<name>"] = func(cfg Config) (Adapter, error) {
|
||||
url, err := cfg.Required("url")
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return &myAdapter{url: url, key: cfg.Optional("counter_key", "counter")}, nil
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
`ConfigKeys` drives the unknown-key warnings; a test fails if an adapter is registered without it. `counter_key` is supplied by the config loader and is not listed.
|
||||
|
||||
3. **Follow the conventions** that keep the runner's guarantees:
|
||||
- Honour `ctx` in `Connect` and `Increment`; the runner bounds both.
|
||||
- Store the client on the adapter only after `Connect` succeeds, so `Close` never closes a failed client twice.
|
||||
- Make `Connect` idempotent: it re-runs after every failure and should recreate a missing counter.
|
||||
- Implement `ConnectTimeout() time.Duration` only if setup legitimately needs longer than 1 minute.
|
||||
- Keep the driver pure Go (`CGO_ENABLED=0`).
|
||||
|
||||
4. **Document it:** add the adapter to [`config.example.yml`](../config.example.yml), its keys to [Configuration](configuration.md), its writes to [How it works](how-it-works.md), and a row to the README's adapter table.
|
||||
|
||||
5. **Test it:** `go vet ./...` and `go test -race ./...`.
|
||||
@@ -0,0 +1,106 @@
|
||||
# Configuration
|
||||
|
||||
keepalive reads one YAML file: the first that exists of `config.yml`, `config.yaml`, `/config.yml`, then `/config.yaml`. The app reads no environment variables. [`config.example.yml`](../config.example.yml) shows every adapter.
|
||||
|
||||
The file holds datastore credentials, so keep it out of git (`.gitignore` and `.dockerignore` already exclude `config.yml` and `config.yaml`).
|
||||
|
||||
## Structure
|
||||
|
||||
```yaml
|
||||
interval: 1m # optional, default 1m
|
||||
counter_key: counter # optional, default counter
|
||||
|
||||
services:
|
||||
- name: cache-a # optional
|
||||
adapter: redis # required
|
||||
interval: 30s # optional, overrides the root interval
|
||||
counter_key: hits # optional, overrides the root counter_key
|
||||
config: # adapter-specific keys, see below
|
||||
url: rediss://default:PASSWORD@cache-a.example.com:6379
|
||||
namespace: keepalive
|
||||
```
|
||||
|
||||
| Key | Where | Meaning |
|
||||
| ------------- | ----------------- | ------- |
|
||||
| `interval` | root, service | Time between writes. Go duration syntax (`30s`, `5m`, `1h30m`, `1.5h`); a plain integer means seconds (`90` = `90s`). Must be greater than zero. Default `1m`. |
|
||||
| `counter_key` | root, service | Key, row, or document ID the adapter increments. Default `counter`. Set it at the root or on a service, never inside `config`. |
|
||||
| `name` | service | Log prefix for the service. Must be unique. When omitted, keepalive builds one from the adapter and host, such as `redis-cache-a-example-com`, adding `-2`, `-3`, and so on when it is taken. |
|
||||
| `adapter` | service | One of the adapters below. |
|
||||
| `config` | service | Connection settings for that adapter. |
|
||||
|
||||
Pick an interval well inside the provider's idle window. The default `1m` suits everything; hourly or daily writes are enough for policies measured in days.
|
||||
|
||||
## Adapter keys
|
||||
|
||||
### `redis`
|
||||
|
||||
Covers every server that speaks the Redis protocol: Redis, Valkey, Dragonfly, KeyDB, Garnet, and hosted variants.
|
||||
|
||||
| Key | Required | Meaning |
|
||||
| ----------- | -------- | ------- |
|
||||
| `url` | yes | `redis://` or `rediss://` (TLS) URL, with optional user, password, `/db`, and [go-redis query options](https://pkg.go.dev/github.com/redis/go-redis/v9#ParseURL). `valkey://` is not accepted; use `redis://` or `rediss://`. |
|
||||
| `namespace` | no | Key prefix. With `namespace: keepalive`, the key is `keepalive:counter`. |
|
||||
|
||||
Cluster and Sentinel endpoints are not supported; point `url` at a single node or a proxy endpoint.
|
||||
|
||||
### `postgresql` (alias `postgres`)
|
||||
|
||||
| Key | Required | Meaning |
|
||||
| ----- | -------- | ------- |
|
||||
| `url` | yes | `postgres://` or `postgresql://` URL, or a key=value DSN (`host=... user=... dbname=...`). Without `connect_timeout`, keepalive adds `connect_timeout=30`. |
|
||||
|
||||
The user must be able to create a table in the database's `public` schema; see [Known issues](known-issues.md#postgresql-permission-denied-for-schema-public).
|
||||
|
||||
For Supabase, use the session pooler connection string (port `5432`) with `sslmode=require`; the direct host is IPv6-only.
|
||||
|
||||
### `mysql`
|
||||
|
||||
| Key | Required | Meaning |
|
||||
| ----- | -------- | ------- |
|
||||
| `dsn` | yes | [go-sql-driver DSN](https://github.com/go-sql-driver/mysql#dsn-data-source-name), for example `user:pass@tcp(host:3306)/keepalive?tls=true`. |
|
||||
|
||||
### `mongodb` (alias `mongo`)
|
||||
|
||||
| Key | Required | Meaning |
|
||||
| ------------ | -------- | ------- |
|
||||
| `uri` | yes | `mongodb://` or `mongodb+srv://` connection string. |
|
||||
| `database` | yes | Database holding the counter. |
|
||||
| `collection` | yes | Collection holding the counter document. |
|
||||
|
||||
The driver needs MongoDB 4.2 or later. Cosmos DB and AWS DocumentDB need `retrywrites=false` in the URI.
|
||||
|
||||
### `couchbase`
|
||||
|
||||
| Key | Required | Meaning |
|
||||
| --------------------- | -------- | ------- |
|
||||
| `connection_string` | yes | `couchbase://` or `couchbases://` (TLS) address. |
|
||||
| `username` | yes | Database user. |
|
||||
| `password` | yes | Database user's password. |
|
||||
| `bucket_name` | yes | Bucket holding the counter. |
|
||||
| `scope_name` | yes | Scope; created when missing. Use `_default` for the default scope. |
|
||||
| `collection_name` | yes | Collection; created when missing. Use `_default` for the default collection. |
|
||||
| `ready_timeout` | no | How long to wait for the bucket and each setup step. Duration or seconds. Default `30s`. |
|
||||
| `bucket_ram_quota_mb` | no | When set, create the bucket with this RAM quota if it does not exist. Needs a user allowed to create buckets. |
|
||||
|
||||
## Validation
|
||||
|
||||
keepalive checks the whole file at startup.
|
||||
|
||||
Startup fails, with the offending `services[N]` path in the error, when:
|
||||
|
||||
- the file is missing, is a directory, or is not valid YAML;
|
||||
- `services` is empty;
|
||||
- a service has no `adapter`, or names an unknown one;
|
||||
- a required `config` key is missing;
|
||||
- `counter_key` appears inside `config`;
|
||||
- an `interval` is not a positive duration;
|
||||
- two services share a `name`.
|
||||
|
||||
Keys the schema does not define only log a warning, and keepalive ignores them:
|
||||
|
||||
```
|
||||
warning: /config.yml: line 1: field intervall not found in type main.appConfig; ignoring it
|
||||
warning: /config.yml: services[0].config: unknown key "namspace" for adapter redis (known: url, namespace); ignoring it
|
||||
```
|
||||
|
||||
Warnings name the key, never its value.
|
||||
@@ -0,0 +1,62 @@
|
||||
# Deployment
|
||||
|
||||
keepalive is a background worker: it opens no port, serves no HTTP, and needs only outbound network access to your datastores. Every option lives in the [config file](configuration.md).
|
||||
|
||||
The image is built from the repository's [`Dockerfile`](../Dockerfile): a static `CGO_ENABLED=0` binary on `gcr.io/distroless/static-debian12:nonroot`, running as user `65532`.
|
||||
|
||||
## Docker Compose
|
||||
|
||||
```bash
|
||||
cp config.example.yml config.yml # then edit it
|
||||
docker compose up -d --build
|
||||
docker compose logs -f
|
||||
```
|
||||
|
||||
[`compose.yml`](../compose.yml) mounts `./config.yml` read-only at `/config.yml` and restarts the container unless you stop it.
|
||||
|
||||
Before the first start:
|
||||
|
||||
- **Create `config.yml` first.** If it is missing, Docker mounts an empty directory in its place and keepalive exits with `/config.yml is a directory, not a config file`.
|
||||
- **Make it readable by user `65532`.** A file with mode `0600` owned by your user cannot be read inside the container; `0644` works.
|
||||
|
||||
After editing `config.yml`, restart the service (`docker compose restart`); keepalive reads its config only at startup.
|
||||
|
||||
## Coolify
|
||||
|
||||
1. Create an application from the repository with the **Docker Compose** build pack and compose file `/compose.yml`.
|
||||
2. Coolify turns the `./config.yml` bind mount into an editable file storage. Paste your config there.
|
||||
3. Leave the service without a domain; keepalive has no port and no healthcheck.
|
||||
4. Deploy, then restart after any config change.
|
||||
|
||||
## Docker
|
||||
|
||||
```bash
|
||||
docker build -t keepalive:local .
|
||||
docker run -d --name keepalive --restart unless-stopped \
|
||||
-v "$PWD/config.yml:/config.yml:ro" \
|
||||
keepalive:local
|
||||
```
|
||||
|
||||
To mount the config into a working directory instead of the container root:
|
||||
|
||||
```bash
|
||||
docker run -d --name keepalive --restart unless-stopped \
|
||||
--workdir /workspace \
|
||||
-v "$PWD/config.yml:/workspace/config.yml:ro" \
|
||||
keepalive:local
|
||||
```
|
||||
|
||||
## From source
|
||||
|
||||
Requires Go 1.26 or later.
|
||||
|
||||
```bash
|
||||
git clone https://github.com/tiennm99/keepalive
|
||||
cd keepalive
|
||||
cp config.example.yml config.yml # then edit it
|
||||
go run .
|
||||
```
|
||||
|
||||
## Stopping
|
||||
|
||||
On `SIGTERM` or `SIGINT`, keepalive stops every service and exits within 7 seconds, inside Docker's default 10-second stop grace period.
|
||||
@@ -0,0 +1,40 @@
|
||||
# How it works
|
||||
|
||||
Each configured service runs in its own goroutine and goes through the same loop:
|
||||
|
||||
1. **Connect.** Open a client and create the minimum resource the adapter owns (a key, table row, or document). One attempt, including this setup, has 1 minute (Couchbase: `ready_timeout` plus 1 minute).
|
||||
2. **Write now.** Increment the counter once right away, so every start or restart writes even when `interval` is long.
|
||||
3. **Write on schedule.** Increment once per `interval`. Each write has 3 seconds and logs `[name] counter: N`.
|
||||
4. **Recover.** When a connect or a write fails, log one line, close the connection, wait 1 minute, and go back to step 1. Reconnecting re-runs setup, so a dropped table, row, key, or collection is recreated.
|
||||
|
||||
Services are independent: one failing service never stops or slows the others.
|
||||
|
||||
## What each adapter writes
|
||||
|
||||
`key` below is the service's `counter_key` (default `counter`).
|
||||
|
||||
| Adapter | On connect | On each tick |
|
||||
| ------------ | ---------- | ------------ |
|
||||
| `redis` | `PING`, then `SETNX key 0` (key is `namespace:key` when `namespace` is set) | `INCR key` |
|
||||
| `postgresql` | `CREATE TABLE IF NOT EXISTS keepalive (key TEXT PRIMARY KEY, value BIGINT)`, then insert `key` with `0` if missing | `UPDATE keepalive SET value = value + 1 WHERE key = $1 RETURNING value` |
|
||||
| `mysql` | `CREATE TABLE IF NOT EXISTS keepalive`, then `INSERT IGNORE` `key` with `0` | `UPDATE` then `SELECT` the value, in one transaction |
|
||||
| `mongodb` | Upsert `{_id: key, count: 0}` | `FindOneAndUpdate({_id: key}, {$inc: {count: 1}})` with upsert |
|
||||
| `couchbase` | Create the bucket when `bucket_ram_quota_mb` is set, create the scope and collection when missing, insert `key = 0` if missing | Atomic binary increment of `key` |
|
||||
|
||||
## Timeouts
|
||||
|
||||
| Step | Limit |
|
||||
| ----------------------- | ----- |
|
||||
| Connect and setup | 1 minute; Couchbase gets `ready_timeout` plus 1 minute |
|
||||
| PostgreSQL handshake | `connect_timeout=30` seconds, added to the connection string unless you set one |
|
||||
| One write | 3 seconds |
|
||||
| Retry after a failure | 1 minute |
|
||||
| Shutdown | 7 seconds, then keepalive exits |
|
||||
|
||||
## Logging
|
||||
|
||||
keepalive logs one line per write and one line per failure, prefixed with the service name. Connection strings never reach the logs: errors from a malformed URL or DSN are replaced with a generic hint, and the generated service name uses only the host. Driver-internal logging from go-redis is silenced, since keepalive already reports each failure once.
|
||||
|
||||
## History
|
||||
|
||||
This repository replaces six single-datastore repositories (`redis-keepalive`, `valkey-keepalive`, `postgresql-keepalive`, `mysql-keepalive`, `mongodb-keepalive`, `couchbase-keepalive`). Their histories were absorbed here; earlier commits keep each implementation under its own subfolder.
|
||||
@@ -0,0 +1,45 @@
|
||||
# Known issues
|
||||
|
||||
Problems seen when pointing keepalive at hosted datastores, listed by the log line they produce. keepalive creates its table, key, or document on first connect, so a dedicated keepalive user needs write access plus permission to create that object. Until the cause is fixed, the service logs the error and retries every minute; no restart is needed after the fix.
|
||||
|
||||
## PostgreSQL: `permission denied for schema public`
|
||||
|
||||
```
|
||||
[aiven-postgres] connect: pq: permission denied for schema public at position 2:28 (42501); retrying in 1m0s
|
||||
```
|
||||
|
||||
**Cause.** On connect, keepalive runs `CREATE TABLE IF NOT EXISTS keepalive` in the `public` schema. Since PostgreSQL 15, only the database owner can create objects in `public` by default, so a non-owner user is refused. PostgreSQL checks this permission even when the table already exists.
|
||||
|
||||
**Fix.** Give keepalive its own database and make its user the owner. Run this as an admin user (`avnadmin` on Aiven):
|
||||
|
||||
```sql
|
||||
CREATE DATABASE keepalive OWNER keepalive; -- new database
|
||||
ALTER DATABASE keepalive OWNER TO keepalive; -- existing database
|
||||
```
|
||||
|
||||
Then point `url` at that database, for example `postgres://keepalive:...@host:port/keepalive?sslmode=require`.
|
||||
|
||||
On Aiven for PostgreSQL, changing the owner was the fix that worked; `GRANT CREATE ON SCHEMA public TO keepalive` did not. If you try the grant anyway, note that it applies only inside the database it runs in, so a SQL editor connected to `defaultdb` does not change the database in `url`.
|
||||
|
||||
## Redis / Valkey: `NOPERM No permissions to access a key`
|
||||
|
||||
```
|
||||
[aiven-valkey] connect: NOPERM No permissions to access a key; retrying in 1m0s
|
||||
```
|
||||
|
||||
**Cause.** The user's ACL key pattern does not cover the counter key. keepalive writes `<namespace>:<counter_key>` when `namespace` is set, otherwise `<counter_key>` (default `counter`).
|
||||
|
||||
On Aiven for Valkey, a common trigger is typing the key pattern with a leading `~`. Aiven adds the `~` itself, so `~keepalive:*` becomes a pattern that only matches keys starting with a literal `~`, and every write is refused.
|
||||
|
||||
**Fix.** Allow the commands `PING`, `SETNX`, and `INCR`, and set a key pattern that covers the counter key. On Aiven, enter the keys field without the `~`:
|
||||
|
||||
| Field | Value |
|
||||
| ---------- | ------------------------------------------------------- |
|
||||
| Categories | `+@all -@admin -@dangerous` (or `+@connection +@string`) |
|
||||
| Keys | `keepalive:*` (matching `namespace: keepalive`) |
|
||||
|
||||
## Couchbase: `CONNECTION_ERROR` or bucket not ready
|
||||
|
||||
**Cause.** The cluster is unreachable or the bucket cannot be opened with the configured user.
|
||||
|
||||
**Fix.** Check the connection string, `bucket_name`, the database user's permissions, and Capella's allowed IP/network access. If the cluster or bucket was just created, raise `ready_timeout`; keepalive allows `ready_timeout` plus 1 minute for each connect attempt.
|
||||
Reference in new issue
Block a user