mirror of
https://github.com/tiennm99/coolify.git
synced 2026-10-13 10:15:40 +00:00
feat(dev): bootstrap coold VMs via cooldctl
This commit is contained in:
Andras Bacsai committed
1 parent
1b1b0726a2
commit
d92c6a8dab
5 files changed
+515
-123
No files matched your search
@@ -0,0 +1,182 @@
|
||||
# coold Dev Environment Notes
|
||||
|
||||
This file documents the current local v5/coold dev setup in Coolify.
|
||||
|
||||
## Roles
|
||||
|
||||
- `scripts/dev.sh` owns local developer convenience.
|
||||
- Lima VMs act like real deployment servers.
|
||||
- `cooldctl init bootstrap` owns host wiring:
|
||||
- WireGuard
|
||||
- Podman mesh networks
|
||||
- Corrosion config/schema/service
|
||||
- coold install/service
|
||||
- builder install/config
|
||||
- default-deny firewall service
|
||||
- Coolify/Laravel dev owns Flux-specific wiring:
|
||||
- starts Flux inside the Coolify container
|
||||
- mints dev host JWTs
|
||||
- installs `/etc/coolify/host-jwt` into each VM
|
||||
- adds a `coold.service` systemd drop-in with `COOLIFY_COOLD_FLUX_URL`
|
||||
|
||||
## Main commands
|
||||
|
||||
```bash
|
||||
scripts/dev.sh up
|
||||
scripts/dev.sh down
|
||||
scripts/dev.sh clean-vms
|
||||
scripts/dev.sh list
|
||||
```
|
||||
|
||||
`clean-vms` deletes the Lima VM instances and VM-local state, including disks,
|
||||
containers, WireGuard keys, Corrosion DB, installed binaries, and firewall state.
|
||||
It does not delete the Coolify repo.
|
||||
|
||||
## cooldctl helper
|
||||
|
||||
```bash
|
||||
scripts/dev.sh cooldctl install
|
||||
scripts/dev.sh cooldctl path
|
||||
scripts/dev.sh cooldctl bootstrap-command
|
||||
scripts/dev.sh cooldctl run <args>
|
||||
```
|
||||
|
||||
On macOS, the helper builds `cooldctl` from the local coold repo because the
|
||||
nightly release currently publishes Linux binaries. On Linux, it downloads the
|
||||
nightly release artifact.
|
||||
|
||||
The generated bootstrap command uses Lima's forwarded SSH ports and dev
|
||||
WireGuard endpoint overrides, for example:
|
||||
|
||||
```bash
|
||||
.dev/bin/cooldctl init bootstrap \
|
||||
--nodes "127.0.0.1:<ssh1>,127.0.0.1:<ssh2>" \
|
||||
--ssh-key "$HOME/.lima/_config/user" \
|
||||
--ssh-user "$USER" \
|
||||
--wg-listen-port-overrides "127.0.0.1:<ssh1>=51821,127.0.0.1:<ssh2>=51822" \
|
||||
--wg-endpoint-overrides "127.0.0.1:<ssh1>=host.lima.internal:51821,127.0.0.1:<ssh2>=host.lima.internal:51822" \
|
||||
--coold-version "nightly" \
|
||||
--corrosion-version "v1.0.0" \
|
||||
--yes
|
||||
```
|
||||
|
||||
Lima does not allow direct root SSH by default, so dev uses the normal Lima user
|
||||
with passwordless sudo. `cooldctl` wraps remote commands in `sudo -n bash -lc`
|
||||
when the SSH user is not `root`.
|
||||
|
||||
## Default dev topology
|
||||
|
||||
After `cooldctl init bootstrap`, defaults are:
|
||||
|
||||
| VM | WireGuard IP | WireGuard endpoint | Podman subnet | Gateway |
|
||||
| --- | --- | --- | --- | --- |
|
||||
| `coold-dev` | `100.64.0.1` | `host.lima.internal:51821` | `10.210.0.0/24` | `10.210.0.1` |
|
||||
| `coold-dev-2` | `100.64.0.2` | `host.lima.internal:51822` | `10.210.1.0/24` | `10.210.1.1` |
|
||||
|
||||
## Checking state
|
||||
|
||||
```bash
|
||||
scripts/dev.sh corrosion check
|
||||
scripts/dev.sh corrosion containers
|
||||
scripts/dev.sh corrosion config
|
||||
scripts/dev.sh corrosion logs 1
|
||||
scripts/dev.sh corrosion logs 2
|
||||
```
|
||||
|
||||
`corrosion containers` shows both Corrosion `service_endpoints` and rootful /
|
||||
rootless Podman containers.
|
||||
|
||||
## Example nginx containers
|
||||
|
||||
```bash
|
||||
scripts/dev.sh example-nginx up
|
||||
scripts/dev.sh example-nginx check-dns
|
||||
scripts/dev.sh example-nginx down
|
||||
```
|
||||
|
||||
The example containers are started with coold DNS:
|
||||
|
||||
```bash
|
||||
--dns <local-mesh-gateway>
|
||||
--dns-search default.coolify.internal
|
||||
```
|
||||
|
||||
Expected service discovery format:
|
||||
|
||||
```text
|
||||
<container-name>.default.coolify.internal
|
||||
```
|
||||
|
||||
Example:
|
||||
|
||||
```text
|
||||
coolify-example-nginx-2.default.coolify.internal -> 10.210.1.x
|
||||
```
|
||||
|
||||
## Firewall behavior
|
||||
|
||||
The mesh firewall is default-deny for inter-container traffic. Host-to-host
|
||||
WireGuard traffic can work while container-to-container traffic is blocked.
|
||||
|
||||
Allow traffic:
|
||||
|
||||
```bash
|
||||
scripts/dev.sh firewall allow 10.210.0.2 10.210.1.2 tcp 80
|
||||
scripts/dev.sh firewall list
|
||||
```
|
||||
|
||||
Revoke traffic:
|
||||
|
||||
```bash
|
||||
scripts/dev.sh firewall revoke
|
||||
scripts/dev.sh firewall revoke <rule-id>
|
||||
```
|
||||
|
||||
### Why dev adds allow rules to both hosts
|
||||
|
||||
`business traffic` from container A on host 1 to container B on host 2 crosses
|
||||
forwarding/firewall logic on both sides:
|
||||
|
||||
```text
|
||||
10.210.0.2
|
||||
-> source host bridge/firewall
|
||||
-> coold-dev wg0
|
||||
-> WireGuard
|
||||
-> coold-dev-2 wg0
|
||||
-> destination host bridge/firewall
|
||||
-> 10.210.1.2
|
||||
```
|
||||
|
||||
The default-deny hooks can drop the packet on either the source or destination
|
||||
host. For the two-node dev setup, `scripts/dev.sh firewall allow` writes the
|
||||
same allow tuple to every coold VM so the path works reliably.
|
||||
|
||||
Production should become topology-aware instead of blindly writing to every
|
||||
host:
|
||||
|
||||
- cross-host traffic: write allow rules to the source and destination hosts
|
||||
- same-host traffic: write the allow rule only to that host
|
||||
- larger clusters: do not install unrelated allow rules on unaffected hosts
|
||||
|
||||
## Manual connectivity checks
|
||||
|
||||
Before allow rule, this should time out:
|
||||
|
||||
```bash
|
||||
scripts/dev.sh shell 1
|
||||
sudo podman exec coolify-example-nginx wget -T 3 -qO- http://10.210.1.2
|
||||
```
|
||||
|
||||
After allow rule, this should return nginx HTML:
|
||||
|
||||
```bash
|
||||
scripts/dev.sh firewall allow 10.210.0.2 10.210.1.2 tcp 80
|
||||
scripts/dev.sh shell 1
|
||||
sudo podman exec coolify-example-nginx wget -T 5 -qO- http://10.210.1.2 | head -n 1
|
||||
```
|
||||
|
||||
Expected:
|
||||
|
||||
```html
|
||||
<!DOCTYPE html>
|
||||
```
|
||||
+5
-69
@@ -1,6 +1,5 @@
|
||||
# Lima VM for testing Coolify v5 against a packaged coold endpoint.
|
||||
# Lima VM for testing Coolify v5 against cooldctl-provisioned hosts.
|
||||
# Start with: scripts/coold-vm.sh up
|
||||
# Run endpoint: scripts/coold-vm.sh dev
|
||||
#
|
||||
# This file is copied into .dev/lima/coold.generated.yaml by the wrapper script,
|
||||
# with {{COOLIFY_REPO}}, {{COOLIFY_COOLD_VERSION}}, and {{COOLIFY_CORROSION_VERSION}} replaced before Lima sees it.
|
||||
@@ -41,85 +40,22 @@ provision:
|
||||
export DEBIAN_FRONTEND=noninteractive
|
||||
echo "[coold-vm] Updating Ubuntu package index..."
|
||||
apt-get update
|
||||
echo "[coold-vm] Installing VM dependencies..."
|
||||
echo "[coold-vm] Installing minimal VM dependencies..."
|
||||
apt-get install -y --no-install-recommends \
|
||||
bash \
|
||||
buildah \
|
||||
ca-certificates \
|
||||
clang \
|
||||
curl \
|
||||
git \
|
||||
iproute2 \
|
||||
iptables \
|
||||
jq \
|
||||
libssl-dev \
|
||||
mold \
|
||||
nftables \
|
||||
openssl \
|
||||
perl \
|
||||
pkg-config \
|
||||
podman \
|
||||
protobuf-compiler \
|
||||
sqlite3 \
|
||||
sudo \
|
||||
wireguard-tools
|
||||
echo "[coold-vm] Enabling Podman socket..."
|
||||
systemctl enable --now podman.socket
|
||||
sudo
|
||||
cat >/usr/local/bin/rtk <<'SH'
|
||||
#!/usr/bin/env bash
|
||||
exec "$@"
|
||||
SH
|
||||
chmod +x /usr/local/bin/rtk
|
||||
mkdir -p /etc/coolify /var/lib/corrosion /run/corrosion /var/lib/coolify-dev /var/lib/coolify-builder/work
|
||||
mkdir -p /etc/coolify /var/lib/coolify-dev
|
||||
chmod 755 /etc/coolify
|
||||
chmod 777 /var/lib/corrosion /run/corrosion
|
||||
|
||||
install_coolify_binary() {
|
||||
name="$1"
|
||||
version="{{COOLIFY_COOLD_VERSION}}"
|
||||
arch_raw="$(uname -m)"
|
||||
case "$arch_raw" in
|
||||
x86_64) arch=amd64 ;;
|
||||
aarch64) arch=arm64 ;;
|
||||
*) echo "unsupported arch: $arch_raw" >&2; exit 1 ;;
|
||||
esac
|
||||
|
||||
url="https://github.com/coollabsio/coold/releases/download/${version}/${name}-linux-${arch}.tar.gz"
|
||||
echo "[coold-vm] Installing ${name} from ${url}..."
|
||||
tmpdir="$(mktemp -d)"
|
||||
trap 'rm -rf "$tmpdir"' RETURN
|
||||
curl -fsSL --retry 3 --max-time 120 -o "$tmpdir/${name}.tar.gz" "$url"
|
||||
tar -xzf "$tmpdir/${name}.tar.gz" -C "$tmpdir"
|
||||
test -f "$tmpdir/$name"
|
||||
install -m 0755 "$tmpdir/$name" "/usr/local/bin/${name}.tmp"
|
||||
mv "/usr/local/bin/${name}.tmp" "/usr/local/bin/${name}"
|
||||
echo "$version" > "/usr/local/bin/${name}.version"
|
||||
}
|
||||
|
||||
install_corrosion_binary() {
|
||||
version="{{COOLIFY_CORROSION_VERSION}}"
|
||||
arch_raw="$(uname -m)"
|
||||
case "$arch_raw" in
|
||||
x86_64) arch=x86_64-unknown-linux-gnu ;;
|
||||
aarch64) arch=aarch64-unknown-linux-gnu ;;
|
||||
*) echo "unsupported arch: $arch_raw" >&2; exit 1 ;;
|
||||
esac
|
||||
|
||||
url="https://github.com/superfly/corrosion/releases/download/${version}/corrosion-${arch}.tar.gz"
|
||||
echo "[coold-vm] Installing corrosion from ${url}..."
|
||||
tmpdir="$(mktemp -d)"
|
||||
trap 'rm -rf "$tmpdir"' RETURN
|
||||
curl -fsSL --retry 3 --max-time 180 -o "$tmpdir/corrosion.tar.gz" "$url"
|
||||
tar -xzf "$tmpdir/corrosion.tar.gz" -C "$tmpdir"
|
||||
test -f "$tmpdir/corrosion"
|
||||
install -m 0755 "$tmpdir/corrosion" /usr/local/bin/corrosion.tmp
|
||||
mv /usr/local/bin/corrosion.tmp /usr/local/bin/corrosion
|
||||
echo "$version" > /usr/local/bin/corrosion.version
|
||||
}
|
||||
|
||||
install_coolify_binary coold
|
||||
install_coolify_binary builder
|
||||
install_corrosion_binary
|
||||
echo "[coold-vm] Endpoint binaries installed."
|
||||
|
||||
echo "[coold-vm] Provisioning complete."
|
||||
echo "[coold-vm] Minimal provisioning complete. cooldctl bootstrap will install WireGuard, Podman, Corrosion, coold, and builder."
|
||||
Reference in new issue
Block a user