awaylab-vke

Provisions a Vultr Kubernetes Engine (VKE) cluster with OpenTofu. VKE is Vultr's managed offering — free control plane, built-in CNI/CCM/CSI — so unlike a self-managed setup there's no image building or addon installation required.

Prerequisites

• A Vultr account and API key, exported as VULTR_API_KEY in ~/.config/secrets.env
• A git.goeks.net access token, exported as GIT_GOEKS_USERNAME and GIT_GOEKS_TOKEN in ~/.config/secrets.env (used by ArgoCD to pull this repo)
• OpenTofu, kubectl, and kustomize, pinned in mise.toml — run mise install

Setup

cp terraform.tfvars.example terraform.tfvars
# edit terraform.tfvars: check node_plan/k8s_version against Vultr's current offerings

mise run plan       # preview
mise run apply      # provision the cluster
mise run kubeconfig # writes ./kubeconfig

KUBECONFIG=./kubeconfig kubectl get nodes

VULTR_API_KEY is read from ~/.config/secrets.env by the plan/apply/destroy tasks and passed in as TF_VAR_vultr_api_key — it's never written to a tfvars file.

GitOps (ArgoCD)

ArgoCD is installed with the "app-of-apps" pattern: a root Application watches argocd/apps (a kustomize directory) in this same repo, and new workloads are added by dropping an Application manifest in there (referenced from argocd/apps/kustomization.yaml) — the root app auto-syncs whatever it finds.

mise run argocd-install     # installs ArgoCD core components (pinned to v3.5.5)
mise run argocd-bootstrap   # applies the repo credential secret + root Application
mise run argocd-password    # prints the initial admin password

kubectl -n argocd port-forward svc/argocd-server 8080:443
# open https://localhost:8080, log in as admin with the password above

To add a new workload: create its kustomize base, add an Application manifest pointing at that base, and add the manifest's filename to argocd/apps/kustomization.yaml's resources: list.

Stalwart (staging)

Stalwart is deployed for testing only: the goal is to exercise bootstrapping through stalwart-cli and holding config.json in a ConfigMap. After the migration cutover, 0.16 is stopped and the mail.goeks.net Ingress routes to the 1.0 management service; mail protocol ports remain internal to the cluster.

Unlike the other workloads, Stalwart publishes no hosted Helm repo — the official docs provide a reference chart instead, so it is vendored at argocd/apps/stalwart/ and the Application is path-based (like cert-manager-issuer). Upstream chart changes have to be pulled in by hand.

The chart's only file-based config is config.json, rendered from .Values.config into a ConfigMap and mounted at /etc/stalwart/config.json. It describes just the DataStore (RocksDB on a PVC here); everything else lives in the database and is managed through the WebUI or stalwart-cli. That split is the thing worth testing.

recoveryAdmin is enabled so the first sign-in works without scraping pod logs for a generated password.

mise run stalwart-forward   # port-forward 9188 + print the recovery admin credentials
# open http://127.0.0.1:9188/admin

# in another shell — declarative bootstrap
curl --proto '=https' --tlsv1.2 -LsSf \
  https://github.com/stalwartlabs/cli/releases/latest/download/stalwart-cli-installer.sh | sh

STALWART_DRY_RUN=1 mise run stalwart-apply   # validate the plan
mise run stalwart-apply                      # apply it

stalwart/plan.ndjson is the bootstrap plan: one NDJSON operation per line. It uses upsert throughout, so re-applying converges rather than duplicating or failing on primary-key constraints.

Staging caveats

• Ingress ownership. The original stalwart Application still owns the mail.goeks.net Ingress and stalwart-tls certificate, but the Ingress backend is now stalwart-v1:mgmt. The 1.0 release does not create a competing Ingress. Confirm DNS and certificate readiness before using the public URL; port-forward remains available.
• recoveryAdmin.password is plaintext in git. Fine for a throwaway cluster, not for anything reachable. Rotate it and drop recoveryAdmin once a permanent admin exists (helm upgrade / ArgoCD sync), per the bootstrap guidance.
• Mail ports are ClusterIP only. SMTP/IMAP/POP3/ManageSieve are reachable only inside the cluster; the Ingress fronts the management listener alone.

Stalwart 1.0 and the migration toolbox

Stalwart 1.0 cannot open a 0.16 data store, so both versions run against one ReadWriteOnce claim (data-stalwart-0, created by the primary stalwart StatefulSet) with the 1.0 store in a second directory:

Release	Data path on the shared claim
stalwart (0.16)	claim root, mounted at /var/lib/stalwart
stalwart-v1 (1.0)	v1/, mounted at /var/lib/stalwart-v1
stalwart-toolbox	claim root at /var/lib/stalwart, plus v1/ at /var/lib/stalwart-v1

The three Stalwart workloads pin themselves to one node with nodeSelector, because the shared ReadWriteOnce claim carries no PV node affinity (vultr-block-storage binds Immediate). Update that nodeSelector in the chart values when the node pool is replaced. After the 0.16 migration cutover, the desired replicas are 0 for 0.16 and 1 for 1.0; do not restart 0.16 against the migrated data.

The stalwart-toolbox pod is an ubuntu:24.04 image running as root (sudo available) with stalwart-cli, a stalwart-migrate v1.0.0 checkout at /src/stalwart-migrate, the stalwart-migrate binary (installed from the pinned release; set tools.stalwartMigrate.buildFromSource: true to compile the checkout instead), and Python 3 (python3, 3.12 on 24.04) for the migration helpers in the checkout. Both binaries land on PATH in /usr/local/bin. Tooling is installed afresh in each new container; the shared claim stores data, not installed packages. Set FORCE_BOOTSTRAP=1 to redo it in a running container. Bootstrap takes several minutes (package install plus tool downloads), during which /usr/local/bin stays empty.

Paths inside the toolbox:

Path	Contents
/usr/local/bin/stalwart-cli	stalwart-cli (symlink to the installer's ~/.cargo/bin or ~/.local/bin)
/usr/local/bin/stalwart-migrate	stalwart-migrate (symlink, or the compiled binary when building from source)
python3	Python 3.12, for cli-plan/convert_plan.py and the other helpers in the checkout
/src/stalwart-migrate	source checkout at tools.stalwartMigrate.ref
/scripts/bootstrap.sh	startup installer
/etc/stalwart/config.json	0.16 DataStore config (--source)
/etc/stalwart-v1/config.json	1.0 DataStore config (--target, path /var/lib/stalwart-v1)
/var/lib/stalwart	shared claim root (0.16 store)
/var/lib/stalwart-v1	shared claim's v1/ directory (1.0 store)
/var/lib/stalwart-migrate	migration state directory on its own claim, /var/lib/stalwart-migrate
/run/stalwart-toolbox/bootstrap.done	container-local readiness marker, created only when bootstrap completes

The three workloads come from two charts: argocd/apps/stalwart/ (shared by the stalwart and stalwart-v1 releases, the latter via the values-v1.yaml overlay) and argocd/apps/stalwart-toolbox/. Their Applications are argocd/apps/stalwart.yaml, stalwart-v1.yaml, and stalwart-toolbox.yaml, all listed in argocd/apps/kustomization.yaml.

mise run stalwart-toolbox                       # interactive shell

mise run stalwart-migrate -- prepare \
  --source /etc/stalwart/config.json \
  --state /var/lib/stalwart-migrate

mise run stalwart-migrate -- run \
  --source /etc/stalwart/config.json \
  --target /etc/stalwart-v1/config.json \
  --state /var/lib/stalwart-migrate \
  --owner 2000:2000

# scale `stalwart` (0.16) to 0, then the final copy:
mise run stalwart-migrate -- run --final \
  --source /etc/stalwart/config.json \
  --target /etc/stalwart-v1/config.json \
  --state /var/lib/stalwart-migrate \
  --owner 2000:2000

# then scale `stalwart-v1` to 1

/var/lib/stalwart-migrate is its own 2 Gi PersistentVolumeClaim (stalwart-toolbox-migration-state), so the state survives restarts of the toolbox pod and stays outside both data stores, as the migration tool requires.

Notes:

• Claim name. data-stalwart-0 is <volumeClaimTemplate>-<fullname>-<ordinal> from the primary release. The chart gained a persistence.existingClaim value so stalwart-v1 and the toolbox mount that claim instead of creating their own; the primary release still renders a volumeClaimTemplate.
• Migration state. The toolbox creates its own stalwart-toolbox-migration-state claim and mounts it at /var/lib/stalwart-migrate, so stalwart-migrate --state survives pod restarts. That claim is a normal Argo CD-managed resource, so deleting it (e.g. via prune) drops the state and requires prepare to run again. To keep state deliberately, annotate the PVC with argocd.argoproj.io/sync-options: Prune=false,Delete=false.
• Scheduling. All three workloads set nodeSelector to the same node. Do not use mutual podAffinity to co-locate them: it deadlocks when a node from the pool is replaced, because neither pod can schedule first. The vultr-block-storage class binds Immediate and stamps no PV node affinity, so the scheduler has no co-location hint of its own.
• Image tag. stalwart-v1 pins v1.0.0 with pullPolicy: Always, since a cached v1 tag can shadow a newer 1.x release.
• Toolbox lifecycle. The toolbox uses a Recreate strategy and must never run more than one pod (single writer to the shared claim). Its base image is pinned to ubuntu:24.04.
• Migration order. The v1 directory starts empty and must be populated by stalwart-migrate before scaling stalwart-v1 to 1.
• Bootstrap runs as the entrypoint, not an init container. bootstrap.sh is the container's command (followed by exec sleep infinity), so the pod is attachable immediately and a failure leaves it debuggable instead of in CrashLoopBackOff. The pod reports Ready only once the marker exists; running ./bootstrap.sh by hand while the automatic run is in flight waits on a flock instead of fighting over the dpkg lock.
• Bootstrap lifecycle. Each new Ubuntu container starts without apt packages or downloaded binaries; the readiness marker lives at /run/stalwart-toolbox/bootstrap.done inside the container, never on the shared claim. A restart re-installs tools before reporting Ready. To rerun in the current container, use FORCE_BOOTSTRAP=1 /scripts/bootstrap.sh (or delete the marker). Old .toolbox/bootstrap*.done files on the claim are obsolete.

Monitoring

kube-prometheus-stack (Prometheus, Grafana, Alertmanager, node-exporter, kube-state-metrics) is deployed by the monitoring Application into the monitoring namespace. Grafana is published at https://metrics.goeks.net through Traefik with a letsencrypt-prod certificate.

Because VKE runs the control plane outside the cluster, the etcd, controller-manager, and scheduler monitors are disabled (along with their alert rules) so they don't sit permanently firing.

Grafana's admin credentials come from a Secret you create out of band, so the password stays stable across syncs (the chart would otherwise regenerate a random password every render, which ArgoCD cannot settle because it has no Helm lookup):

kubectl -n monitoring create secret generic grafana-admin \
  --from-literal=admin-user=admin \
  --from-literal=admin-password="$(openssl rand -base64 24)"

This stack does not provide the Kubernetes resource metrics API, so k9s' CPU/MEM columns and kubectl top still need metrics-server (or prometheus-adapter) separately.

The prometheus-adapter Application fills that gap: it serves metrics.k8s.io from the Prometheus above, so kubectl top and k9s CPU/MEM columns work without a separate metrics-server. Verify with:

kubectl top nodes
kubectl get apiservice v1beta1.metrics.k8s.io   # Available should be True

Kargo, Argo Workflows, and Zot

Three more Applications round out the platform:

App	Namespace	URL	Chart
kargo	kargo	https://kargo.goeks.net	ghcr.io/akuity/kargo-charts/kargo 1.12.4 (OCI)
argo-workflows	argo	https://workflows.goeks.net	argo-workflows 2.0.13
zot	zot	https://oci.goeks.net	zot 0.1.128

All three hostnames need DNS records pointing at the cluster, since the letsencrypt-prod HTTP-01 challenge must resolve.

Kargo is exposed through Traefik with a letsencrypt-prod certificate; Traefik terminates TLS and proxies plain HTTP to the API server (api.tls.enabled=false, api.tls.terminatedUpstream=true). Admin credentials live in a kargo-admin Secret created out of band:

pass=$(openssl rand -base64 48 | tr -d "=+/" | head -c 32)
kubectl -n kargo create secret generic kargo-admin \
  --from-literal=ADMIN_ACCOUNT_PASSWORD_HASH="$(htpasswd -bnBC 10 "" "$pass" | tr -d ':\n')" \
  --from-literal=ADMIN_ACCOUNT_TOKEN_SIGNING_KEY="$(openssl rand -base64 48 | tr -d '=+/' | head -c 32)"
echo "admin / $pass"

Argo Workflows serves its UI at workflows.goeks.net. crds.full is set to false so the CRDs are plain ArgoCD-managed resources rather than a Helm hook Job; the server runs in its default server auth mode, so the login page takes a bearer token (there is no username/password).

The argo-workflows-token Application creates a long-lived (kubernetes.io/service-account-token) Secret for the server's service account, so the token does not expire hourly like kubectl create token:

kubectl -n argo get secret argo-workflows-server-token -o jsonpath='{.data.token}' | base64 -d
# paste the result into the token field at https://workflows.goeks.net

The Application ignores the Secret's /data so ArgoCD doesn't fight the token controller that populates it.

Zot serves an OCI registry at oci.goeks.net, backed by a 20Gi vultr-block-storage PVC. The workload uses a Recreate strategy because that volume is ReadWriteOnce.

Linkwarden

Linkwarden (bookmarking with page preservation) is published at https://links.goeks.net. It publishes no Helm chart, so the Deployment, Meilisearch companion, Services, Ingress, and PVCs are vendored at argocd/apps/linkwarden/ and the Application is path-based (like attic). The app image is pinned to v2.16.3, and Meilisearch to the version that release pins (v1.13.3).

Postgres lives on Neon — the linkwarden database in the same AwayLab project as the other workloads (aws-us-east-2, owner nextcloud_owner). The connection string is held in the linkwarden-db Secret, created out of band:

cd argocd/apps/linkwarden   # or wherever
neon databases create --project-id square-water-04926277 \
  --branch production --name linkwarden --owner-name nextcloud_owner
# copy the value from `neon connection-string --project-id ... --database-name linkwarden --role-name nextcloud_owner`
kubectl -n linkwarden create secret generic linkwarden-db \
  --from-literal=database-url='postgresql://...@ep-...neon.tech/linkwarden?sslmode=require'

The remaining Secrets are also created out of band:

kubectl -n linkwarden create secret generic linkwarden-secrets \
  --from-literal=nextauth-secret="$(openssl rand -base64 32)"
kubectl -n linkwarden create secret generic linkwarden-meili \
  --from-literal=master-key="$(openssl rand -base64 32)"
kubectl -n linkwarden create secret generic linkwarden-oidc \
  --from-literal=client-id=linkwarden \
  --from-literal=client-secret="$(openssl rand -base64 32)"

The OIDC client secret must match the LINKWARDEN_SECRET key in the keycloak-clients Secret (read by keycloak-config), so the Keycloak client and the app agree:

kubectl -n keycloak get secret keycloak-clients -o json | \
  jq --arg s "<same value>" '.data.LINKWARDEN_SECRET = ($s | @base64)' | kubectl apply -f -

Sign-in is Keycloak SSO only (NEXT_PUBLIC_DISABLE_REGISTRATION and NEXT_PUBLIC_CREDENTIALS_ENABLED are both off). On a fresh database the first account to sign in becomes user ID 1, which Linkwarden treats as the administrator (NEXT_PUBLIC_ADMIN). The homepage widget needs a per-user API key, stored as HOMEPAGE_VAR_LINKWARDEN_KEY in the homepage-widgets Secret.

links.goeks.net needs a DNS record pointing at the cluster so the letsencrypt-prod HTTP-01 challenge can complete.

kro

kro (Kube Resource Orchestrator) runs in kro-system, deployed by the kro Application from the upstream OCI chart (registry.k8s.io/kro/charts/kro, pinned to 0.9.4). It is a controller with no UI or hostname: you apply a ResourceGraphDefinition (RGD), kro validates it, mints a CRD for the resource type it describes, and reconciles the underlying resources for each instance of that CRD.

The chart's crds/ directory is rendered too, so resourcegraphdefinitions.kro.run and graphrevisions.internal.kro.run are ArgoCD-managed like every other resource.

Two replicas run with leader election, spread across nodes, so the standby takes over quickly when a node is lost. Controller metrics are scraped via a ServiceMonitor carrying release: monitoring, the label kube-prometheus-stack's Prometheus selects on.

kubectl -n kro-system get deploy kro
kubectl get crds | rg kro.run
kubectl get resourcegraphdefinitions

RGDs and instances are not managed by ArgoCD — apply them with kubectl or a separate repo. config.allowCRDDeletion is left at its default (false), so deleting an RGD leaves the generated CRD and its instances in place. The chart grants the controller cluster-wide access by default; set rbac.mode: aggregation to restrict it to its own resources plus ClusterRoles labelled rbac.kro.run/aggregate-to-controller: "true".

AppPack

The first RGD, apppack, packages "a ConfigMap that belongs to one Argo app". An AppPack instance (in the argocd namespace) creates a ConfigMap named after the bundle with app.kubernetes.io/part-of: <app> plus any labels the instance adds, and — when the instance sets repoURL — an ArgoCD Application (OCI/Helm chart only) whose Helm values come from the instance spec, with the app's sync/health status gated through readyWhen. Never set the argocd.argoproj.io/instance label on kro-created resources: the root app syncs with prune: true and would delete anything carrying it.

The base lives at argocd/apps/apppack/base (the RGD, deployed by the apppack-rgd Application), and each bundle is an overlay under argocd/apps/apppack/overlays/<name> deployed by its own Application — apppack-grafana-datasources is the first: the Grafana datasource ConfigMaps for Loki/Tempo/Pyroscope moved out of argocd/apps/grafana-datasources into the instance spec, into a single grafana_datasource: "1" ConfigMap (the sidecar projects every data key). Because instances are kubectl-applied rather than git-rendered, a bundle's config edits are no longer ArgoCD diffs.

Notes

• terraform.tfvars.example defaults to region = ewr (New Jersey), 3 nodes on vc2-1c-2gb ($10/month regular compute), HA control plane, Kubernetes v1.36.2+1 (latest supported as of 2026-09-06). Verify current plan IDs/pricing and supported Kubernetes versions via the Vultr dashboard or API before relying on these.
• ha_controlplanes = true in main.tf — VKE runs multiple control-plane replicas at no extra Vultr charge.

Teardown

mise run destroy