Keel in Go: one binary for control plane, edge, agent and CLI

[ThallesP]

Experiment: Keel's backend rewritten in Go as one keel binary. It replaces self-hosted Convex, Better Auth, the Bun worker and the keel-proxy image. The dashboard and the CLI now run on its HTTP API. Durable workflows are left for a later run, as agreed.

What the binary is

Command	Runs	Replaces
keel serve	control plane container: HTTP API, WebSocket, SQLite, Swarm driver, embedded dashboard	Convex backend + functions, Better Auth, keel-web
keel proxy	edge container: Caddy + caddy-l4 + the keel plugin as libraries	apps/proxy image
keel agent	keel-agent Swarm global service, created by serve	apps/worker + scripts/deploy-worker.sh
keel login, ship, …	laptops and agents	apps/cli (output contract unchanged)

serve and proxy stay two containers on purpose. The process that parses internet traffic never holds the Docker socket.

Stack

• HTTP: Huma on net/http. openapi.json is generated from the handlers and committed; CI fails when it drifts.
• Errors: RFC 9457 problems carrying the CLI's own codes, so the CLI no longer matches error strings.
• Dashboard: Kubb generates TanStack Query hooks from the spec. Query keys start with the request path.
• Realtime: one centrifuge WebSocket per tab. The server publishes invalidation topics, which are API path prefixes. Writes also return them in a Keel-Invalidate header, so a mutation sees its own write.
• Data: SQLite through modernc.org/sqlite and sqlc, with embedded migrations, one writer and a read pool.
• Layers: domain (pure rules), app (use cases and the ports they need), adapters (sqlite, swarm, caddy, axiom, realtime, jobs, password), transport/http. Contract: docs/go/ARCHITECTURE.md.
• Scheduling: Convex's scheduler and crons become an in-memory job runner. A start-up Recover pass re-observes Swarm, re-arms timeouts, re-queues applies and resyncs the proxy.
• Auth: sessions are random tokens stored hashed. Passwords use argon2id. Device login for the CLI, invite-only sign-up, CSRF origin check.
• tsnet: opt-in in the agent (KEEL_TS_AUTHKEY). Swarm already needs host tailscaled, so it isn't the default.

How it was built

docs/go/spec/ holds seven porting specs written from the TypeScript, plus a completeness pass. Seven backend areas were then built in parallel, each reviewed against its spec, and merged with integration fixes. The CLI and the dashboard followed on the generated API.

Then the port was made idiomatic. There is no Convex migration (Keel has no users), so parity with the TypeScript stopped being a goal:

• JavaScript emulation removed. No JSONObject, no js* helpers, no JS string lengths or whitespace. Axiom rows, Docker events, proxy reports, Caddy config and worker config decode into typed structs.
• Zero values over pointers. Unset columns are NOT NULL DEFAULT 0/'', integer fields are integers on the wire, and runtime state lives on App instead of package-level maps.
• Unused code deleted. That covers wrappers, rename layers, pointer helpers, the settings table, test-only exports and dead fakes. Go code carries no comments (tools/nocomments in CI).

Verified

• go build, go vet, go test ./... pass; GOOS=darwin go vet ./... is clean. Area tests cover every exact algorithm: status, task summary, reconcile, variable references, port allocation, the Caddy config builder, the device flow, log shipping.
• Swarm integration test (new CI job swarm): serve and agent in one process against a real Swarm. It covers deploy, a variable shipped as an update (checked in the service spec), stop, start, redeploy and delete. It passes locally against an isolated Docker-in-Docker Swarm.
• Dashboard in headless Chromium against the embedded build and the isolated Swarm, with no console errors:
  • sign-up, the canvas, and adding a service that deploys live;
  • "Ship · 1 change", rolling out, then settled; the Logs tab streaming from Docker;
  • the Observability gate, the account menu, the invite dialog, and CLI device approval.
• CLI end to end against a real serve: login, whoami, project, service, var, status, ship, redeploy, deployment, tracing, run, logs and logout, with the JSON envelope and exit codes unchanged.
• API checks: invite-only sign-up refuses an uninvited email; an invited one joins as a member and spends the invitation.

Not verified, or known gaps

• No upgrade from Convex-era Keel. Re-running install.sh on a server that runs the Convex-based Keel starts an empty control plane; its data stays in keel_convex-data.
• External services. Axiom (sign-in, queries, ingest, the OTLP relay) is tested only against fakes and golden files from the TypeScript providers. Certificates through keel proxy are unit-tested only, not with real ACME.
• Session cookie. It is a host cookie, and cookies aren't scoped by port. On an install reached over plain http by IP, another server on the same host and IP can see it. Over the tailnet https name (tailscale serve) this doesn't apply.
• Not built in either backend: the Add-server join flow, placement pinning, and the volumes feature.
• Apply concurrency. Applies are serialized per node, but there's no global limit (Convex ran at most 16 actions at once).
• Binary size. About 110 MB unstripped, 76 MB stripped; Caddy is most of it. -tags keel_noproxy builds a lean CLI.

Test plan

• CI: go, swarm, web, install jobs green
• Fresh install.sh on a VM, then deploy, expose over https, Sign in with Axiom

🤖 Generated with Claude Code

[pullfrog]

ℹ️ No regressions in the new commits. There's one question for you below: what re-running install.sh now does on a server that still runs the Convex-era Keel.

Reviewed changes

This review covers the 142 commits from 50547c7 to 545240f. The merge base is still 36c2ded. CI's go, web, swarm and install checks pass on 545240f. These also pass locally: go vet, go test ./..., tools/nocomments and the openapi.json drift check. Separate passes compared nine areas against 50547c7: auth, SQLite, deploy, ingress, observability, the agent, the CLI, realtime and jobs, and the dashboard's wire types. Apart from the deliberate removals below, none found a different outcome on any reachable path.

• Dropped the Convex upgrade path. These are gone: keel import-convex and the importer, Better Auth scrypt verification, the export and import step in install.sh, KEEL_SKIP_CONVEX_IMPORT, /config.js, the CLI's --convex-* flags and its Convex-era discovery hint, and the convexUrl, convexSiteUrl and convexImport* fields in the install JSON. A new CLAUDE.md rule bans backfills and compatibility code.
• Banned comments in Go. tools/nocomments parses every Go file in CI and in make test. check-comments.sh now counts TypeScript only.
• Moved generated code. sqlc output is now in internal/gen/sqlc. Kubb, the route tree and varlock output are in apps/web/src/gen.
• Replaced the JavaScript emulation with plain Go.
  • Typed Axiom rows replace observability_jsvalues.go. Rows of an unexpected shape are skipped with a warning.
  • The golden-file changes come from edited inputs or new scenarios. APL text and scoping are unchanged.
  • Lengths are counted in runes, and database URLs are built with net/url.
• Unset values are zero, not NULL.
  • 0001_init.sql makes optional text columns NOT NULL DEFAULT '', and unset times and revisions default to 0.
  • The endpoint unique indexes are now partial (domain <> '', public_port <> 0), so blank values don't collide.
  • 0002_canvas.sql was folded into 0001_init.sql. The settings table and deployments.sha were dropped.
  • The API keeps omitempty, so unset fields look the same on the wire.
• Typed the wire. Ports, replicas and tail are integers in openapi.json. Node type, engine, statuses and roles use domain types, with the same enum values.
• Narrowed invalidations. A canvas write publishes /api/environments/<id> and /api/nodes/ instead of one topic per node. The dashboard's prefix match covers the same queries.
• Rewrote the agent.
  • Sinks are keyed by their full config, and events are posted as typed arrays. /worker/events now accepts only a JSON array.
  • State now lives in the keel-agent-state volume.
  • Delivery checkpoints, retries and event order are unchanged.
• Made some inputs stricter. Unexpose now rejects a malformed selector instead of treating it as a no-op. /proxy/events rejects an empty name.

ℹ️ Re-running install.sh on a Convex-era server now starts an empty Keel without warning

The README's upgrade path is "re-run the install command". On a server running main's Convex-based Keel, the new installer does the following:

• It overwrites compose.yml.
• It rewrites .env without INSTANCE_SECRET, BETTER_AUTH_SECRET and CONVEX_SELF_HOSTED_ADMIN_KEY.
• --remove-orphans removes the old backend and web containers.
• It starts an empty control plane and tells the user to sign up, with no warning.

The data stays in keel_convex-data, but the old compose file and secrets are gone. The code at 50547c7 said a rollback needs those secrets. The old keel-worker service also keeps running against a backend that no longer exists. The deployed services keep running, but Keel no longer manages them, and exposed services lose their public endpoints. The PR description still lists keel import-convex, scrypt sign-in for imported accounts, and a Convex upgrade in its test plan.

Technical details

# A Convex-era upgrade now starts empty without saying so

## Affected sites
- `install.sh:158` `save_state`: `.env` is rewritten with only the new keys, so the first run of this installer deletes the Convex-era secrets an earlier install saved.
- `install.sh:151` `write_state`: `compose.yml` is overwritten without the old `compose.convex.yml` copy.
- `install.sh:195` and `:248`: `--remove-orphans` removes the old `backend` and `web` containers. Nothing checks for `keel_convex-data` or a running `keel-worker`, and the closing message says to sign up.
- PR description: the `keel import-convex` row, "imported accounts keep their passwords", "Upgrade a Convex install" in the test plan.

## Required outcome
- Whoever re-runs the installer on a Convex-era server knows before anything is replaced that its data won't come along.

## Open questions for the human
- Will any Convex-era install (for example the Asgard copy in the test plan) run this installer? If so, should `install.sh` stop when `keel_convex-data` exists, with a `die` that names the volume, or is a README note enough?
- Should `.env` keep the Convex-era secret lines it doesn't write, so a rollback stays possible?

  ^{｜ Fix it ➔ ｜ View workflow run ｜ Using claude-opus-5-5 ｜ 𝕏}

[pullfrog]

✅ No new issues found. The one new commit fixes the four open findings from earlier reviews: row naming, the suspense waterfall, the tab reset and the comment regex. All review threads on the PR are now resolved.

Reviewed changes

This review covers 532490c, the one commit since the review of 545240f. CI's go, web, swarm and install checks pass on it. These also pass locally: go test for internal/app and internal/domain, check-types for packages/ui and apps/web, oxlint --type-aware (0 errors) and scripts/check-comments.sh.

• Row keys get prefixed names. A source's own URL, HOST or PORT row now goes through domain.SuggestedKey, the same as its provided keys. So picking api.PORT fills in API_PORT instead of overwriting the service's own PORT. Other row keys keep their names, because SuggestedKey only prefixes those three. connectionSuggestion still only reads provided keys, so suggestions are unchanged.
• Variables load in parallel. The tab reads both lists with one useSuspenseQueries over the generated …SuspenseQueryOptions. Their query keys and fetchers match the …Suspense hooks they replace, so the path-prefix invalidations still reach them.
• Errors clear on a tab switch. The bottom panel's ErrorBoundary adds resetKeys={[tab]}, so an error on one tab no longer stays up after switching to another. It still resets through useQueryErrorResetBoundary.
• Pointer lines don't count as comments. check-comments.sh now counts * only when whitespace, / or the end of the line follows it. Every file's count still equals its .comments-baseline entry, so no file gains room for new comments.

  ^{｜ View workflow run ｜ Using claude-opus-5-5 ｜ 𝕏}

[ThallesP]

[Claude] On the install question: install.sh now stops before changing anything when the keel_convex-data volume exists, and its fix line says how to remove the old install (766227c). There is still no upgrade path from the Convex-era Keel, by design. The PR description no longer mentions keel import-convex, scrypt sign-in or the Convex upgrade test.

[pullfrog]

✅ No new issues found. The new commit answers the last review's question about Convex-era servers: the installer now stops before it changes anything.

Reviewed changes

This review covers 766227c, the one commit since the review of 532490c. bash -n and shellcheck -S warning pass on install.sh. CI's go, web and swarm checks pass on this commit, and install was still running when I checked.

• Installer stops on a Convex-era server. ensure_docker now exits with an error when the volume keel_convex-data exists. The check runs right after preflight, before Tailscale, write_state or any compose call, so compose.yml and .env are left alone. The volume name matches main's compose file (name: keel, volume convex-data). The fix: line repeats main's uninstall steps (compose -p keel down, service rm keel-worker), and -p keel down works without a compose file in the current directory. If old svc-* services are left running, the new Keel ignores them: they carry the same keel.service label, but the full sweep only looks up nodes in its own database and never removes services it doesn't know. No CI job creates that volume, so the install job doesn't hit the check.
• README documents it. The upgrade section now says the Convex-era Keel doesn't upgrade and that the installer stops on a server that still has the volume. That keeps install.sh and its README contract in sync.

One thing outside the code: the PR description's "Not verified, or known gaps" section still says re-running install.sh on a Convex-era server "starts an empty control plane". Since this commit, it stops instead.

  ^{｜ View workflow run ｜ Using claude-opus-5-5 ｜ 𝕏}