In-memory instance cache leaks deleted instances, causing runner scale-up starvation

[benoit-nexthop]

Summary

The in-memory instance cache (cache/instance_cache.go) accumulates stale entries for instances that were long ago deleted from the database. Over ~9 days of normal operation our GARM instance's cache grew to ~900 instances while the database contained only 77, of which 819 cached entries were tombstones with status: deleted. Anything served from the cache (e.g. the /api/v1/ws/metrics dashboard snapshot, per-pool/scale-set runner counts, etc.) reports wildly inflated numbers. Restarting GARM "fixes" it (cache reloads from the DB), and the leak then starts again.

Environment

• GARM main (reproduced against current main, commit 2edcc2ff)
• 6 scale sets (~60–80 concurrent runners, frequent scale up/down churn)

Observed behavior

Dashboard reported:

Instances: 895   (16 active / 329 idle / 53 pending / 497 offline)

while GET /api/v1/instances returned 77 instances, all running. Dumping a snapshot from /api/v1/ws/metrics shows the cache contents:

total instances in snapshot: 896
by instance status: deleted: 819, running: 74, deleting: 2, pending_delete: 1
by runner status:   terminated: 494, idle: 328, active: 16, pending: 48, installing: 5, offline: 5

819 cache entries have status: deleted — the terminal status set just before the DB row is removed — and they are never evicted.

Root cause

Instance deletion emits two watcher events in sequence:

1. UpdateOperation with Status = InstanceDeleted (set by the provider worker via SetInstanceStatus)
2. DeleteOperation when the row is removed (DeleteInstanceByName / DeleteInstance)

The cache worker (workers/cache/cache.go, handleInstanceEvent) stores the instance on create/update and evicts it on delete. This relies on events being delivered in order and reliably. The watcher guarantees neither:

1. Events are delivered out of order

database/watcher/watcher.go, serviceProducer:

case payload := <-prod.messages:
    w.mux.Lock()
    for _, c := range w.consumers {
        go c.Send(payload)   // one goroutine per event per consumer
    }
    w.mux.Unlock()

Each event is handed to each consumer in a fresh goroutine. Goroutine scheduling order is unspecified, so two events emitted back-to-back can be enqueued to a consumer in reverse order. When the DeleteOperation is processed before the UpdateOperation(status=deleted), the update re-inserts the already-deleted instance into the cache. Since InstanceDeleted is a terminal state, no further event ever evicts it — the entry leaks until process restart.

This matches what we observed: every leaked entry had status: deleted.

2. Events are silently dropped under load

database/watcher/consumer.go, Send:

timer := time.NewTimer(1 * time.Second)
...
select {
...
case <-timer.C:
    slog.DebugContext(w.ctx, "timeout trying to send payload", "payload", payload)  // event lost
case w.messages <- payload:
}

Consumer channels are buffered at 1 (messages: make(chan common.ChangePayload, 1)). During deletion bursts (scale-down of many runners), sends time out after 1s and the event is dropped with only a debug log. A dropped DeleteOperation leaks the cache entry the same way.

There is no periodic reconciliation of the cache against the database — it is populated once at startup and then maintained purely from this lossy, unordered event stream, so any missed/reordered eviction persists forever.

Impact

• GARM's built-in dashboard instance counts and runner-status breakdowns drift ever upward (ours was off by >10x after 9 days).
• Any consumer of cache.GetAllInstancesCache() / GetInstancesForPool / GetInstancesForScaleSet sees phantom instances; other watcher consumers (scale set workers, etc.) are exposed to the same reordering/drop hazards.

Secondary effect: scale-up starvation (jobs wait many minutes for a runner)

The scale-set autoscaler is driven by the worker-local w.runners map (workers/scaleset/scaleset.go), which is maintained from the same unordered/lossy watcher stream. handleAutoScale compares len(w.runners) against min(min_idle_runners + desired_runner_count, max_runners) and only creates runners when below target.

Two ways stale entries inflate that count:

1. Reordered events: a late UpdateOperation (status running/idle) processed after the DeleteOperation re-inserts a dead runner. The periodic cleanup only evicts entries with status deleted, so it stays.
2. Zero-value re-insertion: when consolidation finds a w.runners entry missing from GitHub, it calls setRunnerDBStatus then UpdateInstance. If the DB row is already gone, ErrNotFound is swallowed and a zero-value params.Instance{} is returned with nil error, then stored back via w.runners[runner.ID] = instance. An entry with Status: "" matches no cleanup path and permanently occupies a runner slot. The same pattern exists in removeRunnerFromGithubAndSetPendingDelete.

Each stale entry makes the worker believe it is closer to (or above) target, so it refuses to scale up — and may scale down real idle runners. We observed queued jobs waiting 10-20 minutes for a runner during a near-idle weekend while the cache held 389 entries for a scale set with max_runners: 100 (actual DB instances: 45). Restarting GARM resolved this too.

Suggested additional fix: in setRunnerDBStatus callers, do not store the returned instance when UpdateInstance returned ErrNotFound — evict the entry from w.runners instead.

Workaround

Restart GARM so that the cache is rebuilt from the DB and counts are correct again until the leak re-accumulates.

Suggested fixes

Any of these (ideally the first two together):

1. Preserve per-consumer ordering: replace go c.Send(payload) with a per-consumer serialized queue (single dispatcher goroutine per consumer draining a larger buffer), so events arrive in emission order.
2. Defense in depth in the cache worker: treat UpdateOperation with Status == InstanceDeleted as an eviction rather than an upsert.
3. Periodic reconciliation: have the cache worker periodically re-list instances from the DB and evict cache entries that no longer exist, so any missed event self-heals.
4. Increase consumer channel buffers and/or surface send timeouts at warn/error level so drops are visible.

[benoit-nexthop]

The GARM restart also caused memory usage to drop from ~504MB down to ~110MB

(graph based on the container_memory_working_set_bytes metric from k8s)

Longer time periods show that this is gradually leaking over time.

[gabriel-samfira]

Hi @benoit-nexthop

I will be on vacation for the next couple of weeks, after which I will look at all issues and PRs.

However, that memory usage looks suspiciously high, even post restart. It looks high even for a 800 leaked cache (which will be addressed as soon as I get back). Those 800 objects should not take up more than about 20-30 MB of memory. 500 MB of memory usage looks suspicious.

To give you an idea, here is my local instance after almost 3 weeks of being online:

root@garm:~# systemctl status garm | cat
● garm.service - GitHub Actions Runner Manager (garm)
     Loaded: loaded (/etc/systemd/system/garm.service; enabled; preset: enabled)
     Active: active (running) since Wed 2026-07-29 06:00:47 EEST; 2 weeks 5 days ago
   Main PID: 16458 (garm)
      Tasks: 12 (limit: 29896)
     Memory: 44.5M (peak: 125.5M)
        CPU: 2h 23min 8.927s
     CGroup: /system.slice/garm.service
             └─16458 /usr/local/bin/garm --config /etc/garm/config.toml

And the garm CI container after 6 weeks:

root@garm-ci:~# docker container stats --no-stream garm
CONTAINER ID   NAME      CPU %     MEM USAGE / LIMIT     MEM %     NET I/O           BLOCK I/O        PIDS
939b62180069   garm      0.00%     79.89MiB / 7.143GiB   1.09%     3.56GB / 3.71GB   177MB / 7.91GB   13

would you be open to enabling the debug server and profiling your GARM deployment? Details here:

https://github.com/cloudbase/garm/blob/main/doc/configuration.md#default

You can grab the heap, allocations and go routines when the process consumes a lot of memory using:

curl -so heap.pb.gz "http://127.0.0.1:9997/debug/pprof/heap"
curl -so allocs.pb.gz "http://127.0.0.1:9997/debug/pprof/allocs"
curl -so goroutine.pb.gz "http://127.0.0.1:9997/debug/pprof/goroutine"

You can add ?debug=1 to get a human readable text version.

If you do grab the .pb.gz versions, you can then inspect them with something like:

go tool pprof -top -inuse_space heap.pb.gz

This will give you the top 5 or 10 (can't remember) sources of memory usage by bytes.

go tool pprof -top -inuse_objects heap.pb.gz

This will give you object counts in memory.

go tool pprof -top -alloc_space allocs.pb.gz

This will give you the top allocators of memory (not necessarily who holds memory, but who allocates it).

The whole cache thing was implemented to reduce the amount of DB queries in extremely busy deployments. But I will reconsider if it causes more issues than it's worth. I will investigate more thoroughly once I get back into the office.