Hello, running into an issue with scalesets where tons of runners are being identified as "missing" which is causing GARM to delete the runner even though it has picked up a job.
Timeline:
- 11:21:57 - Runner is pruned because it was not detected as existent in github.
time=2026-07-29T11:21:57.250Z level=INFO source=/runner/_work/cloudbase-garm/cloudbase-garm/workers/scaleset/scaleset.go:489 msg="runner does not exist in github; removing from provider" runner_name=gssdue4clml-zaict7cxmusy worker=scaleset-worker-gssdue4clml-2 entity=OPG endpoint=github.com
- 11:22:01.107 - Runner picks up a job
time=2026-07-29T11:22:01.107Z level=INFO source=/runner/_work/cloudbase-garm/cloudbase-garm/workers/scaleset/scaleset_listener.go:166 msg="job started" job_id=23b2785d-8345-5b9e-b726-a0e97794469f job_name="" runner_name=gssdue4clml-zaict7cxmusy sub_worker=scaleset-listener worker=scaleset-worker-gssdue4clml-2 entity=ORG endpoint=github.com
- 11:22:01.567 - Runner is deleted in kubernetes API
I also found these logs right before GARM started deleting runners, which might be related:
time=2026-07-29T11:21:56.438Z level=INFO source=/runner/_work/cloudbase-garm/cloudbase-garm/workers/provider/instance_manager.go:426 msg="updates channel closed" worker=instance-worker-gssdue4clmm-pcfkjfe8h9zm
Some thoughts:
When scaleset goes to deregister runner from github it is told by github that the runner is currently running a job, the flow should probably be deregister first then delete in provider? That way github has a chance to tell you its running something.
time=2026-07-29T11:33:38.801Z level=ERROR source=/runner/_work/cloudbase-garm/cloudbase-garm/workers/scaleset/scaleset.go:448 msg="error removing runner" runner_name=gssdue4clml-zaict7cxmusy error="request failed for https://broker.actions.githubusercontent.com/rest/_apis/distributedtask/pools/0/agents/44587480?api-version=6.0-preview: bad request while calling https://broker.actions.githubusercontent.com/rest/_apis/distributedtask/pools/0/agents/44587480?api-version=6.0-preview: "{\"$id\":\"1\",\"errorCode\":0,\"eventId\":3000,\"innerException\":null,\"message\":\"Runner gssdue4clml-zaict7cxmusy is currently running a job and cannot be deleted.\",\"typeKey\":\"TaskAgentJobStillRunningException\",\"typeName\":\"Microsoft.TeamFoundation.DistributedTask.WebApi.TaskAgentJobStillRunningException, Microsoft.TeamFoundation.DistributedTask.WebApi\"}"" worker=scaleset-worker-gssdue4clml-2 entity=ORG endpoint=github.com
Hello, running into an issue with scalesets where tons of runners are being identified as "missing" which is causing GARM to delete the runner even though it has picked up a job.
Timeline:
I also found these logs right before GARM started deleting runners, which might be related:
Some thoughts:
When scaleset goes to deregister runner from github it is told by github that the runner is currently running a job, the flow should probably be deregister first then delete in provider? That way github has a chance to tell you its running something.