Repository navigation
gitproxy: serve git fetches from cache-server on an instance - #165
Conversation
firerunner git-proxy is the loopback proxy git is pointed at, through http.https://github.com/.proxy. It answers a fetch gitcache accepts from cache-server, through the same API as actions/cache, so a git entry is scoped, evicted and stored as any other, and stores what it had to fetch. The cache must never fail a checkout GitHub would have served. A cache error falls through to GitHub. A hit that breaks part way aborts the connection, so git retries, and a key is served from the cache once per proxy, which also gets a job past a poisoned entry. A response is stored only if it is whole, as GitHub reports a failure inside a 200. A hit skips GitHub's own check, so it needs GitHub to have accepted the same credentials for the same repository moments earlier. Nothing starts this yet. It needs a tier option, a CA for github.com and the AMI, which come next. Signed-off-by: Yi-Ru Lin <deduce@gmail.com>
There was a problem hiding this comment.
🟡 Changes recommended
An opaque request target can bypass the GitHub-only restriction, causing the proxy to forward Authorization headers to an unintended host.
3 open issues
What changed in this PR
Adds firerunner git-proxy, the instance-side component of the git checkout cache. Automatic startup, certificate provisioning, and AMI integration remain deferred.
Changes:
- Adds a loopback TLS proxy that caches eligible fetches and forwards other git traffic.
- Integrates with cache-server and checks response framing before storing entries.
- Adds fault-injection, repository-isolation, CLI-validation, and real-git integration tests.
| File | Description |
|---|---|
internal/gitproxy/proxy.go |
Implements proxying, credential gating, cache hits, and background saves. |
internal/gitproxy/proxy_test.go |
Tests forwarding, cache failures, retries, and response validation. |
internal/gitproxy/git_test.go |
Exercises real git fetches through the TLS proxy. |
internal/gitproxy/check.go |
Checks streamed fetch responses for complete framing and errors. |
internal/gitproxy/cache.go |
Implements cache-server lookup, download, and upload operations. |
internal/gitproxy/cache_test.go |
Tests cache-server integration and repository/ref isolation. |
cmd/firerunner/main.go |
Registers and documents the new subcommand. |
cmd/firerunner/gitproxy.go |
Configures certificates, loopback listening, and shutdown. |
cmd/firerunner/gitproxy_test.go |
Tests loopback restrictions and invalid CLI arguments. |
Review effort: Balanced
A request for https:@example.com/path has the RequestURI @example.com/path, so pasting it after the host sent the job's Authorization header to example.com. Build the URL from a fixed scheme and host, and refuse a target that is not a plain path. Signed-off-by: Yi-Ru Lin <deduce@gmail.com>
An entry that fails on its first read has sent nothing, so aborting failed a fetch GitHub could still have served. Signed-off-by: Yi-Ru Lin <deduce@gmail.com>
The store's goroutine removes its file after Save returns, so the test could look before it did. Signed-off-by: Yi-Ru Lin <deduce@gmail.com>
…aborts Signed-off-by: Yi-Ru Lin <deduce@gmail.com>
Aborting a broken hit relied on the client retrying. actions/checkout does; a workflow's own git fetch does not, so a cache fault failed a step GitHub would have served. Download the entry and check it first, so any failure goes to GitHub. That also bounds a stream that stalls, and closes a race between the lookup's timer and the first read. The check now verifies the pack's checksum, which catches an entry damaged in store or in transit, and refuses a packfile section with no pack, an ERR line inside one and an unknown sideband. Signed-off-by: Yi-Ru Lin <deduce@gmail.com>
git matches its proxy setting that way, so a URL written GitHub.com was sent here and refused, failing a fetch that works without the proxy. Signed-off-by: Yi-Ru Lin <deduce@gmail.com>
A 502 said nothing about its cause, and a failed blob request logged a URL holding its signature. Signed-off-by: Yi-Ru Lin <deduce@gmail.com>
…used Signed-off-by: Yi-Ru Lin <deduce@gmail.com>
Signed-off-by: Yi-Ru Lin <deduce@gmail.com>
Two tests could pass without observing what they claim, one of them flaky, and three comments still described the hit path before it spooled. Signed-off-by: Yi-Ru Lin <deduce@gmail.com>
|
babysit-pr · head |
|
babysit-pr · head |



Second part of the git checkout cache. Adds
firerunner git-proxy. Nothing starts it yet: the tier option, the CA forgithub.comand the AMI come next.What it does
A loopback proxy that git alone is pointed at, through
http.https://github.com/.proxy. It carriesCONNECT github.com:443and refuses anything else, terminates TLS, and handles three kinds of request:gitcacheaccepts: answered from the cache when it can be, otherwise forwarded, and the response stored.It uses the same Twirp API as
actions/cache, through the instance's nginx, withgit/<hash>as the key. Isolation is cache-server's, and tests run against the real server: another repository cannot read an entry, and the default branch cannot read a pull request's.Failure
The rule is that the cache must never fail a checkout GitHub would have served, and not every client retries:
actions/checkoutdoes, a workflow's owngit fetchdoes not. So an entry is downloaded to a temporary file and checked before its first byte goes out. That costs git the chance to start early, and it means any failure until then goes to GitHub.ERRline, sideband 3, truncated, bad checksum or non-pack responseGitHub answers 200 before it knows how a pack will go, so
responseCheckreads the stream, sideband by sideband, and verifies the pack's own checksum. The fourth case is the only one a client can see, and only a job writing to its own ref can cause it.A hit needs credentials GitHub just accepted
A hit skips GitHub's own check, so the same credentials must have listed refs for the same repository within 10 minutes. git always lists refs first.
This is correctness for an honest job, not a boundary. A job can read the leaf key and, with
sudo, replace the process. What a job cannot do is widen what cache-server gives it.Known limit: the proxy cannot tell a job's own repository from another it has access to. A fetch of private repository B from a job in repository A is stored under A. Reaching it from a later job in A needs the commit's ID, and it is the same exposure as that job saving B to
actions/cache. I would rather document this than add a config input.Verified
git http-backend: miss then hit with no upstream fetch; a damaged pack and a cache that is down being invisible; another commit's valid response refused by git, with the retry recovering.-race.scripts/mutate.shon about 35 guards. Surviving mutants led to tests for the host check, the content type, the request target and a pack's magic, and to removing two redundant conditions.git http-backendis.Not verified: EC2, a private repository with a real token, and nginx in the path. How long the first checkout waits for the job's identity also needs measuring: the 15 s lookup limit is a judgement, and if identity takes longer, a first step waits 15 s and still misses, which decides whether this pays.
For the wiring PR: the temporary directory shares the job's disk, and a dead proxy fails git, as git has no fallback.