Skip to content

Apply Repository Timeout To Response Header Wait - #95

Merged
smartinov merged 1 commit into
mainfrom
feature/repository-response-header-timeout
Sep 10, 2026
Merged

smartinov merged 1 commit into
mainfrom
feature/repository-response-header-timeout

Conversation

@smartinov

Copy link
Copy Markdown
Contributor

Problem

Since 1.21.0 the repository client is built with keel's NewInternalHTTPClient (http command) and NewExternalHTTPClient (socket command). keel 0.29 hardcodes ResponseHeaderTimeout on those transports: 5s internal, 10s external. --repository-timeout only sets http.Client.Timeout, so the header timeout fires first.

A cold contentserverexport poll builds the export synchronously and takes longer than 5s. Every poll is cancelled with net/http: timeout awaiting response headers, the export keeps working on already-cancelled requests, and each poller adds another request every poll interval. The new pod never loads a repository, its startup probe stays 503, and it restarts until the export happens to finish for a still-connected client. This aborted two production rollouts on 2026-09-10.

Fix

newRepositoryHTTPClient builds the client for both commands and passes the repository timeout to HTTPClientWithResponseHeaderTimeout as well, before telemetry wraps the transport (keel panics otherwise).

Verification

TestNewRepositoryHTTPClientWaitsForSlowHeaders serves headers after 6s (internal) and 11s (external) with a 30s repository timeout. Against the previous construction it fails with the production error on both paths; with this change it passes. make tidy, make lint, make test clean locally.

@smartinov
smartinov merged commit 9589bd9 into main Sep 10, 2026
4 checks passed
@smartinov
smartinov deleted the feature/repository-response-header-timeout branch September 10, 2026 17:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant