Repository navigation
prometheus: serve protobuf exposition with created timestamps - #240
Conversation
The handler now records when each series was created, at its first update, and serves the Prometheus protobuf exposition when a scraper ranks it first in its Accept header. Counters and histograms carry created_timestamp, so a scraper with start-timestamp zero ingestion writes a zero at that time and increase() counts a new series' first value instead of taking it as the baseline. A series re-created after MetricTimeout gets a new created time, so comebacks are counted too. Text 0.0.4 stays the default and is unchanged. Scrapers on their default protocols (Prometheus, Alloy) never rank protobuf first, so they keep receiving text. Encoding uses the generated types from prometheus/client_model, which depends only on google.golang.org/protobuf. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Review SummaryThis PR adds a protobuf output format to the Prometheus handler, which a scraper gets only by ranking it first in its Recommendation: APPROVE ✅ Approve Basis
Non-blocking findings[medium] Scrapers with some feature flags already put protobuf first —
|
sathvik09
left a comment
There was a problem hiding this comment.
Inline findings from the review summary comment. All are non-blocking.
| // | ||
| // Protobuf is served only when the scraper ranks it above every text type it | ||
| // accepts. Prometheus and Alloy list their scrape protocols with descending q | ||
| // values, and neither puts protobuf first unless configured to, so a scraper | ||
| // on its defaults keeps receiving text. Equal q values keep header order. |
There was a problem hiding this comment.
[medium] Scrapers with some feature flags already put protobuf first
Prometheus changes its default protocol order to protobuf-first when native histograms or created-timestamp zero ingestion are on (config/config.go:181, 598-608, 948-957 in v0.314.0). So "neither puts protobuf first unless configured to" is too broad: those scrapers switch format on upgrade with no config change. The PR body and HISTORY.md say the same thing.
Suggestion: Say this here, in the PR body and in HISTORY.md.
There was a problem hiding this comment.
Fixed in 04d508b. The docstring, HISTORY.md and the PR body now say that Prometheus also puts protobuf first by default when native histograms or created-timestamp zero ingestion are on, and that such a scraper switches to protobuf when the application upgrades, with no change to its configuration. The behaviour stays as it is: ranking by q is the right negotiation, and those scrapers are asking for protobuf.
There was a problem hiding this comment.
Prometheus changes its default protocol order to protobuf-first when either of these is on:
- native histograms
- created-timestamp zero ingestion
| switch mediatype { | ||
| case "application/vnd.google.protobuf": | ||
| if params["proto"] != "io.prometheus.client.MetricFamily" || params["encoding"] != "delimited" { | ||
| continue | ||
| } | ||
| pb = true | ||
| case "text/plain", "text/*", "*/*": | ||
| default: | ||
| continue |
There was a problem hiding this comment.
[medium] OpenMetrics is ignored when ranking formats
application/openmetrics-text falls through to default: continue, so openmetrics;q=0.5, protobuf;q=0.4, text/plain;q=0.3 serves protobuf. That breaks the docstring rule ("above every text type it accepts").
Suggestion: Add application/openmetrics-text to the text case, or reword the docstring. Either way, add this header to the negotiation test.
There was a problem hiding this comment.
Keeping the behaviour and fixing the docstring.
stats serves two formats: text 0.0.4, as before, and protobuf, which this PR adds. We picked protobuf over OpenMetrics deliberately, because it is the format Prometheus recommends for created timestamps (see the feature flag docs). OpenMetrics carries them as _created lines, which are error-prone, expensive to parse, and add extra series to storage. Enabling zero ingestion puts PrometheusProto first for the same reason.
Because stats does not serve OpenMetrics, acceptsProtobuf only ranks the two formats it does serve. For openmetrics;q=0.5, protobuf;q=0.4, text/plain;q=0.3, the scraper prefers protobuf over plain text, so it gets protobuf. Counting OpenMetrics as text would give that scraper plain text even though it ranked protobuf higher. This header only comes from an explicit scrape_protocols that lists OpenMetrics above PrometheusProto; Prometheus's default lists either leave protobuf out or put it first.
The docstring now says protobuf is served when the scraper ranks it above the text format. TestAcceptsProtobuf covers this header and expects protobuf. Fixed in 9ae25aa.
| store.mutex.RLock() | ||
|
|
||
| for key, entry := range store.entries { | ||
| var mtype dto.MetricType | ||
| switch entry.mtype { | ||
| case counter: | ||
| mtype = dto.MetricType_COUNTER | ||
| case gauge: | ||
| mtype = dto.MetricType_GAUGE | ||
| case histogram: | ||
| mtype = dto.MetricType_HISTOGRAM | ||
| default: | ||
| // WriteStats publishes only these three types. | ||
| continue | ||
| } | ||
|
|
||
| // Two entries can render to the same name, a.b and a_b for example. | ||
| // The text format declares the family once and lists both under it, | ||
| // so they are merged here too. | ||
| name := string(appendMetricScopedName(nil, entry.scope, store.exposedName(key, entry))) | ||
| fk := protoFamilyKey{name: name, mtype: entry.mtype} | ||
|
|
||
| f := byKey[fk] | ||
| if f == nil { | ||
| f = &protoFamily{family: &dto.MetricFamily{Name: proto.String(name), Type: mtype.Enum()}} | ||
| byKey[fk] = f | ||
| } | ||
|
|
||
| entry.mutex.RLock() | ||
| for _, states := range entry.states { | ||
| for _, state := range states { | ||
| f.series = append(f.series, protoSeries{ | ||
| labels: state.labels, | ||
| metric: state.protoMetric(entry.mtype), | ||
| }) | ||
| } | ||
| } | ||
| entry.mutex.RUnlock() | ||
| } | ||
|
|
||
| store.mutex.RUnlock() |
There was a problem hiding this comment.
[medium] Protobuf scrape holds the store read lock about 7x longer
All dto messages are built under store.mutex.RLock(). At 100k series that is 85 ms and 2.2M allocations, against 12 ms for collect(). A waiting writer (a new metric name) then blocks the read locks HandleMeasures needs.
Suggestion: Copy plain values under the lock, as collect() does, and build the messages after RUnlock.
There was a problem hiding this comment.
Fixed in 1ccab19. families() now copies each series into a plain struct under the locks (protoEntries), as collect() does, and builds the dto messages after releasing them. Bucket counts are copied into one shared backing array, so a histogram series costs no allocation under the lock.
Benchmarks at 100k series (90k counters and 10k histograms, Apple M4 Pro), now in protobuf_test.go:
| Time under the store lock | Allocs under the lock | |
|---|---|---|
collect() (text) |
28.8 ms | 15 |
families() before |
170 ms (the whole build) | 2.59M |
protoEntries() after |
19.6 ms | 79 |
The whole protobuf build still takes about the same time (~187 ms) and allocates more (162 MB against 108 MB, for the extra copy), but none of it blocks writers any more.
| h.Bucket = append(h.Bucket, &dto.Bucket{ | ||
| CumulativeCount: proto.Uint64(cumulativeCount), | ||
| UpperBound: proto.Float64(bucket.limit), | ||
| }) |
There was a problem hiding this comment.
[medium] Integer le values change on Prometheus 2.x scrapers
The scraper renders UpperBound 1 as le="1.0", while the text path writes le="1". Prometheus 3.x normalizes both to the same value; 2.x (v0.53.0 promparse) does not. So on 2.x, switching format starts new _bucket series, which contradicts "a scrape that switches format keeps the same series".
Suggestion: Limit the claim to Prometheus 3.x, or document the le change for 2.x.
There was a problem hiding this comment.
Fixed in c8f1cb3 by narrowing the claim. stats cannot fix this side: with protobuf the scraper formats UpperBound into the le label itself, and stats only sends a float.
The writeProtobuf docstring, HISTORY.md and the PR body now say series names are unchanged on Prometheus 3.x. They add that on 2.x, histogram buckets with integer bounds start new series (le="1" becomes le="1.0") when a scrape switches format, because only 3.x normalizes the text value to match. Grafana Alloy embeds Prometheus 3.x (v0.312.0 in Alloy v1.18.1, v0.314.0 in v1.21.3), so Alloy scrapers keep the same series.
| func TestProtobufMatchesTextNames(t *testing.T) { | ||
| now := time.Date(2017, 6, 4, 22, 12, 0, 0, time.UTC) | ||
|
|
||
| handler := &Handler{} | ||
| handler.HandleMeasures(now, | ||
| stats.Measure{Name: "http.server", Fields: []stats.Field{stats.MakeField("requests.total", 1, stats.Counter)}, Tags: []stats.Tag{stats.T("status.code", "200")}}, | ||
| stats.Measure{Name: "http.server", Fields: []stats.Field{stats.MakeField("hits", 1, stats.Counter)}}, | ||
| stats.Measure{Name: "http.server", Fields: []stats.Field{stats.MakeField("hits_total", 1, stats.Counter)}}, | ||
| ) | ||
|
|
||
| var text strings.Builder | ||
| handler.WriteStats(&text) | ||
|
|
||
| for _, mf := range scrapeProtobuf(t, handler, false) { | ||
| for _, m := range mf.Metric { | ||
| series := mf.GetName() | ||
| if len(m.Label) != 0 { | ||
| series += "{" + m.Label[0].GetName() + "=\"" + m.Label[0].GetValue() + "\"}" | ||
| } | ||
| if !strings.Contains(text.String(), "\n"+series+" ") && !strings.HasPrefix(text.String(), series+" ") { | ||
| t.Errorf("protobuf series %s not in text output:\n%s", series, text.String()) | ||
| } | ||
| } | ||
| } | ||
| } |
There was a problem hiding this comment.
[medium] The name-parity test covers only counters with one label
It reads only m.Label[0] and checks with strings.Contains. Histograms, le values, dotted scopes and series with several labels are never compared.
Suggestion: Parse both outputs into full label sets and compare them, including a histogram with integer bounds.
There was a problem hiding this comment.
Fixed in ac502e6. TestProtobufMatchesText replaces the old test. It expands both outputs into the series a scraper stores, compares them in both directions and requires the values to match:
- protobuf: a series per counter and gauge; per histogram,
_bucketfor each bound plus+Inffromsample_count, then_sumand_count - text: each line parsed into its name, full label set and value
le is compared as a number, matching Prometheus 3.x normalization (see the comment on le above).
The inputs cover a histogram with integer and fractional bounds (0.25, 1, 5), a dotted scope, two labels including one that needs sanitizing (status.code), a gauge, a second label set on the same counter, and the hits / hits_total collision.
I checked that the test fails on purpose-made bugs: dropping all but the first label, off-by-one bucket counts, and a wrong _sum. It caught all three.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ot serve Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
families held store.mutex.RLock while it built every dto message. A writer waiting on the store (a new metric name, or cleanup) then blocked every HandleMeasures call queued behind it for the length of the build. The series are now copied out under the locks, as collect does, and the messages are built after. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The parity test read only a series' first label and matched it as a substring, so histograms, gauges and further labels were never compared. It now expands both outputs into the series a scraper stores, with full label sets and values, and requires them to be equal. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
tobias-eyl-twilio
left a comment
There was a problem hiding this comment.
Review of correctness and production readiness for created timestamps over protobuf.
Verified OK: -race passes, and the lock order (store → entry → state) matches collect(). Accept negotiation is correct against the real Prometheus 3.x default headers, including the proto-first ones. With the real Prometheus 3.x parser, text and protobuf yield the same series (le normalization, _total, derived +Inf). Gauges carry no created time, a series re-created after MetricTimeout gets a new created time, go vet is clean, and v5.13.0 is the right next version.
Needs changes before merge: non-deterministic ordering of colliding series/families, and a late created after a zero-time first update (inline).
Not run: golangci-lint, Prometheus 2.x parsing, and a live scrape → Alloy → Mimir test.
| slices.SortFunc(f.series, func(a, b protoSeries) int { | ||
| switch { | ||
| case a.labels.less(b.labels): | ||
| return -1 | ||
| case b.labels.less(a.labels): | ||
| return 1 | ||
| } | ||
| return 0 | ||
| }) | ||
| f.family.Metric = make([]*dto.Metric, len(f.series)) | ||
| for i := range f.series { | ||
| f.family.Metric[i] = f.series[i].protoMetric(fk.mtype) | ||
| } | ||
| families = append(families, f.family) | ||
| } | ||
|
|
||
| slices.SortFunc(families, func(a, b *dto.MetricFamily) int { |
There was a problem hiding this comment.
Major: colliding series and families are emitted in random order.
byKey is a map, and both slices.SortFunc calls are unstable with no tie-break. When a.b and a_b render to the same name with identical labels (or a counter and a gauge share a name), the order changes from scrape to scrape. Prometheus keeps the first duplicate and drops the other, so the stored value flips between two different counters, which increase() reads as resets.
A local probe over 200 calls put value 1 first 169 times and value 2 first 31 times; family order flipped too. The text path is deterministic.
Suggested fix: slices.SortStableFunc plus a tie-break on scope, raw name and type for both series and families, and a test that scrapes the collision case repeatedly and asserts identical bytes.
There was a problem hiding this comment.
Fixed in 11e0021. Entries are now sorted by scope, raw name and type before merging, which is the order WriteStats uses. Series sort stably on labels, and when a counter and a gauge share a name, their families are ordered by their first entry. So protobuf keeps the same duplicate as text (a.b before a_b). TestProtobufCollisionOrder scrapes the collision case 100 times and asserts identical bytes, plus the text-matching order. It fails without the fix.
| // Two entries can render to the same name, a.b and a_b for example. | ||
| // The text format declares the family once and lists both under it, | ||
| // so they are merged here too. |
There was a problem hiding this comment.
Nit: this comment does not match the text path. In the collision case WriteStats declares # TYPE twice, because declared is keyed by {scope, rootName} (handler.go:183-220), not by the rendered name. Suggest rewording to say what protobuf does without claiming text parity.
There was a problem hiding this comment.
Fixed in 20b23c4. The comment now says only what protobuf does: colliding entries are merged into one family, so the name is declared once. The text-parity claim is gone.
| if state.created.IsZero() { | ||
| state.created = time | ||
| } |
There was a problem hiding this comment.
Major: a zero-time first update leaves created too late.
If the first update has time.IsZero() (ReportAt with a zero time, or a direct HandleMeasures), created stays zero, and the next update sets it to that later time. Example: Add 5 at zero time, then Add 1 at now, publishes value 6, created=now. The scraper writes a 0 at now after it already holds a sample of 5, which reads as a reset and double-counts.
Today the engine's only zero-time measure is the go_version gauge, so user ReportAt callers are the exposure. Suggested fix: track first-seen with a bool, and fall back to time.Now() when the measure time is zero. Worth a test.
There was a problem hiding this comment.
Fixed in 8d11553. When the series has no created time yet and the measure time is zero, created falls back to time.Now(). That is never zero, so created can't fall back into the unset state, and no extra bool is needed. TestCreatedTimestampZeroTime adds 5 at zero time, then 1 a minute later, and asserts value 6 with created at the first update. It fails without the fix (created lands on the second update).
| // The +Inf bucket is left out, as client_golang does: its count is | ||
| // sample_count, and the scraper adds the le="+Inf" series itself. |
There was a problem hiding this comment.
Minor: +Inf can differ between formats. Text writes le="+Inf" as the sum of bucket counts, while protobuf lets the scraper derive it from sample_count. After a NaN observation or a bucket rebuild (which zeroes buckets but not _count), the two disagree (2 vs 3 in a probe). The text-side drift predates this PR, but switching a target from text to protobuf will step +Inf. Options: emit an explicit +Inf bucket, or count NaN consistently. A NaN/rebuild test would pin whichever is chosen.
There was a problem hiding this comment.
Fixed in e1b2131 by making text follow the protobuf behaviour rather than the other way round. Text now writes le="+Inf" from _count, so +Inf == _count holds in both formats, including after a NaN or a rebuild. Sending an explicit +Inf in protobuf would have kept the formats in step, but both would then break that rule. Text output only changes in those two cases, where it was already inconsistent. Routing NaN into the +Inf bucket is no longer needed, because +Inf no longer depends on the bucket total. TestHistogramInfBucket covers a NaN observation and a bucket rebuild and checks +Inf in both formats. Without the fix, text gives 2 and 1 against _count=3.
| } | ||
|
|
||
| if protobuf { | ||
| _ = h.writeProtobuf(w) |
There was a problem hiding this comment.
Minor:
- The error from
writeProtobuf(and gzipClosevia defer) is dropped. A partial body is silently served; at least worth a comment, or logging if the handler has a logger. - Responses vary by
Acceptnow; considerVary: Acceptalongside the content type. - Memory: at 100k series, protobuf built ~155 MiB / 2.59M allocations per scrape vs ~126 MiB / 17 allocations for text in a local benchmark. Building and marshalling one family at a time would lower the peak.
There was a problem hiding this comment.
Addressed in 470da00:
- Dropped errors: added a comment instead of logging. The only way the write (or the gzip
Close) can fail is the connection failing. By then the status and part of the body have been sent, so there is nothing left to report the error with, and the scraper sees a response that ends early and records a failed scrape. The text path already behaves this way, andHandlerhas no logger. Adding one would be new public API for little gain. - Vary: the handler now sends
Vary: Accept, Accept-Encoding. The gzip choice depends on a request header too and never had one.scrapeProtobufasserts it. - Memory:
families()now returns the sorted families with their copied series, andwriteProtobufbuilds each family's messages just before writing it, so only one family is held at a time. I did not reuse messages between families, so the allocation count stays at about 2.59M. NewBenchmarkWriteProtobuf(build and marshal, 100k series, 10 runs, benchstat):
| before | after | |
|---|---|---|
| peak heap, 2 families | 111.7 MiB | 88.3 MiB (−21%) |
| peak heap, 100 families | 111.7 MiB | 98.6 MiB (−12%) |
| time | 287 / 210 ms | 284 / 205 ms |
| allocs | 2.59M | 2.59M |
Peak heap is HeapAlloc sampled at each write, above the level before the call. It includes garbage the GC has not collected yet, so read it as relative, not absolute.
| encoding=delimited`). A scraper with created-timestamp zero ingestion enabled | ||
| (Prometheus `created-timestamp-zero-ingestion`, Alloy | ||
| `start_timestamp_zero_ingestion`) writes a zero at that time, so the first | ||
| update is counted. | ||
|
|
||
| Protobuf is served only to a scraper that ranks it above the text format in | ||
| its `Accept` header. That is a scraper whose scrape protocols put | ||
| `PrometheusProto` first, or a Prometheus with native histograms or | ||
| created-timestamp zero ingestion on, which puts it first by default: such a |
There was a problem hiding this comment.
Wording suggestions:
- Alloy's
start_timestamp_zero_ingestiondoes not put protobuf first;scrape_protocolsmust be set explicitly. Readers may assume the option alone is enough. - In Prometheus 3.x the native-histograms feature flag is a no-op; the trigger is the
scrape_native_histogramsconfig option. - Prometheus is renaming created timestamps to start timestamps (wire field numbers unchanged); mentioning both names would help searchability.
There was a problem hiding this comment.
Thanks, all three applied in 0f58552:
- Alloy: the entry now says Alloy puts protobuf first by default only with
scrape_native_histograms, and thatstart_timestamp_zero_ingestionalone does not change the order, soscrape_protocolshas to listPrometheusProtofirst. This matches theprometheus.scrapedocs. - Native histograms: it names the
scrape_native_histogramsoption instead of the feature flag. - Naming: it says Prometheus is renaming created timestamps to start timestamps on the same wire field, and uses "created (start) timestamp" for the zero-ingestion option.
The entry also now covers the text le="+Inf" change and the Vary header from this round.
| // le is compared as a number. The scraper formats a protobuf bound itself, as | ||
| // le="1.0" where the text format writes le="1", and Prometheus 3.x normalizes | ||
| // the text value to the same form. | ||
| func TestProtobufMatchesText(t *testing.T) { |
There was a problem hiding this comment.
Test gaps:
- Comparing via maps can't catch duplicates or ordering (see the collision comment in protobuf.go).
- Nothing covers NaN observations, a bucket rebuild, a zero-time first update, colliding names, or a concurrent scrape under
-race. - The Accept header used here isn't the real Prometheus 3.x one (its protobuf entry starts at q=0.7 by default); adding the real headers as cases would harden
acceptsProtobuf. - Nit:
client_modelv0.6.3 is out.
There was a problem hiding this comment.
Addressed. Each gap and its test:
- Duplicates and ordering:
TestProtobufCollisionOrder(11e0021) scrapes the collision case 100 times, asserts identical bytes, and checks the order directly rather than through a map. - NaN and bucket rebuild:
TestHistogramInfBucket(e1b2131). - Zero-time first update:
TestCreatedTimestampZeroTime(8d11553). - Colliding names:
TestProtobufCollisionOrder(11e0021). - Concurrent scrape under
-race:TestProtobufConcurrentScrape(1ca7a60) scrapes while measures arrive and series expire. - Real Accept headers: added Prometheus'
acceptHeaderoutput as cases (1ca7a60): the defaults, protobuf first at q=0.8 (main, 7 protocols), and q=0.7 (3.0). The weights start at one above the number of protocols Prometheus knows, so they shift between releases. - client_model: bumped to v0.6.3 (1ca7a60).
…cally Entries came out of a map and both sorts were unstable, so where two entries rendered to the same name and labels the scraper kept a different sample on each scrape. Sort entries the way WriteStats does, sort series stably and break family ties on their first entry. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A zero created time means none is set, so a first update with a zero time left it unset and the next update set it after the series already held a value. Fall back to the wall clock instead. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A NaN observation lands in no bucket and a rebuild zeroes the buckets but not _count, so the bucket total could fall short of _count. The scraper derives +Inf from sample_count in the protobuf exposition, so the two formats disagreed. Write +Inf from _count in text too. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Build each family's messages just before writing it, so only one family is held at a time; peak heap at 100k series drops 12-21%. Send Vary: Accept, Accept-Encoding, since both headers choose the response, and say why write errors are ignored. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Add the Accept headers Prometheus 3.x sends by default and with protobuf first, a scrape racing measures and expiry, and bump client_model to v0.6.3. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… and Vary changes Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
tobias-eyl-twilio
left a comment
There was a problem hiding this comment.
Re-review at 164519f. Every earlier finding is fixed, and I verified each fix by reverting it locally: the new tests (TestProtobufCollisionOrder, TestCreatedTimestampZeroTime, TestHistogramInfBucket, the concurrent scrape) each fail without it.
Verified:
go build,go vet,go test -race ./prometheus/...(-count=10on the new tests),go mod verify,go mod tidy -diffand gofmt are clean.- 500 protobuf scrapes gave byte-identical output.
+Inffrom_countcan never fall below a finite bucket, and NaN handling matches client_golang.- Lock order is unchanged (store → entry → state), and no lock is held while writing.
Varyis set on both formats.- HISTORY matches the Prometheus 3.15.1 and Alloy 1.21.3 sources.
Memory at 100k series: peak heap dropped about 20%, to ~94 MB (text is ~67 MB). Allocations are still 2.59M per scrape.
The remaining comments are minor or nits and don't block merging. Leaving write errors unlogged, with the comment explaining why, is fine.
| // value alternate between two different series, and increase() would | ||
| // read every switch as a reset. | ||
| entries := store.protoEntries() | ||
| slices.SortFunc(entries, func(a, b protoEntry) int { |
There was a problem hiding this comment.
Minor: one collision case still keeps a different series in protobuf than in text. Histogram series sort under the histogram's root name, but text sorts _count/_sum/_bucket under their full names.
- A gauge
x.countnext to histogramx: text keeps the gauge'sx_count(70), and protobuf keeps the histogram'sx_count(1). - A gauge
y_countnext to histogramy: the text output itself changed in 132 of 500 scrapes in a probe.sort.Sort(byNameAndLabels)(handler.go:216,metric.go:552) is unstable and ties on scope/name/labels after map iteration. That part predates this PR.
Suggested fix: add a type-then-raw-name tie-break to byNameAndLabels.Less, and either order protobuf histograms by their sub-series names or document this edge case.
There was a problem hiding this comment.
Fixed in 5fcbb1a. byNameAndLabels.Less now breaks ties on the root name and then the type, so the text format always lists the histogram's y_count first, as protobuf does. TestHistogramCollisionOrder covers it and failed on the first run without the change. For x.count next to histogram x I documented the difference at the protobuf sort instead: matching it would mean reordering the text format for every metric, and each format already keeps the same series on every scrape.
| if t.IsZero() { | ||
| t = time.Now() | ||
| } | ||
| state.created = t | ||
| } |
There was a problem hiding this comment.
Nit: the fallback overwrites t, so a zero-time first update also sets state.time to now, while later zero-time updates set it back to zero. Using a separate local for the created time would keep state.time consistent:
if state.created.IsZero() {
created := t
if created.IsZero() {
created = time.Now()
}
state.created = created
}
state.time = tThere was a problem hiding this comment.
Fixed in 516b501 with a separate created local, so state.time stays zero and expiry treats the first zero-time update like the rest. TestZeroTimeUpdateKeepsZeroTime covers it and fails without the change.
| { | ||
| // Prometheus' acceptHeader output for DefaultScrapeProtocols. Its | ||
| // weights start at one above the number of protocols it knows, so | ||
| // they shift between releases: q=0.8 on main, q=0.7 on 3.0. |
There was a problem hiding this comment.
Nit: the version labels look swapped. Prometheus v3.15.1's ScrapeProtocolsHeaders has 6 entries (it added OpenMetrics 2.0.0), so its weights start at q=0.7. The q=0.7 string is therefore current main/3.15, not 3.0, and 3.0 would start lower (I couldn't confirm 3.0's exact header). The behaviour under test is fine; only the comments and case names need fixing.
There was a problem hiding this comment.
Right, thanks. acceptHeader starts at one above the number of known protocols: 5 on 3.0–3.14 (q=0.6), 6 on 3.15 (q=0.7), and 7 on main (q=0.8). Fixed in 4b983a7: the cases are now named main and 3.15, and I added 3.0's exact header from the v3.0.0 source, which starts at q=0.6 and appends escaping=allow-utf-8 to every protocol, protobuf included. It still picks protobuf.
|
|
||
| require ( | ||
| github.com/mdlayher/taskstats v0.0.0-20241219020249-a291fa5f5a69 | ||
| github.com/prometheus/client_model v0.6.3 |
There was a problem hiding this comment.
Nit: the bump from v0.6.2 to v0.6.3 came in with test commit 1ca7a60 and no stated reason. The only change in v0.6.3 is a newer go/protobuf requirement, and nothing here uses it. Either drop the bump or say why in the commit.
There was a problem hiding this comment.
Dropped in f522143. There was no reason for it: v0.6.3's generated Go code is the same as v0.6.2's, and nothing here needs it.
A gauge "y_count" and the _count series of a histogram "y" tie on scope, name and labels. sort.Sort is not stable and metrics come out of a map, so the text format listed them in a different order between scrapes, and the value a scraper kept alternated between the two. Break the tie on the root name, then the type, which lists the histogram first as the protobuf format already does. Document the collision the two formats still resolve differently: a gauge "x.count" next to a histogram "x". Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… time A zero-time first update took time.Now() as its created time, but the fallback overwrote t, so state.time was set to now as well. Later zero-time updates set it back to zero, and MetricTimeout expiry reads it, so the first update kept a series alive that the others would not. Use a separate local for the created time. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
acceptHeader starts its weights at one above the number of scrape protocols Prometheus knows: 5 on 3.0 to 3.14, 6 on 3.15, and 7 on main. The q=0.7 header is therefore 3.15, not 3.0. Rename the cases, and add 3.0's header, which starts at q=0.6 and appends escaping=allow-utf-8 to every protocol. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The bump to v0.6.3 came in with a test commit and nothing needs it: its generated Go code is the same as v0.6.2's. Requiring only what is used keeps consumers of stats from being moved to v0.6.3 for nothing. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
increase()never counts the first sample of a series, because that sample is taken as the series' baseline. The handler creates a series on its first update and deletes it afterMetricTimeout, so sparse, high-cardinality counters lose their first value again every time a series is created. Prometheus fixes this with created timestamps: the target publishes when each series started, and a scraper with start-timestamp zero ingestion writes a0at that time. This PR has the handler publish them.What changes:
metricState.createdis set on a series' first update. A first update with a zero time uses the wall clock instead, because a zero created time means none is set. A series re-created afterMetricTimeoutgets a new one. A histogram bucket rebuild leaves it unchanged, because_sumand_countkeep counting.ServeHTTPserves the protobuf exposition (application/vnd.google.protobuf; proto=io.prometheus.client.MetricFamily; encoding=delimited) when theAcceptheader ranks it above every text type. Counters and histograms carrycreated_timestamp; gauges don't. Gzip works for both formats. Responses carryVary: Accept, Accept-Encoding, since both headers choose the response. The messages are built after the store lock is released, one family at a time.WriteStatsrenders them. That covers the_totalsuffix,exposedNamecollisions and invalid-byte replacement, so a Prometheus 3.x scrape that switches format keeps the same series. On Prometheus 2.x, histogram buckets with integer bounds start new series (le="1"becomesle="1.0"), because the scraper formats a protobuf bound itself and only 3.x normalizes the text value to match. Two entries that render to the same name (a.banda_b) are merged into one family and ordered asWriteStatsorders them, so a scraper keeps the same duplicate from either format on every scrape. Histograms leave out the+Infbucket, as client_golang does; the scraper derives it fromsample_count.le="+Inf"from_countinstead of the bucket total, so+Inf == _countholds in both formats. The two used to differ after a NaN observation, which lands in no bucket, or a bucket rebuild, which zeroes the buckets but not_count.+Inffix above. A scraper receives protobuf only when it ranks it first: a scrape configured withPrometheusProtofirst, or a Prometheus with native histograms or created-timestamp zero ingestion on, which puts it first by default. Such a Prometheus switches to protobuf when the application upgrades, with no change to its configuration. Every other scraper keeps receiving text.New dependency:
github.com/prometheus/client_modelv0.6.3, for the generatedMetricFamilytypes. Its only dependency isgoogle.golang.org/protobuf, which stats already uses.Tests:
Varyheader.MetricTimeout, and a zero-time first update.+Infequal to_countin both formats after a NaN and after a bucket rebuild.go test -race ./...andgolangci-lintpass.Benchmarks:
BenchmarkCollect,BenchmarkProtoEntriesandBenchmarkWriteProtobufinprotobuf_test.go, at 100k series (90k counters, 10k histograms).BenchmarkWriteProtobufspreads them over 2 families and over 100. The protobuf path copies series under the store lock, ascollect()does, then builds and writes one family at a time after releasing it. Ten runs each, compared withbenchstaton an Apple M4 Pro:collect())The first version built the messages under the lock, holding it for 169 ms with 2.59M allocations. Building and writing one family at a time instead of all of them first lowers peak heap from 112 MiB to 88 MiB (2 families) and 100 MiB (100 families), with time and allocations unchanged. Peak heap is
HeapAllocsampled at each write, above its level before the call. It includes garbage not yet collected, so compare it between rows rather than reading it as absolute.🤖 Generated with Claude Code