HISTORY.md

History

v5.13.0

The prometheus handler can serve the protobuf exposition, with created timestamps. increase() and rate() never count a counter's first sample: it is taken as the starting point. A series that starts at 5 and stays there shows an increase of 0, and a series that goes idle long enough for MetricTimeout to delete it loses its first update again when it comes back.

The handler now records when each counter and histogram series started from zero (its first update, or its first update after MetricTimeout deleted it) and publishes it as created_timestamp in the protobuf exposition (application/vnd.google.protobuf; proto=io.prometheus.client.MetricFamily; encoding=delimited). Prometheus is renaming created timestamps to start timestamps; the wire field is the same. A scraper with created (start) timestamp zero ingestion enabled (Prometheus created-timestamp-zero-ingestion, Alloy start_timestamp_zero_ingestion) writes a zero at that time, so the first update is counted.

Protobuf is served only to a scraper that ranks it above the text format in its Accept header. That is a scraper whose scrape protocols put PrometheusProto first, or a Prometheus with scrape_native_histograms or created-timestamp zero ingestion on, which puts it first by default: such a Prometheus switches to protobuf when the application upgrades, with no change to its configuration. Alloy puts it first by default only with scrape_native_histograms; start_timestamp_zero_ingestion alone does not change the order, so scrape_protocols has to list PrometheusProto first. Every other scraper keeps getting the text format. Sample timestamps are still left to the scraper. Gauges carry no created timestamp.

Both formats render the same metric and label names, so a Prometheus 3.x scraper that switches format keeps writing to the same series. On Prometheus 2.x, histogram buckets with integer bounds start new series: the scraper formats the bound of a protobuf bucket as le="1.0", while the text format writes le="1", and only 3.x normalizes the text value to match.

The text format now writes a histogram's le="+Inf" bucket from _count rather than from the total of the other buckets. The two differed after a NaN observation, which lands in no bucket, or after a change to the registered buckets, which resets them but not _count. Where a gauge and a histogram series render to the same name and labels (a gauge y_count next to a histogram y), the text format now always lists the histogram's first, as the protobuf format does; their order used to change between scrapes, so the value a scraper kept alternated between the two. Responses now carry Vary: Accept, Accept-Encoding.

The module now depends on github.com/prometheus/client_model.

v5.12.0

procstats now reads CPU and memory limits on cgroup v2. It only knew the cgroup v1 files (cpu.cfs_quota_us, cpu.cfs_period_us, cpu.shares, memory.limit_in_bytes) and dropped the 0::/path line that v2 writes to /proc/<pid>/cgroup. On a v2 host or pod the reads failed silently: quota, period and shares came back as 0, so CPU percent was computed against 1/NumCPU instead of the pod's limit, and the memory limit was reported as 0.

On v2, procstats now reads the quota and period from cpu.max, shares from cpu.weight (scaled so the v2 default of 100 maps to the v1 default of 1024), and the memory limit from memory.max. v1 and hybrid hosts read the same files as before.

Also changed:

• When no memory cgroup is found, the memory limit falls back to total system memory instead of 0. This applies on v1 too.
• linux.ProcCGroup.Unified is new and returns the v2 entry. linux.ParseProcCGroup and linux.ReadProcCGroup now include that entry (ID 0, empty name), so on hybrid hosts they return one more element than before. Lookup is unaffected.

v5.11.1

A call-time tag that overrides an engine tag no longer sends a tag twice. Since v4.7.0, SortTags has deduplicated tags in place and returned a shorter slice, but Engine.measureOne (behind Incr, Add, Set, Observe and the clock) and Engine.ReportAt dropped that result. With a derived engine such as eng.WithTags(stats.T("service", "a")) and a call that passes stats.T("service", "b"), the measure kept its original length, and the stale slots at the end repeated the last sorted tags. With three or more tags of the same name, an overridden value came back, too.

The prometheus handler published these as repeated label names, which Prometheus rejects together with the whole scrape (label name "…" is not unique: invalid sample). The datadog handler wrote the tag twice in the packet. Both now get each tag once. Which value wins a collision is unchanged: the call-time tag for Incr and friends, and the engine tag for Report.

v5.11.0 (September 23, 2026)

The prometheus handler's exposition has changed. Nothing fails to compile, but the series it publishes are different, so anyone already scraping this package will see renamed metrics and different staleness behaviour on upgrade. The datadog, influxdb, otlp and veneur handlers are untouched.

What changed, and why:

• Counters are now suffixed with _total. Incr("requests") published app_requests and now publishes app_requests_total. This is not only Prometheus naming convention: the OpenMetrics encoder keys the type line on the suffix, so a counter without it was published as unknown. A name that already ends in _total is left alone — as judged on the name once exposed, not as received: a field is joined to its scope by an _, so Incr("requests.total") reports the bare field total and already publishes app_requests_total, and . renders as _ besides. Where the suffix would land on a name a sibling field already occupies — a counter hits beside a field hits_total in the same scope — the suffix is dropped rather than the two being merged onto one series. svc.hits publishes svc_hits and svc.hits_total publishes svc_hits_total; merging them put two samples with identical labels under one # TYPE line, of which a scraper keeps one. Which names the handler has seen decides this — not the order they arrived in, and not which are still live, so a counter does not change the name it publishes under when its sibling stops being reported and is swept by the MetricTimeout cleanup. Queries and dashboards referring to the old names need updating. This includes the metrics the library reports about itself: go_version_value and stats_version_value become go_version_value_total and stats_version_value_total.

• Histograms always emit a +Inf bucket. The handler allocated exactly one bucket per registered boundary and never appended an overflow bucket, so observations above the highest boundary were counted in _sum and _count but landed in no bucket at all. histogram_quantile() returns NaN unless the highest bucket is +Inf, so any histogram whose registered boundaries did not already end in math.Inf(+1) could not be evaluated.

• Histograms with no registered boundaries fall back to prometheus.DefaultBuckets. stats.Buckets is empty by default and a miss returned a nil slice with no error, so such a histogram published _sum and _count with no _bucket series and nothing looked wrong. The defaults are the reference Prometheus client's, suited to latencies in seconds; they are a floor, not a substitute for choosing boundaries. This adds bucket series for histograms that previously published none: a histogram that published 2 series (_sum, _count) now publishes 14 (11 boundaries, +Inf, _sum and _count), per label set. This applies to every histogram the lookup finds nothing for; the registrations shipped by httpstats and netstats now resolve (below), so those get their own boundaries rather than these. Counters and gauges are unaffected — they publish one series each, as before.

• Bucket le labels sort numerically. They compared as raw strings, which put +Inf first and 10 ahead of 2.

• # TYPE is declared once per scope, not once per field name. The dedup discarded the scope, so same-named fields coming from different engine prefixes looked like repeats and every one after the first was published untyped. Sub-engines derived with WithPrefix exist precisely so subsystems can reuse short field names, so this fired readily. The handler now tracks every family it has declared rather than comparing each metric against the previous one, so a family interrupted in the sorted output is no longer declared twice. A repeated # TYPE makes the text format parser reject the entire exposition, so this also fixes scrapes that failed outright when a histogram shared a scope with a field sorting between its _bucket and _sum series — latency next to latency_size, say.

• Timestamps are no longer exposed. The field is optional, and a series carrying one opts out of Prometheus stale-marker handling — the scraper kept serving the last value for five minutes after a series stopped being exported. The scraper now assigns scrape time. MetricTimeout is unaffected.

• Bucket registrations made by httpstats and netstats now resolve. They never had. HistogramBuckets.Set splits its argument on the last ., but these registrations are written "http.message:body.bytes" — the : form splitMeasureField used before b45dd38 ("fix typo in splitMeasureField()", Aug 2019) changed the separator. The strings were never updated, so each has been keyed on a name nothing looks up since. The string form cannot express these keys at all: Set cuts at the last ., and every one of these fields carries one of its own (body.bytes, header.size, rtt.seconds). A second mismatch sat behind the first — the lookup uses the measure name after the engine prefix is attached, and a package registering from init() cannot know that prefix. This changes the series published for anyone using httpstats or netstats: those histograms published no _bucket series before, and would otherwise have taken the seconds-scale DefaultBuckets above — eleven boundaries no byte count can reach. They now publish the byte and duration boundaries those packages declare. procstats registers "go.memstats:gc_pause.seconds" for fields declared type:"gauge", so that entry remains inert and is left alone.

• New: HistogramBuckets.SetKey, SetUnprefixed and Lookup. SetKey takes the Measure and Field halves directly, reaching keys Set cannot express. SetUnprefixed registers a measure named without the prefix an engine will add to it, for packages registering from init(), and Lookup resolves those by dropping leading segments from the measure name after an exact lookup fails. Exact registrations always win, and only registrations made through SetUnprefixed are matched that way: matching every registration by suffix cannot tell a derived measure from an unrelated one ending the same way. Set is unchanged, and the otlp handler — which reads the registry directly — is untouched.

• New: Engine.SetBuckets(name, buckets...). Observe takes a name relative to the engine, while HistogramBuckets.Set needs the fully-qualified name, so registering buckets meant restating the engine prefix — and a mismatch was an ordinary map miss, indistinguishable from no registration at all. SetBuckets derives the key from the engine's own prefix, so callers pass the same string they pass to Observe. An ancestor can register for a sub-engine by naming the path to it — root.SetBuckets("db.latency", ...) covers root.WithPrefix("db").Observe("latency", ...) — so one init function covers a whole tree. Buckets are not inherited: a sub-engine resolves only what was registered for its own prefix. A Handler holding its own non-nil Buckets never reads the global registry, so SetBuckets has no effect on it. Buckets.Set is unchanged.

The minimum supported Go version is now 1.26. The golang.org/x/* modules (net, sys, sync, text) all declare go 1.26.0 as of their latest releases, and stats depends on them both directly and transitively through gRPC, so the whole module requires 1.26.

Update all dependencies. Most notably, gRPC moves to v1.83.2, which addresses GHSA-vp52-pcj8-j9qc, GHSA-2v4p-qf9q-27wj, and GHSA-qc2q-p7wx-3px3; the OpenTelemetry SDK and exporters move to v1.46.0.

v5.10.0 (July 30, 2026)

The minimum supported Go version is now 1.25. The otlp package, previously its own nested module, has been folded back into github.com/segmentio/stats/v5 so that go test ./..., go build ./..., and lint all cover it and dependency bumps only need to happen in one place. Because the OpenTelemetry SDK and gRPC both declare go 1.25.0, every consumer of stats now requires Go 1.25 and transitively pulls in the OTel SDK and gRPC, even if otlp is unused.

go get github.com/segmentio/stats/v5/otlp is no longer needed (or valid) as a separate module fetch; otlp is available as soon as you depend on github.com/segmentio/stats/v5.

v5.9.0 (June 15, 2026)

Apply 'go fix ./...' on the codebase. Several references to interface{} have been replaced by any; there should be no API or performance differences.

Add full OpenTelemetry OTLP exporter support with official SDK integration.

New Feature: OpenTelemetry OTLP Exporter (SDKHandler)

The otlp package now includes a production-ready SDKHandler that uses the official OpenTelemetry SDK with comprehensive support for modern observability requirements:

• Dual Transport Support: Both gRPC and HTTP/Protobuf protocols
• Environment Variables: Support for the standard OTEL_* environment variables including OTEL_EXPORTER_OTLP_ENDPOINT, OTEL_EXPORTER_OTLP_PROTOCOL (and the OTEL_EXPORTER_OTLP_METRICS_PROTOCOL override), and OTEL_RESOURCE_ATTRIBUTES
• Resource Detection: Host and process metadata plus environment attributes by default; cloud (AWS, GCP, Azure) and Kubernetes detection is opt-in via the contrib/detectors/* packages, with examples in the package README
• All Metric Types: Counter, Gauge, and Histogram with proper semantics
• Exponential Histograms: Optional support for exponential histogram aggregation with configurable bucket size and scale
• Temporality Configuration: Configurable metric temporality (cumulative or delta) with cumulative as the default for Prometheus compatibility
• Tag Preservation: Automatic conversion of stats tags to OpenTelemetry attributes
• Production Ready: Thread-safe instrument caching, proper context handling, and comprehensive error handling

Usage Example:

import (
    "context"
    "github.com/segmentio/stats/v5"
    "github.com/segmentio/stats/v5/otlp"
)

// Simple usage with environment variables
handler, err := otlp.NewSDKHandlerFromEnv(ctx)
if err != nil {
    log.Fatal(err)
}
defer handler.Shutdown(ctx)
stats.Register(handler)

// Or with explicit configuration
handler, err := otlp.NewSDKHandler(ctx, otlp.SDKConfig{
    Protocol:    otlp.ProtocolGRPC,
    EndpointURL: "http://localhost:4317",
})

Implementation Details:

• Gauges use native Float64Gauge instrument for instantaneous value recording
• Background context for metric recording to prevent context cancellation issues
• Efficient two-level locking pattern for instrument caching (read locks in hot path)
• Cumulative temporality by default (Prometheus-compatible)
• Comprehensive documentation including cloud resource detector examples
• SDKConfig.EndpointURL takes a full URL including the http:// or https:// scheme
• SDK defaults are used when unset (ExportInterval: 60s, ExportTimeout: 30s)

Deprecated:

• otlp.Handler is now deprecated in favor of otlp.SDKHandler (will be removed in v6.0.0)
• otlp.HTTPClient is now deprecated in favor of otlp.SDKHandler with ProtocolHTTPProtobuf (will be removed in v6.0.0)
• otlp.NewHTTPClient() is now deprecated (will be removed in v6.0.0)

The legacy Handler has been marked as Alpha since 2022 and has minimal to zero usage. Migration is straightforward - see deprecation notices in code for examples.

See the otlp package documentation for complete details and examples.

v5.8.0 (December 15, 2025)

When reporting go/stats versions, ensure that any user provided tags are included with the go-version and stats version reporting tags, to ensure better correlation on the Datadog side.

At the same time, don't report a timestamp with this metric, to avoid problems where Prometheus says that the metric is too old (we only report it one time).

v5.7.0 (November 24, 2025)

More lenient sanitization for tag values, which can contain commas, slashes, and other characters that are not allowed in a metric name.

v5.6.5 (October 14, 2025)

The sanitization process introduced in the v5.6.0 release did not properly handle characters in the extended Latin-1 supplement, e.g. "÷". This issue has been fixed in this release.

v5.6.4 (September 16, 2025)

Remove golang.org/x/exp from the list of dependencies, in favor of the "slices" standard library package. This bumps the minimum supported Go version to 1.23.

v5.6.3 (August 29, 2025)

The Datadog client should have faster performance, by copying less metric data before writing it to the socket.

v5.6.2 (July 10, 2025)

Remove outdated README content and add debugstats examples.

v5.6.1 (May 29, 2025)

Fix an error in the v5.6.0 release related to metric names with a longer buffer size.

v5.6.0 (May 27, 2025)

• In the datadog library: invalid characters in metric names, field names, or tag keys/values will be replaced with underscores. Accents and other diacritics will be removed (e.g. é will be replaced with 'e'). This change also improves performance of HandleMeasures by about 15-20%. #192

• External calls to github.com/segmentio/objconv were replaced by imports of github.com/segmentio/stats/v5/util/objconv, which is a fork of the library (it has since been archived). This allowed to to substantially reduce the surface we import.

  • influxdb: calls to objconv/json were replaced with github.com/segmentio/encoding/json (a library with substantially more production experience and test coverage). #193

• prometheus: Fix a deadlock from concurrent calls to collect() and cleanup(). Thank you Matthew Hooker for this contribution. #194

v5.5.0 (March 26, 2025)

• Add logic to replace invalid unicode in the serialized datadog payload with '\ufffd'.

v5.4.0 (February 21, 2025)

• Fix a regression in configured buffer size for the datadog client. Versions 5.0.0 to 5.3.1 would ignore any configured buffer size and use the default value of 1024. This could lead to smaller than expected writes and contention for the file handle.

v5.3.1 (January 2, 2025)

• Fix version parsing logic.

v5.3.0 (December 19, 2024)

• Add debugstats package that can easily print metrics to stdout.

• Fix file handle leak in procstats in the DelayMetrics collector.

v5.2.0

• go_version.value and stats_version.value will be emitted the first time any metrics are sent from an Engine. Disable this behavior by setting GoVersionReportingEnabled = false in code, or setting environment variable STATS_DISABLE_GO_VERSION_REPORTING=true.

v5.1.0

Add support for publishing stats to Unix datagram sockets (UDS).

v5.0.0 (Released on September 11, 2024)

In the httpstats package, replace misspelled http_req_content_endoing and http_res_content_endoing with http_req_content_encoding and http_res_content_encoding, respectively. This is a breaking change; any dashboards or queries that filter on this tag must be updated.