Skip to content

Docker container labels with '/' in the key break Prometheus exposition parsing entirely #439

Open

Description

@jaimeib

Description

When running scaphandre with --containers enabled, Docker container labels are exposed as Prometheus labels named container_label_<sanitized-key>. The key sanitization in ProcessTracker (src/sensors/utils.rs) escapes -, ., : and by replacing them with _, but does not escape /.

Container labels using Kubernetes/OCI-style namespaced keys (e.g. nuclio.io/function-spec, or any domain/key convention) therefore produce a Prometheus label name containing a literal /, which is not a valid character in a Prometheus label name.

This isn't just a cosmetic issue with that one label: it breaks the Prometheus text exposition format parser for scrapers that enforce it strictly (we hit this with Grafana Alloy). The scrape fails to parse the entire /metrics response, so every metric from that target is lost, not just the one with the bad label.

Example

A container with the Docker label:

nuclio.io/function-spec: {...}

Currently produces:

container_label_nuclio_io/function_spec="{...}"

(the . is escaped to _, but the / is not) — this is invalid Prometheus exposition syntax and causes the whole scrape to fail parsing.

With the fix, it produces:

container_label_nuclio_io_function_spec="{...}"

which parses correctly.

Fix

Add / to the existing escape_list alongside -, ., : and in ProcessTracker's container label handling. PR: #440

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions