Observability - #10
Merged
Merged
Conversation
Metrics, logs and traces, as native NixOS services rather than containers -
the modules exist and handle users, state directories and unit hardening, so a
compose file would only add moving parts.
Ports: grafana 3000, prometheus 9090, loki 3100, tempo 3200, and tempo's OTLP
receivers on 4317 (gRPC) and 4318 (HTTP). Loki's push API and the OTLP
receivers bind 0.0.0.0 and the firewall admits 172.16.0.0/12, because the point
is for Docker services on this host to reach them: a container talking to
host.docker.internal - which litellm.nix already maps to host-gateway - arrives
from the Docker bridge, and a 127.0.0.1 bind would be unreachable. Everything
else follows litellm.nix's policy of localhost, LAN and WireGuard only. Loki's
gRPC port stays closed; nothing off-host speaks it.
Grafana's datasources are provisioned rather than clicked in, so a rebuild is
enough to get a working instance. Its secret_key lost its default in NixOS
26.05 and is now required, so it is generated and encrypted in
secrets/grafana.enc.json and read through $__file{} to keep it out of the
world-readable store copy of grafana.ini. That file needs lucie's host age key
as well as the PGP key for sops-nix to decrypt at boot, so it joins litellm
under the .sops.yaml rule that carries both.
Prometheus only scrapes itself so far - the Docker services that will expose
/metrics get added as further jobs later.
Entire-Checkpoint: e7f8764a74b0
Tempo restarted every 30s (848 times in one boot) with: module=live-store err="failed to create shutdown marker directory: mkdir /var/tempo: read-only file system" DynamicUser=true implies ProtectSystem=strict, so StateDirectory=tempo is the only writable path. storage.trace already pointed under /var/lib/tempo, but Tempo 3.0 added modules whose paths still default under /var/tempo: live-store.shutdown_marker_dir /var/tempo/live-store/shutdown-marker live-store.wal.path /var/tempo/live-store/traces block-builder.wal.path /var/tempo/block-builder/traces backend-scheduler.local-work-path /var/tempo live-store failed on the first, distributor depends on live-store, so trace ingestion never came up. block_builder is inactive in single-binary mode but is set too - it fails identically if it ever activates. Also moves every port into a 320xx range. Grafana on 3000 and Prometheus on 9090 collide with anything else that wants the usual defaults, and lokiPort and tempoPort were briefly both 32030. Verified: tempo NRestarts=0, /ready returns 200, /var/tempo is never created, and /var/lib/tempo holds live-store/ traces/ wal/. Entire-Checkpoint: baef4153a49b
Registers the binfmt handlers and adds aarch64-linux to nix.conf's extra-platforms so this x86_64 box can build the devbox-arm image for the mac. Entire-Checkpoint: 197d69023cbc
Loki keeps everything forever by default: retention_period was 0s and compactor.retention_enabled false, so nothing ever deleted a chunk. Since /var/lib/loki sits on the root filesystem - already at 96% - an unbounded log store is a slow-motion outage. Deletion needs both halves. retention_period on its own is inert without retention_enabled, and the compactor refuses to start unless delete_request_store is set. Verified against loki 3.7.6 with this config: compactor reaches ACTIVE in the ring and logs "this instance has been chosen to run the compactor", with no config errors. Entire-Checkpoint: fd6175e529d5
Adds a file-backed dashboard provider pointing at ./dashboards, plus a first dashboard: error/warning/info/unclassified counts, log volume by level, warnings and errors by container, nginx edge HTTP status, and the matching log lines. allowUiUpdates is false, so the JSON in ./dashboards is the only source of truth and the UI serves it read-only. Editing in Grafana means exporting the JSON model back into the repo and rebuilding. Panels bind to the "loki" datasource uid already provisioned above. Entire-Checkpoint: a61237526802
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.