Repository navigation
fix(prometheus): map pending alerts to the pending status - #6897
Open
RishavTiwari25 wants to merge 1 commit into
Open
RishavTiwari25 wants to merge 1 commit into
RishavTiwari25 wants to merge 1 commit into
Conversation
Prometheus' /api/v1/alerts returns alerts that are still inside their 'for:' window with state 'pending'. STATUS_MAP had no entry for it, so they fell back to FIRING. Grafana and VictoriaMetrics already map 'pending' to AlertStatus.PENDING.
Contributor
|
No linked issues found. Please add the corresponding issues in the pull request description. |
|
|
Author
|
The only failing check is the linked-issue one: I could not find an existing issue for this bug. Happy to open one if you prefer, or the |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
When Keep pulls alerts from Prometheus (
PrometheusProvider._get_alerts, which readsGET /api/v1/alerts), every alert comes back as firing, including the ones Prometheus reports as"state": "pending".A pending alert is one whose rule matches but has not lasted for the rule's
for:duration yet. Prometheus itself does not notify on it. In Keep it shows up as firing right away, so workflows and notifications that trigger on firing alerts can run for a condition that may clear before it ever fires.Cause:
_format_alertmaps the state throughSTATUS_MAP, which only hasfiringandresolved. Anything else,pendingincluded, falls back toAlertStatus.FIRING.The Grafana and VictoriaMetrics providers already map
"pending"toAlertStatus.PENDING, so Prometheus was the odd one out.Fix
Add
"pending": AlertStatus.PENDINGtoPrometheusProvider.STATUS_MAP. Alertmanager webhooks only sendfiringandresolved, so the push path does not change.Tests
New
tests/providers/prometheus_provider/test_prometheus_format_alert.py, which feeds_format_alertthedataobject from/api/v1/alerts:test_pending_alert_is_not_reported_as_firingfails onmain(status isfiring) and passes with the fix.test_firing_alert_stays_firingchecks that firing alerts are unchanged.pytest tests/providers/prometheus_provider tests/providers/grafana_providergives 19 passed.ruff check(0.11.4, the pinned version) andruff format --checkare clean. I also triedtests/deduplication/test_deduplications.pylocally on Windows, but it fails the same way with and without this change (it needs the full app environment), so I couldn't use it as a check.