validate: audit Maestro scheduling against the pipeline forecast - #1541
validate: audit Maestro scheduling against the pipeline forecast#1541nuclearcat wants to merge 2 commits into
Conversation
kci config forecast (kernelci-core) predicts which kbuilds each
tree/branch checkout should produce and which test jobs each kbuild
should trigger, but nothing verified that Maestro actually scheduled
them. Add a daily audit, following the same pattern as the dashboard
validation (workflow + supplementary script in tests/):
* tests/validate_forecast.py fetches checkout nodes old enough for
their whole job tree to have settled (24h, over a 24h window),
rebuilds the forecast graph with each checkout's real kernel
revision, and compares expected vs actual kbuild children, then
expected vs actual test jobs per (job, platform) for every kbuild
that passed. A missing child is only a finding when its parent
succeeded: a failed checkout produces no kbuilds and a failed
kbuild triggers no test jobs, so those absences are expected.
Nodes that exist but are not forecast are classified before being
flagged: externally-owned jobs (e.g. owner=qualcomm scheduling on
runtimes outside our config) and runner-created children (the
per-suite kselftest build nodes kbuild.py attaches) are counted as
informational instead of reported as drift.
* .github/workflows/validate-forecast.yml runs it daily, rewinding
config/ to its git state at the start of the validated window so
jobs added to the config since the checkout ran are not falsely
reported as missing. kernelci-core is checked out from the branch
carrying the forecast graph support until it lands upstream.
A trial run against production caught a real gap noticed by @bhcopeland,
missing pull lab jobs, while the same job ran on the same platforms
two days earlier.
Signed-off-by: Denys Fedoryshchenko <denys.f@collabora.com>
| continue | ||
| if not node_ok(kbuild): | ||
| # A failed kbuild legitimately triggers nothing; only count it. | ||
| report["skipped_failed_kbuilds"].append(describe_node(kbuild)) |
There was a problem hiding this comment.
So we will surpress jobs that won't run, but will incomplete builds also be dropped? Might be worth a seperate counter for the other uses cases than just their children.
| parent_ok=node_ok(checkout), | ||
| ) | ||
|
|
||
| for kbuild in kbuilds: |
There was a problem hiding this comment.
I think this will miss 3 levels down, as it will check only its children groups.
I.e.
# Execute nipa-update on the blktests-ddp-x86 node
# complete with the blktests-ddp-x86 event
- job: nipa-update
event:
<<: *job-event
name: blktests-ddp-x86
runtime:
type: shell
There was a problem hiding this comment.
nipa is a bit weird job btw, it must be deprecated as seems netdev changed things since long time
| python -m pip install --upgrade pip | ||
| python -m pip install click requests cloudevents toml pyyaml jinja2 | ||
|
|
||
| - name: Validate scheduled jobs against forecast |
There was a problem hiding this comment.
We should probably have a check here so MISSING_KBUILD does spam. I think for example gcc-15 changes, will cause this to be quite noisey until we get prod up-to-date. And in this setting its merged, but not yet deployed. We probably want to catch this differently, rather than noticing its a scheduler bug or something.
| self.session = requests.Session() | ||
| adapter = requests.adapters.HTTPAdapter( | ||
| max_retries=requests.adapters.Retry( | ||
| total=3, backoff_factor=2, status_forcelist=[500, 502, 504] |
There was a problem hiding this comment.
We should probably add the other status codes here, 503, and 429.
| ref=$(git rev-list -1 --before="${hours} hours ago" HEAD) | ||
| if [ -n "$ref" ]; then | ||
| echo "Using config from $ref ($(git log -1 --format='%ci %s' "$ref"))" | ||
| git checkout "$ref" -- config |
There was a problem hiding this comment.
We probably want to add rm -rf config before this. Also up in line 3 we have python3 commands going on, but setup python is down on like 65. I would put setup python at the top before these steps. (or we use some sort of awk command here.
| # TODO: switch to kernelci/kernelci-core@main once the forecast | ||
| # graph support (evaluate_forecast_entry) lands upstream. | ||
| repository: nuclearcat/kernelci-core | ||
| ref: codex/forecast-missing-job-debug |
There was a problem hiding this comment.
I think we should pin this to a commit sha, until it lands upstream, rathar than a outside repo
|
Thanks for putting this together @nuclearcat this is a good approach to catch this going forward. I wonder if we should add something around the if: failure() step (like create github issue, or some sort of of discord message)? |
I am planning that after we make sure there is no unnecessary noise (as we already have too much from -status latency warnings, and it must be reduced). |
nuclearcat
added
the
staging-skip
2 participants
kci config forecast (kernelci-core) predicts which kbuilds each tree/branch checkout should produce and which test jobs each kbuild should trigger, but nothing verified that Maestro actually scheduled them. Add a daily audit, following the same pattern as the dashboard validation (workflow + supplementary script in tests/):
tests/validate_forecast.py fetches checkout nodes old enough for their whole job tree to have settled (24h, over a 24h window), rebuilds the forecast graph with each checkout's real kernel revision, and compares expected vs actual kbuild children, then expected vs actual test jobs per (job, platform) for every kbuild that passed. A missing child is only a finding when its parent succeeded: a failed checkout produces no kbuilds and a failed kbuild triggers no test jobs, so those absences are expected.
Nodes that exist but are not forecast are classified before being flagged: externally-owned jobs (e.g. owner=qualcomm scheduling on runtimes outside our config) and runner-created children (the per-suite kselftest build nodes kbuild.py attaches) are counted as informational instead of reported as drift.
.github/workflows/validate-forecast.yml runs it daily, rewinding config/ to its git state at the start of the validated window so jobs added to the config since the checkout ran are not falsely reported as missing. kernelci-core is checked out from the branch carrying the forecast graph support until it lands upstream.
A trial run against production caught a real gap noticed by @bhcopeland, missing pull lab jobs, while the same job ran on the same platforms two days earlier.