RFC triage-agent - automated RCA for LAVA job failures - #1565
Conversation
Update qcs615-ride, qcs8300-ride, qcs9100-ride, sm8750-mtp, kaanapali-mtp, lemans-evk, monaco-evk and glymur-crd from fastboot to EFI boot and point them at the qcomflash images hosted on the Qualcomm storage/kernelci lab. Signed-off-by: Yogesh Lal <yogesh.lal@oss.qualcomm.com>
Add EFI-boot platform entries for hamoa-evk and purwa-evk and register both in every existing Qualcomm scheduler job (baseline and kselftest ramdisk jobs on lava-kci-qualcomm). Signed-off-by: Yogesh Lal <yogesh.lal@oss.qualcomm.com>
Enable new pipeline on the current qcom platform set (glymur-crd, hamoa-evk, kaanapali-mtp, kaanapali-qrd, lemans-evk, monaco-evk, purwa-evk, qcs615-ride, qcs6490-rb3gen2, qcs8300-ride, qcs9100-ride, sm8750-mtp, x1e80100): - baseline-arm64-preempt_rt-qualcomm (kbuild-gcc-14-arm64-preempt_rt) - kselftest-device-error-logs-ramdisk - kselftest-devices-exist-ramdisk - kselftest-devices-probe-ramdisk - kselftest-dmabuf-heaps-ramdisk - kselftest-exec-ramdisk Signed-off-by: Yogesh Lal <yogesh.lal@oss.qualcomm.com>
New module that hands failed/incomplete LAVA jobs to an external triage-agent for RCA. Best-effort: maybe_trigger never raises and is a no-op unless TRIAGE_ENABLE=true and TRIAGE_AGENT_URL is set. Dedup by node_id, optional per-hour rate cap, own thread pool, optional GitHub-issue and SMTP delivery of the returned Markdown report. Signed-off-by: Yogesh Lal <yogesh.lal@oss.qualcomm.com>
Hook TriageNotifier into async_job_submit after api.submit_results. No-op unless TRIAGE_ENABLE=true and TRIAGE_AGENT_URL is set. Also adds the CALLBACK_DRY_RUN env knob that skips storage upload and api.submit_results inside async_job_submit, so the flow can reach the notifier without touching production state. Test-only. Signed-off-by: Yogesh Lal <yogesh.lal@oss.qualcomm.com>
Cover the flow, trigger criteria, all TRIAGE_* / LAVA_TOKEN / CALLBACK_DRY_RUN env vars, and design notes on the two timeouts, the pool-full vs dedup ordering, and the in-flight counter. Signed-off-by: Yogesh Lal <yogesh.lal@oss.qualcomm.com>
|
Really like the direction as an RFC, my concern is implementation rather than the idea. kci-dev already provides this layer: the KernelCIClient library (#277) and the MCP server built on it and they're the sanctioned way to talk to KernelCI, fetching nodes/tests/logs and driving actions like retry/checkout (although the latter does need some more testing). This PR adds a second path beside it, the TriageNotifier glue in lava_callback plus the agent doing its own KernelCI/LAVA access, and I'd rather we didn't maintain two systems for the same thing. I'd push on is having it consume KernelCIClient (and the MCP tools) for its data and any follow-up actions, instead of reimplementing fetch/auth and wiring a bespoke notifier into the pipeline. Both the trigger-on-failure and the act-on-it steps fit the MCP model well, so it should integrate more cleanly there. The MCP can drive fetching nodes/tests/logs and driving actions like retry/checkout (although the latter I am testing, so it is experimental). |
|
Thanks @bhcopeland for review, my vision was something like submit job -> job executed (LAVA) -> Test failed -> lava_callback ->triage-agent -> share RCA And for that lava_callback was right point where I can link https://github.com/yogeshlal/triage-agent to triage the failure job and generate RCA for failed job and using that create issue/bug. |
Understood, that LGTM (i.e. thats the right place for that but I am not questioning that bit). My concern is for the one layer down. How the agent reaches KCI.
So keep lava_callback as the trigger and the LAVA-log fetch as-is. I just want the KernelCI interaction and the act-on-it side to go through kci-dev so we're not standing up a second integration path beside it. Bonus is the MCP setup then gets the same capability for free, which is where I think this gets genuinely powerful. |
|
@bhcopeland got your perspective , let me revisit the trigger point but I think we are clear on having triage agent like https://github.com/yogeshlal/triage-agent for log analysis and linking to know issues ? |
2 participants
triage-agentis a small FastAPI service that triage LAVA job. POST /{job_id} submits a run, GET /triage/{run_id} polls for the report. It fetches the LAVA log, drives the skills/kernel-triage skill through a LiteLLM tool-calling loop (Anthropic / OpenAI / Gemini / Ollama any LiteLLM provider), and writes a Markdown RCA with a filled-in applicability-gate table and a root-cause hypothesis.It plugs into kernelci-pipeline via the new TriageNotifier: when a LAVA job comes back fail or incomplete, lava_callback fires TriageNotifier.maybe_trigger, which POSTs to this service and polls until the report is ready. Optional GitHub issue and email delivery live on the pipeline side.