WIP: feat: support CISA KEV catalog ingestion and expose known-exploited data

[waldemar-kindler]

Motivation:
Trustify ingests applicability data (VEX/CSAF product status: affected / not affected / fixed) and severity (CVSS), but carries no curated signal for whether a vulnerability is being actively exploited in the wild. The CISA KEV catalog is the authoritative source for that signal and complements the existing data: applicability narrows findings to what affects you; exploitation status ranks them by real-world urgency (and carries BOD 22-01 remediation deadlines). See #2545 for the full design discussion.

Relation to Exploit Intelligence (#2534): EI answers "is this CVE exploitable in this specific product?" via an external analysis service, per SBOM+CVE job. KEV answers "is this CVE exploited in the wild, globally?" from a public catalog — no external service, no product context, no auth. The signals are complementary.

Changes

Ingestor

• New known_exploited_vulnerability table (m0002300), keyed by (source, cve_id) so additional catalog providers (e.g. VulnCheck KEV) can be supported later. Intentionally no FK to vulnerability: catalog entries and CVE documents can be ingested in any order; consumers join on vulnerability.id.
• Format::CisaKev with JSON content detection (catalogVersion + vulnerabilities), wired through the DocumentDetector.
• KevLoader with full-sync semantics: each load atomically replaces the source's entry set, so entries removed from KEV (this has happened) are removed from the database. Unparsable dates are dropped with a warning instead of failing the catalog.

Importer

• New kev importer kind fetching the KEV JSON feed, skipping unchanged content via the Last-Modified header (same pattern as the CWE catalog importer), with a 64 MiB download cap.
• Disabled-by-default sample importer, daily period.

API

• Vulnerability details now include known_exploited entries (source, dateAdded, dueDate, requiredAction, knownRansomwareCampaignUse).

Validation: exercised end-to-end against the live CISA feed (1,656 entries): import, Last-Modified skip on re-run, and the full-sync semantics confirmed against a real-world removal (CISA has since dropped CVE-2025-6965 from the catalog, and the entry disappears on reload as intended).

Open questions for reviewers:

1. Schema: known_exploited_vulnerability keyed by (source, cve_id) — source-qualified for future providers. Reasonable, or would you prefer a CISA-only shape for now?
2. No FK to vulnerability (ingestion-order independence, same as vulnerability.cwes handling) — acceptable?
3. Full-sync (delete + reinsert per source, in one transaction) instead of upsert-with-tombstones — acceptable for a ~1,300-row catalog?
4. Naming: known_exploited on the API vs. the new Exploit Intelligence module — happy to rename if this is too confusable.

Non-goals (per #2545)

• Risk-score adjustments, additional providers, dedicated UI views, and list-endpoint filtering by KEV membership are follow-ups. Happy to add list filtering in this PR if preferred — I kept it out to keep the data model reviewable first.

🤖 Generated with Claude Code

Summary by Sourcery

Introduce ingestion and exposure of known-exploited-vulnerability catalog data, starting with the CISA KEV feed, and surface catalog entries on vulnerability detail responses.

New Features:

• Add support for detecting and ingesting CISA KEV JSON catalog documents into a dedicated known_exploited_vulnerability table.
• Expose known-exploited catalog entries on vulnerability detail API responses, including source, dates, required action, and ransomware campaign use.
• Provide a configurable KEV importer that periodically downloads the catalog feed, with last-modified-based continuation and size limits.
• Include end-to-end coverage and sample importer configuration for KEV catalog ingestion and vulnerability enrichment.

Enhancements:

• Extend importer and ingestor format enums and document detection to recognize KEV catalogs via a dedicated cisakev format.
• Define a reusable KnownExploitation API model with ISO 8601 date serialization and database lookup helper.
• Add database migration and entity definitions for the known_exploited_vulnerability table keyed by catalog source and CVE ID.

Tests:

• Add unit, integration, and e2e tests covering KEV catalog ingestion, idempotency, entry removal behavior, catalog source overrides, and API exposure of known-exploited data.

Chores:

• Register the new KEV importer in sample data and adjust importer list expectations.

[sourcery-ai]

Reviewer's Guide

Adds end-to-end support for ingesting the CISA Known Exploited Vulnerabilities (KEV) JSON catalog into a new normalized table and exposes catalog entries on the vulnerability details API, including importer configuration, ingestion pipeline wiring, schema/migration, and tests.

Sequence diagram for KEV catalog ingestion pipeline

sequenceDiagram
    participant ImportRunner
    participant KevWalker
    participant CisaKevServer
    participant IngestorService
    participant KevLoader
    participant KnownExploitedVulnerabilityEntity

    ImportRunner->>KevWalker: run_once_kev_catalog(kev_catalog, continuation)
    KevWalker->>CisaKevServer: reqwest::get(source)
    CisaKevServer-->>KevWalker: known_exploited_vulnerabilities.json
    KevWalker->>IngestorService: ingest(body, Format::CisaKev, labels, None, Cache::Skip, tx)
    IngestorService->>KevLoader: load(labels, KevCatalog, digests, tx)
    KevLoader->>KnownExploitedVulnerabilityEntity: delete_many().filter(Source.eq(catalog_source))
    KevLoader->>KnownExploitedVulnerabilityEntity: insert_many(batch.chunked())
    KevLoader-->>IngestorService: IngestResult
    IngestorService-->>KevWalker: IngestResult
    KevWalker-->>ImportRunner: LastModified continuation

Loading

Sequence diagram for exposing KEV data on vulnerability details

sequenceDiagram
    actor Client
    participant VulnerabilityService
    participant KnownExploitation
    participant KnownExploitedVulnerabilityEntity

    Client->>VulnerabilityService: VulnerabilityDetails::from_vulnerability_id(id, tx)
    VulnerabilityService->>KnownExploitation: find_by_vulnerability(id, tx)
    KnownExploitation->>KnownExploitedVulnerabilityEntity: Entity::find().filter(CveId.eq(id)).all(tx)
    KnownExploitedVulnerabilityEntity-->>KnownExploitation: Vec<Model>
    KnownExploitation-->>VulnerabilityService: Vec<KnownExploitation>
    VulnerabilityService-->>Client: VulnerabilityDetails{ known_exploited, ... }

Loading

Entity relationship diagram for vulnerability and known-exploited-vulnerability

erDiagram
    vulnerability {
        string id PK
    }

    known_exploited_vulnerability {
        string source PK
        string cve_id PK
        date date_added
        date due_date
        string required_action
        string known_ransomware_campaign_use
    }

    vulnerability ||--o{ known_exploited_vulnerability : cve_id

Loading

File-Level Changes

Change	Details	Files
Introduce a persistent data model and API representation for known-exploited vulnerability catalog entries and wire them into vulnerability details.	Add known_exploited_vulnerability SeaORM entity and migration with (source, cve_id) composite primary key, nullable catalog metadata fields, and index on cve_id. Define KnownExploitation DTO in the vulnerability model, with ISO-8601 date (time::Date) serialization and conversion from the entity model. Extend VulnerabilityDetails to include a known_exploited array populated via KnownExploitation::find_by_vulnerability, and update OpenAPI schemas to document KnownExploitation and the new field. Add unit and service tests validating KnownExploitation JSON date serialization/deserialization and that vulnerability details expose KEV-backed known_exploited entries correctly.	entity/src/known_exploited_vulnerability.rs migration/src/m0002300_create_known_exploited_vulnerability.rs modules/fundamental/src/vulnerability/model/mod.rs modules/fundamental/src/vulnerability/model/details/mod.rs modules/fundamental/src/vulnerability/service/test.rs openapi.yaml
Implement KEV catalog ingestion in the ingestor pipeline, including format detection, JSON schema, loader behavior, and tests.	Extend ingestor Format enum and DocumentDetector to recognize CisaKev JSON documents using catalogVersion and vulnerabilities keys and parse them into a KevCatalog structure. Add KevCatalog and KevEntry JSON schema matching the CISA KEV feed and a KevLoader that performs full-sync semantics per catalog source (delete-then-insert), chunked inserts, and digest-based IngestResult IDs. Implement catalog source selection via a LABEL_CATALOG label with SOURCE_CISA default, and allow multiple catalog sources to coexist and be independently resynced. Implement robust date parsing for catalog dates, dropping unparsable dates with warnings instead of failing ingestion, and normalize empty CWE lists to None. Add integration-style tests for ingestion behavior: basic ingest, idempotency, handling of unparsable dates, removal of entries between catalog versions, and catalog label overrides.	modules/ingestor/src/service/format.rs modules/ingestor/src/service/detect.rs modules/ingestor/src/service/mod.rs modules/ingestor/src/service/kev/mod.rs modules/ingestor/src/service/kev/schema.rs modules/ingestor/src/service/kev/test-data/* e2e/kev.hurl
Add a KEV importer configuration and runner that periodically fetches the KEV feed, enforces a size limit, performs last-modified based continuation, and feeds documents into the ingestor.	Introduce KevImporter model with CommonImporter flattening, default source URL (DEFAULT_SOURCE_KEV_CATALOG), and optional catalog identifier, including OpenAPI and JSON schema wiring. Extend ImporterConfiguration enum, Deref/DerefMut implementations, and ImportRunner dispatch to support the Kev importer kind. Implement KevWalker in the importer runner that downloads the catalog via reqwest, tracks Last-Modified as a continuation token, skips unchanged catalogs, enforces a 64 MiB size limit, builds appropriate labels (including catalog override), and invokes the ingestor with Format::CisaKev inside a DB transaction. Ensure importer reports are built and errors recorded without advancing the continuation token, and add run_once_kev_catalog orchestration that returns RunOutput with updated continuation. Register a disabled-by-default KEV importer in sample_data with a daily period and adjust tests to account for the extra importer.	modules/importer/src/model/mod.rs modules/importer/src/model/kev.rs modules/importer/src/runner/mod.rs modules/importer/src/runner/kev/mod.rs modules/importer/src/runner/kev/walker.rs modules/importer/schema/importer.json xtask/schema/generate-dump.json server/src/sample_data.rs
Update API and test suites to validate KEV ingestion and exposure via REST endpoints.	Extend OpenAPI importer source enums to include cisakev and document KevImporter configuration and KnownExploitation structures, plus vulnerability details known_exploited field. Add e2e Hurl scenario that uploads a CVE, uploads a KEV catalog via advisory?format=cisakev, and asserts that vulnerability details show the KEV data and that non-listed CVEs have empty known_exploited. Update importer sample-data tests to expect the new KEV importer in the list count. Add ingestor detection tests ensuring CISA KEV JSON documents are correctly detected as Format::CisaKev with JSON wire format.	openapi.yaml e2e/kev.hurl modules/fundamental/src/vulnerability/service/test.rs modules/ingestor/src/service/detect.rs server/src/sample_data.rs

Possibly linked issues

• Support known-exploited-vulnerability catalogs (CISA KEV) as an enrichment source #2545: PR introduces KEV ingestor, database table, importer, and vulnerability API field precisely matching the issue’s v1 scope.

---

Tips and commands

Interacting with Sourcery

• Trigger a new review: Comment @sourcery-ai review on the pull request.
• Continue discussions: Reply directly to Sourcery's review comments.
• Generate a GitHub issue from a review comment: Ask Sourcery to create an
  issue from a review comment by replying to it. You can also reply to a
  review comment with @sourcery-ai issue to create an issue from it.
• Generate a pull request title: Write @sourcery-ai anywhere in the pull
  request title to generate a title at any time. You can also comment
  @sourcery-ai title on the pull request to (re-)generate the title at any time.
• Generate a pull request summary: Write @sourcery-ai summary anywhere in
  the pull request body to generate a PR summary at any time exactly where you
  want it. You can also comment @sourcery-ai summary on the pull request to
  (re-)generate the summary at any time.
• Generate reviewer's guide: Comment @sourcery-ai guide on the pull
  request to (re-)generate the reviewer's guide at any time.
• Resolve all Sourcery comments: Comment @sourcery-ai resolve on the
  pull request to resolve all Sourcery comments. Useful if you've already
  addressed all the comments and don't want to see them anymore.
• Dismiss all Sourcery reviews: Comment @sourcery-ai dismiss on the pull
  request to dismiss all existing Sourcery reviews. Especially useful if you
  want to start fresh with a new review - don't forget to comment
  @sourcery-ai review to trigger a new review!

Customizing Your Experience

Access your dashboard to:

• Enable or disable review features such as the Sourcery-generated pull request
  summary, the reviewer's guide, and others.
• Change the review language.
• Add, remove or edit custom review instructions.
• Adjust other review settings.

Getting Help

• Contact our support team for questions or feedback.
• Visit our documentation for detailed guides and information.
• Keep in touch with the Sourcery team by following us on X/Twitter, LinkedIn or GitHub.

[sourcery-ai]

Hey - I've reviewed your changes and they look great!

---

Sourcery is free for open source - if you like our reviews please consider sharing them ✨

• X
• Mastodon
• LinkedIn
• Facebook

_{Help me be more useful! Please click 👍 or 👎 on each comment and I'll use the feedback to improve your reviews.}

[codecov]

Codecov Report

❌ Patch coverage is 91.15044% with 20 lines in your changes missing coverage. Please review.
✅ Project coverage is 72.88%. Comparing base (e878596) to head (bd2087f).
⚠️ Report is 1 commits behind head on main.

Files with missing lines	Patch %	Lines
modules/importer/src/model/kev.rs	0.00%	9 Missing ⚠️
modules/fundamental/src/vulnerability/model/mod.rs	91.42%	0 Missing and 3 partials ⚠️
modules/ingestor/src/service/detect.rs	78.57%	0 Missing and 3 partials ⚠️
modules/ingestor/src/service/kev/mod.rs	95.16%	0 Missing and 3 partials ⚠️
modules/importer/src/runner/kev/mod.rs	97.82%	0 Missing and 1 partial ⚠️
server/src/sample_data.rs	95.83%	0 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #2553      +/-   ##
==========================================
+ Coverage   72.73%   72.88%   +0.15%     
==========================================
  Files         483      488       +5     
  Lines       30161    30387     +226     
  Branches    30161    30387     +226     
==========================================
+ Hits        21938    22149     +211     
+ Misses       6983     6982       -1     
- Partials     1240     1256      +16     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[rh-jfuller]

before we start adding more entities to trustify (which btw enriching with CISA data is great !) - we need to first define an overarching strategy to associating metadata generically in trustify ... otherwise we are going to experience 'sprawl' with concomitant maintenance ... I started scratching out a few thoughts here https://gist.github.com/rh-jfuller/845c1b17fefb79c9dc3dc3df194ee6d9 - I would close this PR and lets have a conversation. The idea is that we would provide some enrichment internals in trustify to be able to develop such things.

[waldemar-kindler]

before we start adding more entities to trustify (which btw enriching with CISA data is great !) - we need to first define an overarching strategy to associating metadata generically in trustify ... otherwise we are going to experience 'sprawl' with concomitant maintenance ... I started scratching out a few thoughts here https://gist.github.com/rh-jfuller/845c1b17fefb79c9dc3dc3df194ee6d9 - I would close this PR and lets have a conversation. The idea is that we would provide some enrichment internals in trustify to be able to develop such things.

Thanks for sharing your thoughts! Generally I follow your sentiments. Having an entity / data field sprawl was also one of my concerns. Just a little push back from my side. In my mental model KEV is closer to the NVD and CWE than to Exploit Intelligence, EPSS or Conforma.

The distinction I'd draw is feed vs. service call. EI, Conforma and (to a lesser degree) EPSS are request/response: they need context (which SBOM, which component), they need auth, they're per-job, and they benefit from exactly the shared plumbing you describe: Job queue, HTTP client with retry/backoff, error classification. KEV has none of that shape. It's a single public static JSON document with a stable schema . Pretty much like the CWE catalog importer.

On process: I'm glad to have the conversation, and I'll convert this to whatever shape you land on. One request: Could we keep it open as a draft rather than closed? It's working end-to-end so I think it's useful as a concrete reference for what the data looks like while we discuss. If you'd rather have it closed, that's fine too. I'll reopen once there's a direction.

[rh-jfuller]

@waldemar-kindler no worries, lets keep this PR open ... and your comments are spot on (will incorporate to enrichment doc working on) ... generally we want to identify primary entities and give them an importer/exporter treatment - everything else is enrichment. I can be convinced that your approach (eg. exploits as primary) works but want to have a few more conversations with others on the team to get alignment.