fix(dstack-mr): measure the kernel QEMU 10.2 hands to OVMF

[kvinwang]

Problem

TDX CVMs running on QEMU 10.2 fail verification with an RTMR[1] mismatch:

OS image hash verification failed: MRs do not match:
RTMR1 mismatch: expected=4ffc8bcd33abc1bbf1d772196dc6d9f68cbe64c07b16a74ac085aa22ae976731bbb34a59255f119ec1b2ba53018cfa8e,
                  actual=74f00f634fe914dc875dc285d8efe1c1ecda4cf10e207f5abd079b559ac5e02ffd64eb75f03ae353cbe72f6ebdd9eca8

MRTD, RTMR[0] (including the generated ACPI blobs) and RTMR[2] all verify. Only
the kernel event in RTMR[1] is wrong.

The root cause is upstream QEMU commit
a7542a38f399
("x86/loader: Don't update kernel header for CoCo VMs", merged 2025-08-27):

-    if (!sev_enabled() && protocol > 0) {
+    if (!MACHINE(x86ms)->cgs && protocol > 0) {
         memcpy(setup, header, MIN(sizeof(header), setup_size));
     }

QEMU rewrites the Linux setup header (type_of_loader, loadflags,
heap_end_ptr, cmdline_addr, initrd_addr, initrd_size) before serving the
kernel over fw_cfg. It used to skip that only for SEV; the commit widens the
skip to every confidential guest, TDX included — explicitly because rewriting
the header "leads to a different hash and breaks the attestation, e.g., for
TDX". So from 10.2 on, OVMF measures the kernel as built, while dstack-mr
still computed the Authenticode hash of QEMU's patched copy.

The boundary is a single step at 10.2.0, not a per-release quirk. The commit
first appears in v10.2.0-rc1, was not backported to any v10.0.x/v10.1.x
tag, and every release since — 10.2.x, 11.0.x, 11.1.x and master — still carries
the widened !MACHINE(x86ms)->cgs check.

QEMU	setup header under TDX	dstack-mr before this PR
≤ 10.1.x	patched by QEMU	correct
≥ 10.2.0	left as built	wrong

Fix

• dstack-mr derives patch_kernel_header from qemu_version (< 10.2.0) and
  measures either QEMU's patched kernel or the image file as built.

• The measurement document records both Authenticode digests, because an
  image cannot know which QEMU will run it and the no-image-download path has no
  kernel file to hash:

  pub struct TdxImageMeasurement {
      pub kernel_cmdline_sha384: Vec<u8>,
      pub kernel_authenticode: Vec<u8>,          // the kernel as built (QEMU >= 10.2)
      pub patched_kernel_authenticode: Vec<u8>,  // QEMU's rewritten copy (<= 10.1)
      pub initrd_sha384: Vec<u8>,
  }

  Both are required. This is measurement.tdx.cbor v4, and v4 is the only
  version the decoder accepts.

• On QEMU ≥ 10.2 the kernel digest no longer depends on guest RAM, so the lite
  path's memory_size == 2 GiB || >= 0xB0000000 restriction is lifted there. It
  still applies on ≤ 10.1, where QEMU's patched initrd placement moves with RAM.

• vm_config.qemu_patches_kernel_header (and vmm.toml
qemu_patches_kernel_header, dstack-mr measure --patch-kernel-header) is an
  escape hatch for a fork whose behavior disagrees with its version number,
  mirroring the existing qemu_single_pass_add_pages knob.

• tools/vm-runner now emits the detected qemu_version. It previously emitted
  none, which defaults to 9.1.0; that used to be harmless but now picks the
  wrong kernel digest on a 10.2 host.

Compatibility: v3 documents are rejected

A v3 document carries a single image.kernel_authenticode, which is always the
QEMU-patched digest. The as-built digest cannot be derived from it — you need
the bzImage — so a v3 document cannot be read into the two-field shape at all.

Rather than model that absence as an Option, v3 is rejected on its version
number:

TdxOsImageMeasurement: unsupported version 3, expected 4

Reading the version before the payload is what turns this into an actionable
message instead of a missing field patched_kernel_authenticode shape error.

This is a breaking change for images built against v3. v3 shipped in the
v0.6.0-rc0 prereleases and in no stable release. Affected images must be
re-emitted, which gives them a new os_image_hash and therefore needs
re-whitelisting on chain. Mitigating factors: tdx_attestation_variant defaults
to Legacy, which recomputes from the downloaded image and never touches this
document; and a verifier can supply its own vm_config to verify a Lite CVM
over the Legacy path.

Verification

Reproduced and fixed against two live CVMs on QEMU 10.2.1
(Ubuntu 1:10.2.1+ds-1ubuntu3.2), on different images and very different memory
sizes. Replaying RTMR[1] from each CVM's own event log:

CVM	quoted RTMR[1]	replay, kernel as built	replay, patched (old behavior)
1 vCPU / 2 GiB, dstack-0.6.0-rc0	74f00f63…	74f00f63… ✅	4ffc8bcd… ❌
64 vCPU / 768 GiB / 8×GPU, dstack-0.6.0-next	a839af66…	a839af66… ✅	a0e85dfd… ❌

4ffc8bcd… is exactly the expected value the verifier reported, so the two
sides meet.

Also confirmed the ACPI generator is not implicated: regenerating the QEMU
10.2.1 blobs with qemu-acpi reproduces all three RTMR[0] ACPI DATA digests
(acpi-loader, acpi-rsdp, acpi-tables) byte for byte, and
acpi_tables_verified was already true before this change. The
(10, _) => V10 profile mapping is correct for 10.2.

New tests:

• dstack/verifier/fixtures/tdx-lite-qemu-10-2-attestation.json — an
  end-to-end capture from CVM 1 above (quote, event log and runtime events
  unmodified; tdx_measurement regenerated from that CVM's image directory).
  Verified as a genuine regression test by pointing
  QEMU_COCO_KERNEL_HEADER_UNPATCHED above 10.2.1 and re-running: it fails with
  the exact production RTMR[1] mismatch above.
• dstack-mr: the version boundary (8.2.2 … 10.1.9 patched; 10.2.0 … 12.0.0
  not), the override, digest selection pinned to both CVMs' real values, and
  that the memory-size guard only applies on the patched branch.
• dstack-types: CBOR round-trip keeping the two digests distinct, unknown
  versions rejected, and a real captured v3 document rejected on its version.

Fixture migration

The two pre-existing lite fixtures carried v3 documents and had to be migrated:

• tdx-lite-qemu-10-2-attestation.json already carried both digests; only the
  encoding was reshaped.
• tdx-lite-attestation.json / tdx-lite-getquote.json were missing the
  as-built digest. It was taken from ~/.dstack/images/dstack-0.6.0, whose
  patched_kernel_authenticode and initrd_sha384 both equal the captured
  values — so its bzImage is the byte-identical file that CVM booted.

In each case checksum_file and os_image_hash were recomputed over the new
bytes. Nothing the quote commits to changed, and RTMR replay still passes.
Provenance is recorded in dstack/verifier/fixtures/tdx-lite.README.md.

Full workspace suite, clippy -D warnings and prek are clean.

[Copilot]

🔵 Needs a closer look

It changes security-critical TDX measurement/serialization semantics and verifier behavior across versions, warranting final human validation despite added tests.

Pull request overview

Updates TDX kernel measurement handling to match QEMU ≥ 10.2 behavior (no Linux setup-header rewrite for confidential guests), extending the measurement document format to carry both patched and unpatched kernel Authenticode digests so verification can select the correct RTMR[1] input based on the host’s QEMU behavior/version.

Changes:

• Add QEMU-version–aware selection between patched vs. unpatched kernel Authenticode digests, including an explicit override (qemu_patches_kernel_header) for forks.
• Bump the TDX measurement CBOR document to v4 to store {patched, unpatched} kernel digest candidates while keeping v3 readable and byte-for-byte stable on re-encode.
• Add end-to-end verifier regression coverage via a captured QEMU 10.2.1 TDX-lite fixture; emit qemu_version from tools/vm-runner to avoid incorrect defaulting.

File summaries

File	Description
tools/vm-runner/vm-runner.py	Detects and emits qemu_version into vm_config so measurement selection matches the host QEMU behavior.
dstack/vmm/vmm.toml	Documents the new qemu_patches_kernel_header escape-hatch knob.
dstack/vmm/src/config.rs	Adds qemu_patches_kernel_header to CVM config to override version-derived behavior when needed.
dstack/vmm/src/app.rs	Plumbs qemu_patches_kernel_header into the generated vm_config; updates test data for new kernel digest shape.
dstack/verifier/src/verification.rs	Bumps measurement cache version and wires qemu_patches_kernel_header into verifier measurement option resolution; adds QEMU 10.2 fixture test.
dstack/verifier/fixtures/tdx-lite.README.md	Documents the new QEMU 10.2 fixture and the kernel-measurement behavior change.
dstack/verifier/fixtures/tdx-lite-qemu-10-2-attestation.json	Adds captured real-world QEMU 10.2.1 TDX-lite attestation fixture for regression coverage.
dstack/dstack-types/src/lib.rs	Introduces TdxKernelAuthenticodeCandidates and CBOR v4 encoding/decoding that preserves v3 byte-for-byte stability.
dstack/dstack-mr/src/tdx.rs	Implements version/override-based kernel digest selection for RTMR[1] and updates measurement emission to record both candidates.
dstack/dstack-mr/src/main.rs	Updates CLI feature/help text to reference v4 split-measurement behavior.
dstack/dstack-mr/src/machine.rs	Adds version boundary constant and version-derived default for patch_kernel_header, with override support and tests.
dstack/dstack-mr/src/kernel.rs	Adds unpatched-kernel digest path and threads patch/no-patch decision into RTMR[1] event generation.
dstack/dstack-mr/cli/src/main.rs	Exposes --patch-kernel-header CLI override and plumbs config into measurement option resolution.
docs/security/security-model.md	Explains why vm_config.qemu_version is only a selector between committed candidates and doesn’t need to be trusted.

Review details

• Files reviewed: 14/14 changed files
• Comments generated: 1
• Review effort level: Lite

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[kvinwang]

Superseded by #1189.

That PR fixes the same bug by removing the QEMU-version dependency instead of modelling it: the image build and OVMF both normalize the Linux setup header, so RTMR[1] becomes the plain Authenticode hash of the shipped bzImage on every QEMU version and at every guest memory size. patch_kernel(), the two-digest measurement document, the version branch, the qemu_patches_kernel_header escape hatch and the memory_size == 2 GiB || >= 0xB0000000 restriction all go away.

The trade-off is stated in #1189: it needs a firmware change, so MRTD and RTMR[0] move, and the two live CVM captures here could not carry over -- their RTMR[1] is the QEMU-patched digest, signed into a quote. The lite fixtures were recaptured against a CVM running the normalizing firmware on QEMU 8.2.2, which is the harder case since that version does rewrite the header.

Leaving this open until #1189 is reviewed, as a fallback.