feat(os): normalize the Linux setup header so new images measure the kernel file

docs/security/security-model.md

@@ -317,6 +317,45 @@ AML access to encrypted/private guest RAM. Verification now rejects tampered
 tables before the CVM is trusted with keys; the sandbox bounds what tampered
 AML could have done in the first place.
 
+### The kernel measurement does not depend on the host's QEMU
+
+QEMU is the boot loader for `-kernel`: it fills in the setup-header fields the
+Linux boot protocol expects a boot loader to supply, and OVMF measures the
+result into RTMR[1]. QEMU commit `a7542a38f399` ("x86/loader: Don't update
+kernel header for CoCo VMs", first released in 10.2.0) stopped rewriting the
+header for confidential guests, so the same kernel would otherwise measure
+differently depending on which QEMU the host chose to run.
+
+dstack removes that dependency instead of modelling it. The image build zeroes
+the boot-loader-written fields in the kernel it ships, and dstack's OVMF zeroes
+them again before the kernel blob is measured and loaded. RTMR[1] is therefore
+the plain Authenticode hash of the `bzImage` listed in `sha256sum.txt`, and the
+verifier needs nothing from the host to predict it -- not a QEMU version, not a
+memory size.
+
+Images built before this landed keep their original behavior: their firmware
+does not normalize, so their digest still covers QEMU's rewritten copy. Which
+of the two applies is declared by the image itself -- `kernel_header_normalized`,
+recorded in `metadata.json` for the image-download path and mirrored into the
+measurement document for the no-image-download path -- so it is never something
+the host gets to choose.
+
+Both carriers are bound to `os_image_hash`. `sha256sum.txt` hashes to
+`os_image_hash`, a downloaded image is checked file by file against it, and the
+measurement document is one of its entries.
+
+This matters because everything the host declares about its own VM is
+untrusted. A knob the verifier has to consult is a knob the host can lie about;
+here there is no knob. It also removes a class of correct-but-rejected
+deployments, since the previous QEMU-patched digest varied with guest RAM and
+was only reproducible at specific memory sizes.
+
+The normalized field set comes from the boot protocol rather than from QEMU's
+behavior: every field `Documentation/arch/x86/boot.rst` types as `write` is one
+the boot loader fills in and the kernel supplies no value for, so zeroing it
+discards nothing the kernel provided. Fields typed `modify` carry real
+kernel-supplied values and are left measured.
+
 ### TCB status is surfaced, not gated, during verification
 
 dstack's `validate_tcb` does not reject a quote based on its TCB status string (`UpToDate`, `OutOfDate`, `ConfigurationNeeded`, `SWHardeningNeeded`, ...). It only enforces hard invariants: debug mode must be off, and the SEAM/service-TD measurements must be well-formed. The verified report carries the `status` field through to the caller.

dstack/dstack-mr/cli/src/main.rs

@@ -126,6 +126,7 @@ fn main() -> Result<()> {
                 .initrd(&initrd_path)
                 .kernel_cmdline(&cmdline)
                 .maybe_two_pass_add_pages(config.two_pass_add_pages)
+                .normalized_setup_header(image_info.kernel_header_normalized)
                 .maybe_pic(config.pic)
                 .smm(config.smm)
                 .maybe_pci_hole64_size(config.pci_hole64_size)
@@ -341,6 +342,7 @@ fn run_diagnose(config: &DiagnoseConfig) -> Result<()> {
         .root_verity(true)
         .hotplug_off(vm.hotplug_off)
         .maybe_two_pass_add_pages(vm.qemu_single_pass_add_pages)
+        .normalized_setup_header(image_info.kernel_header_normalized)
         .maybe_pic(vm.pic)
         .maybe_qemu_version(vm.qemu_version.clone())
         .maybe_pci_hole64_size(if vm.pci_hole64_size > 0 {

dstack/dstack-mr/src/kernel.rs

@@ -237,15 +237,35 @@ pub(crate) fn patched_kernel_authenticode_sha384(
     authenticode_sha384_hash(&kd).context("Failed to compute kernel hash")
 }
 
-/// Measures a QEMU-patched TDX kernel image.
+/// Compute the first RTMR[1] event digest for an image whose OVMF normalizes
+/// the Linux setup header: the Authenticode SHA-384 of the kernel file itself.
+///
+/// Both sides zero the boot-loader-written fields --
+/// `os/image/normalize-kernel-header.py` in the image build and
+/// `0007-OvmfPkg-QemuKernelLoaderFsDxe-normalize-setup-header.patch` in the
+/// firmware -- so what OVMF measures is the file, on every QEMU version and at
+/// every guest memory size.
+pub(crate) fn kernel_authenticode_sha384(kernel_data: &[u8]) -> Result<Vec<u8>> {
+    authenticode_sha384_hash(kernel_data).context("failed to compute kernel hash")
+}
+
+/// Measures the TDX kernel image OVMF loads.
+///
+/// `normalized_setup_header` says which kernel bytes OVMF ends up measuring:
+/// the kernel file as shipped when the image's firmware normalizes the setup
+/// header, otherwise QEMU's rewritten copy.
 pub(crate) fn rtmr1_log(
     kernel_data: &[u8],
     initrd_size: u32,
     mem_size: u64,
     acpi_data_size: u32,
+    normalized_setup_header: bool,
 ) -> Result<Vec<Vec<u8>>> {
-    let kernel_hash =
-        patched_kernel_authenticode_sha384(kernel_data, initrd_size, mem_size, acpi_data_size)?;
+    let kernel_hash = if normalized_setup_header {
+        kernel_authenticode_sha384(kernel_data)?
+    } else {
+        patched_kernel_authenticode_sha384(kernel_data, initrd_size, mem_size, acpi_data_size)?
+    };
     Ok(vec![
         kernel_hash,
         measure_sha384(b"Calling EFI Application from Boot Option"),
@@ -270,6 +290,56 @@ mod tests {
         u32::from_le_bytes(kernel[0x218..0x21c].try_into().unwrap())
     }
 
+    /// A minimal PE/COFF so `authenticode_sha384_hash` has something to walk.
+    fn pe_kernel() -> Vec<u8> {
+        let mut kernel = vec![0u8; 0x2000];
+        let lfanew = 0x40usize;
+        kernel[0x3c..0x40].copy_from_slice(&(lfanew as u32).to_le_bytes());
+        kernel[lfanew..lfanew + 4].copy_from_slice(&object::pe::IMAGE_NT_SIGNATURE.to_le_bytes());
+        let coff = lfanew + 4;
+        // SizeOfOptionalHeader, then PE32+ magic and SizeOfHeaders.
+        kernel[coff + 16..coff + 18].copy_from_slice(&0xf0u16.to_le_bytes());
+        let opt = coff + 20;
+        kernel[opt..opt + 2].copy_from_slice(&0x020bu16.to_le_bytes());
+        kernel[opt + 60..opt + 64].copy_from_slice(&0x400u32.to_le_bytes());
+        // Setup header: protocol 2.12, and a non-zero heap_end_ptr like a real
+        // build has, so the two branches cannot coincide by accident.
+        kernel[0x202..0x206].copy_from_slice(b"HdrS");
+        kernel[0x206..0x208].copy_from_slice(&0x020cu16.to_le_bytes());
+        // XLF_CAN_BE_LOADED_ABOVE_4G, so QEMU derives the initrd address from
+        // available low memory and the patched digest moves with guest RAM.
+        kernel[0x236..0x238].copy_from_slice(&0x0040u16.to_le_bytes());
+        kernel[0x224..0x226].copy_from_slice(&0x50a0u16.to_le_bytes());
+        kernel
+    }
+
+    /// The flag has to reach the digest, not just the struct: an image whose
+    /// firmware normalizes must measure the file, and one whose firmware does
+    /// not must measure QEMU's rewritten copy.
+    #[test]
+    fn the_normalized_flag_selects_which_kernel_bytes_are_measured() {
+        let kernel = pe_kernel();
+        let normalized = rtmr1_log(&kernel, 0x1000, 0x8000_0000, 0x28000, true).unwrap();
+        let patched = rtmr1_log(&kernel, 0x1000, 0x8000_0000, 0x28000, false).unwrap();
+        assert_ne!(normalized[0], patched[0]);
+        assert_eq!(normalized[0], kernel_authenticode_sha384(&kernel).unwrap());
+        assert_eq!(
+            patched[0],
+            patched_kernel_authenticode_sha384(&kernel, 0x1000, 0x8000_0000, 0x28000).unwrap()
+        );
+    }
+
+    /// Normalizing is what makes the digest independent of guest RAM, so the
+    /// two branches must disagree about that too.
+    #[test]
+    fn only_the_patched_digest_moves_with_guest_memory() {
+        let kernel = pe_kernel();
+        let at = |mem| rtmr1_log(&kernel, 0x1000, mem, 0x28000, true).unwrap()[0].clone();
+        assert_eq!(at(0x8000_0000), at(0xA000_0000));
+        let patched_at = |mem| rtmr1_log(&kernel, 0x1000, mem, 0x28000, false).unwrap()[0].clone();
+        assert_ne!(patched_at(0x8000_0000), patched_at(0xA000_0000));
+    }
+
     #[test]
     fn tdx_kernel_patch_uses_precomputed_digest_at_2g_and_high_memory() {
         let mut kernel = vec![0u8; 0x1000];

dstack/dstack-mr/src/machine.rs

@@ -20,6 +20,12 @@ pub struct Machine<'a> {
     pub initrd: &'a str,
     pub kernel_cmdline: &'a str,
     pub two_pass_add_pages: Option<bool>,
+    /// Whether this image's OVMF normalizes the Linux setup header before
+    /// measuring the kernel. Defaults to `false`, which is the behavior of
+    /// every image built before the normalization landed; callers that have
+    /// the image metadata set it from `kernel_header_normalized`.
+    #[builder(default = false)]
+    pub normalized_setup_header: bool,
     pub pic: Option<bool>,
     pub qemu_version: Option<String>,
     #[builder(default = false)]
@@ -137,6 +143,7 @@ impl Machine<'_> {
             initrd_data.len() as u32,
             self.memory_size,
             0x28000,
+            self.normalized_setup_header,
         )?;
         debug_print_log("RTMR1", &rtmr1_log);
         let rtmr1 = measure_log(&rtmr1_log);

dstack/dstack-mr/src/tdx.rs

@@ -11,8 +11,9 @@
 //! intentionally excluded and must come from `VmConfig`.
 
 use crate::kernel::{
-    patched_kernel_authenticode_sha384, tdx_kernel_hash_uses_precomputed_high_mem,
-    TDX_KERNEL_HASH_COMPAT_2G_MEMORY, TDX_KERNEL_HASH_STABLE_MIN_MEMORY,
+    kernel_authenticode_sha384, patched_kernel_authenticode_sha384,
+    tdx_kernel_hash_uses_precomputed_high_mem, TDX_KERNEL_HASH_COMPAT_2G_MEMORY,
+    TDX_KERNEL_HASH_STABLE_MIN_MEMORY,
 };
 use crate::tdvf::{rtmr0_log_from_td_hob_hash_with_acpi_hashes, AcpiTableHashes, Tdvf};
 use crate::util::{measure_log, measure_sha384};
@@ -34,6 +35,11 @@ struct ImageMetadata {
     bios: String,
     #[serde(default)]
     ovmf_variant: Option<OvmfVariant>,
+    /// Declares whether this image's OVMF normalizes the Linux setup header
+    /// before measuring the kernel. Absent on every image built before that
+    /// landed, and `false` is exactly their behavior.
+    #[serde(default)]
+    kernel_header_normalized: bool,
 }
 
 #[derive(Debug, Clone)]
@@ -151,6 +157,12 @@ fn read_varuint(input: &mut &[u8]) -> Result<u64> {
     }
 }
 
+/// q35 keeps guest RAM in one block below 4G until it reaches this size, then
+/// caps the below-4G block at 2 GiB and moves the remainder above 4G
+/// (`lowmem = 0xb0000000 unless ram_size >= 0xb0000000`). The TD HOB memory
+/// ranges follow that split.
+const Q35_HIGH_MEMORY_SPLIT: u64 = 0xB000_0000;
+
 fn measure_td_hob_from_witness_data(data: &[u8], memory_size: u64) -> Result<Vec<u8>> {
     let mut input = data;
     let base_page = read_varuint(&mut input)?;
@@ -217,7 +229,7 @@ fn measure_td_hob_from_witness_data(data: &[u8], memory_size: u64) -> Result<Vec
     if last_end < last_start {
         bail!("Invalid last memory range: end < start");
     }
-    if memory_size >= TDX_KERNEL_HASH_STABLE_MIN_MEMORY {
+    if memory_size >= Q35_HIGH_MEMORY_SPLIT {
         if last_start < 0x80000000u64 {
             add_memory_resource_hob(0x07, last_start, 0x80000000u64 - last_start);
         }
@@ -321,15 +333,22 @@ pub fn tdx_os_image_measurement_for_image_dir(image_dir: &Path) -> Result<TdxOsI
     let kernel_path = image_dir.join(&meta.kernel);
     let kernel =
         fs::read(&kernel_path).with_context(|| format!("cannot read {}", kernel_path.display()))?;
-    let kernel_authenticode = patched_kernel_authenticode_sha384(
-        &kernel,
-        initrd.len() as u32,
-        TDX_KERNEL_HASH_STABLE_MIN_MEMORY,
-        0x28000,
-    )
-    .context("failed to compute high-memory QEMU-patched kernel hash")?;
+    // Which bytes OVMF will measure is a property of this image's firmware, so
+    // it is read from the image rather than from anything the host says.
+    let kernel_authenticode = if meta.kernel_header_normalized {
+        kernel_authenticode_sha384(&kernel)?
+    } else {
+        patched_kernel_authenticode_sha384(
+            &kernel,
+            initrd.len() as u32,
+            TDX_KERNEL_HASH_STABLE_MIN_MEMORY,
+            0x28000,
+        )
+        .context("failed to compute high-memory QEMU-patched kernel hash")?
+    };
 
     Ok(TdxOsImageMeasurement {
+        kernel_header_normalized: meta.kernel_header_normalized,
         image: TdxImageMeasurement {
             kernel_cmdline_sha384: crate::kernel::measure_cmdline(&measured_kernel_cmdline(
                 &base_cmdline,
@@ -361,31 +380,30 @@ pub fn tdx_measurement_hash_for_image_dir(image_dir: &Path) -> Result<[u8; 32]>
 /// Compute expected TDX measurements from self-contained TDX measurement
 /// material and the three ACPI table digests captured in RTMR[0].
 ///
-/// This path intentionally does not download or read the OS image. Because
-/// QEMU's patched kernel Authenticode hash depends on exact guest RAM below
-/// `TDX_KERNEL_HASH_STABLE_MIN_MEMORY`, the no-image-download path supports
-/// CVMs at or above that threshold plus the exact 2 GiB placement, which QEMU
-/// patches to the same kernel bytes as the high-memory case.
+/// This path intentionally does not download or read the OS image. Every
+/// guest memory size is supported: the kernel digest is a plain hash of the
+/// image file, because the setup header is normalized on both sides.
 pub fn tdx_measurements_from_measurement_document(
     document: &TdxOsImageMeasurementDocument,
     vm_config: &VmConfig,
     acpi_hashes: &TdxRtmr0AcpiHashes,
 ) -> Result<crate::TdxMeasurements> {
-    if !tdx_kernel_hash_uses_precomputed_high_mem(vm_config.memory_size) {
+    let measurement = document
+        .decode_measurement()
+        .map_err(anyhow::Error::msg)
+        .context("failed to decode TDX measurement CBOR")?;
+    if !measurement.kernel_header_normalized
+        && !tdx_kernel_hash_uses_precomputed_high_mem(vm_config.memory_size)
+    {
         bail!(
-            "TDX lite attestation without image download requires memory_size == {} bytes ({} MiB) or >= {} bytes ({} MiB); got {} bytes",
+            "TDX lite attestation without image download requires memory_size == {} bytes ({} MiB) or >= {} bytes ({} MiB); got {} bytes. This restriction only applies to images whose OVMF does not normalize the kernel setup header, because QEMU's rewrite moves the initrd with guest RAM; re-emit the image to remove it",
             TDX_KERNEL_HASH_COMPAT_2G_MEMORY,
             TDX_KERNEL_HASH_COMPAT_2G_MEMORY / 1024 / 1024,
             TDX_KERNEL_HASH_STABLE_MIN_MEMORY,
             TDX_KERNEL_HASH_STABLE_MIN_MEMORY / 1024 / 1024,
             vm_config.memory_size
         );
     }
-
-    let measurement = document
-        .decode_measurement()
-        .map_err(anyhow::Error::msg)
-        .context("failed to decode TDX measurement CBOR")?;
     let mrtd = select_mrtd(&measurement, vm_config)?;
 
     let td_hob_hash =
@@ -482,6 +500,7 @@ pub fn tdx_measurements_for_image_dir_without_rtmr0(
         .root_verity(true)
         .hotplug_off(vm_config.hotplug_off)
         .maybe_two_pass_add_pages(vm_config.qemu_single_pass_add_pages)
+        .normalized_setup_header(meta.kernel_header_normalized)
         .maybe_pic(vm_config.pic)
         .maybe_qemu_version(vm_config.qemu_version.clone())
         .maybe_pci_hole64_size(if vm_config.pci_hole64_size > 0 {
@@ -507,6 +526,7 @@ pub fn tdx_measurements_for_image_dir_without_rtmr0(
         initrd_data.len() as u32,
         vm_config.memory_size,
         0x28000,
+        meta.kernel_header_normalized,
     )
     .context("failed to compute RTMR1")?;
     let rtmr1 = measure_log(&rtmr1_log);
@@ -573,6 +593,7 @@ pub fn tdx_measurements_for_image_dir_with_acpi_hashes(
         .root_verity(true)
         .hotplug_off(vm_config.hotplug_off)
         .maybe_two_pass_add_pages(vm_config.qemu_single_pass_add_pages)
+        .normalized_setup_header(meta.kernel_header_normalized)
         .maybe_pic(vm_config.pic)
         .maybe_qemu_version(vm_config.qemu_version.clone())
         .maybe_pci_hole64_size(if vm_config.pci_hole64_size > 0 {
@@ -611,6 +632,7 @@ pub fn tdx_measurements_for_image_dir_with_acpi_hashes(
         initrd_data.len() as u32,
         vm_config.memory_size,
         0x28000,
+        meta.kernel_header_normalized,
     )
     .context("failed to compute RTMR1")?;
     let rtmr1 = measure_log(&rtmr1_log);