[Coldwings]

Summary

switch_context() / switch_context_defer() in thread/thread.cpp put a call to the switch routine inside an inline asm statement. Other photon threads run before that call "returns", and they can change any register and any memory. The compiler cannot see this, so it relies on the clobber list being complete — and the list is not. It names only general-purpose registers (plus xmm6–xmm15 on Win64). As a result, the compiler may legitimately:

1. keep a live value in a vector / mask register across the switch;
2. keep a value cached in a register, or delay a store to memory that the switch or the defer function reads (no "memory" clobber).

Separately, the Win64 _photon_thread_stub reserves 32 bytes of shadow space for its ordinary calls to the thread entry and _photon_thread_die.

Item 1 is the demonstrated cause of the recurring GCC 10/11/12 CI failures (see Evidence). The memory clobber and Win64 thread-entry shadow space are separate corrections.

Why caller-saved registers must be listed

On x86-64 SysV, all xmm / k registers are caller-saved (volatile). That does not make them "implicitly clobbered" here:

• The compiler does not parse the asm template, so it does not treat the asm statement as a call. For the compiler, the only registers the asm changes are the ones in the clobber list.
• The C++ call to switch_context() does not help either:
  • It is an inline function in the same TU. In the failing build it is inlined into thread_yield_to(), so there is no call switch_context boundary there.
  • Even if it were not inlined, GCC's -fipa-ra (enabled at -O2) lets the caller keep values in volatile registers that the callee does not use. The callee's register usage is derived from these very clobber lists.

Conversely, callee-saved vector registers (xmm6–xmm15 on Win64, v8–v15 on aarch64) must also be listed, because the switch does not save them the way a real callee would.

Problematic code (at 6c529d5)

Location	Issue
x86-64 SysV switch_context / switch_context_defer clobbers	No xmm* / k* registers, no "memory"
Win64 _photon_thread_stub	Ordinary calls do not have sufficient 32-byte shadow space
Win64 clobbers	Only xmm6–xmm15; xmm0–xmm5, AVX-512 regs and "memory" missing
aarch64 clobbers	No v0–v31 (incl. callee-saved low 64 bits of v8–v15, which the switch does not save), no "cc", no "memory"

References

• GCC manual, Extended Asm — Clobbers and Scratch Registers
  • On "memory": "The "memory" clobber tells the compiler that the assembly code performs memory reads or writes to items other than those listed in the input and output operands (for example, accessing the memory pointed to by one of the input parameters)."
    • This is exactly the case for from->stack._ptr and to->stack._ptr, and for the defer_arg that defer_migrate_current() passes by address from its own stack frame.
  • On calling functions from asm (the same page, "Assembler Template"): "calling functions directly from an assembler template requires a detailed understanding of the target assembler and ABI."
• GCC manual, Optimize Options — -fipa-ra
  • "Use caller save registers for allocation if those registers are not used by any called function. In that case it is not necessary to save and restore them around calls. This is only possible if called functions are part of same compilation unit as current function and they are compiled before it. Enabled at levels -O2, -O3, -Os …"
• System V x86-64 psABI, §3.2.1 "Registers", Figure "Register Usage"
  • xmm0–xmm15, xmm16–xmm31 and k0–k7 are all "temporary registers", i.e. not preserved across calls. The compiler may only keep them live across the asm if they are not clobbered.
• Microsoft, x64 software conventions — register volatility
  • XMM0–XMM5 are volatile, XMM6–XMM15 are nonvolatile, and the upper halves of YMM are volatile.
• Microsoft, x64 stack usage
  • "a callee is guaranteed that space has been allocated for all its parameters … these 4 stack locations are effectively owned by the called function"
• AAPCS64
  • "SIMD and Floating-Point registers": "Registers v8-v15 are Callee-saved and the remaining registers (v0-v7, v16-v31) are Caller-saved … only the bottom 64 bits of each value stored in v8-v15 need to be Callee-saved."
  • "General-purpose Registers": "The N, Z, C and V flags are undefined on entry to and return from a public interface."

Evidence: GCC 10/11/12 CI crashes

The crashes are recurring, e.g. run 40221375889 / job 121404632180 and its re-run job 121455468864.

Symptom: the same 5 tests SEGFAULT every time — test-thread, test-socket, test-pool, test-executor-std, test-exportfs. All 5 go through thread_yield_to().

Only one class of runner is affected. The crc64 throughput printed by test-checksum splits the runners into two classes. I checked 56 jobs across 5 runs:

Runner class	crc64 throughput	gcc-10/11/12	gcc-8/9/13/14
A	~68–71 GB/s	always fail	pass
B	~48 GB/s	always pass	pass

Why GCC version matters. CI builds with -march=native (CMAKE_CXX_FLAGS_MINSIZEREL). What -march=native expands to depends on whether that GCC version recognises the CPU model:

• GCC 11.2 / 12.2 recognise Sapphire Rapids (0x8F) but not Emerald Rapids (0xCF). Compare cpuinfo.h @ gcc-12.2.0 with cpuinfo.h @ gcc-13.3.0, which adds case 0xcf: /* Emerald Rapids. */.
• For an unknown family-6 model, the driver guesses -march from CPU features: AVX512_BF16 without VP2INTERSECT gives cooperlake. It then falls back to -mtune=generic. See driver-i386.c @ gcc-11.2.0 and L689.
• GCC 10.3 recognises neither model.

Simulation matches CI 14/14. I compiled thread.cpp on Compiler Explorer for each GCC version × node class, using the expansion above together with the CI flags. I assumed class A = 0xCF (C8i/G8i) and class B = Ice Lake 0x6A (C7/G7/R7). Only gcc-10.3/11.2/12.2 with -march=cooperlake -mtune=generic produce a value read from %xmm0 after the switch. That matches the CI pass/fail matrix exactly. The generated code in thread_yield_to() (gcc-11.2, -O2 -march=cooperlake -mtune=generic), viewable on Compiler Explorer — before / after (source is thread.cpp with its photon headers inlined; search the output for thread_yield_to):

        vmovq   -72(%rbp), %xmm0         # pointer kept in xmm0 ...
        vmovq   %xmm0, %rax
        movl    $0, 36(%rax)             # rq.current->error_number = 0
        ...
        call _photon_switch_context      # ... other threads run here ...
        vmovq   %xmm0, %rax              # ... and it is read back from xmm0
        movl    36(%rax), %r12d          # return rq.current->error_number

After this PR, the same configuration produces:

        call _photon_switch_context
        movq    -72(%rbp), %rax
        movl    36(%rax), %r12d
        vzeroupper

Caveat: that class A is 0xCF is inferred, not directly confirmed. It is the only model in the pool that GCC ≤ 12 does not recognise and GCC 13 does, so it is the only one that explains "gcc-12 fails, gcc-13 passes". Printing gcc -march=native -Q --help=target | grep -E 'm(arch|tune)=' in CI would confirm it. Either way, the root cause is the incomplete clobber list. The CPU model only decides whether it shows up, so pinning -march in CI would hide the bug rather than fix it.

Changes

1. Complete clobber lists — new VECTOR_CLOBBERS macro:
   • x86-64 (SysV and Win64): xmm0–xmm15; when __AVX512F__ is defined, also xmm16–xmm31 and k0–k7. Clobbering xmmN also covers ymmN / zmmN.
   • aarch64: v0–v31, plus p0–p15 when __ARM_FEATURE_SVE is defined. ffr is left out because clang rejects it as a clobber name.
   • Added "memory" on all targets and "cc" on aarch64.
2. Win64 thread-entry shadow space
   • _photon_thread_stub does sub $32,%rsp before its ordinary calls to the thread entry and _photon_thread_die; both calls reuse the reserved area.
   • Without it, the entry function's home area is [top-16, top+16) relative to the initial stack top. With reserved_space == 0, top is exactly the address of struct thread, so a callee that homes its arguments (e.g. an -O0 build) can overwrite the thread's intrusive list links. The new area is [top-48, top-16).

Notes:

• Clobbering the vector registers makes GCC insert a vzeroupper after the switch in some functions. It is a single cheap instruction, and correct in any case, since the resumed code cannot know the upper-half state.
• Not covered: x87 / MMX registers, and the callee-saved control state mxcsr / x87 CW. Their handling is unchanged and outside this PR's scope.

Verification

• Compiler Explorer: the clobber fix was checked across 14 compiler/target configurations. No immediate post-switch vector-register reads were reported (before: 3 configurations × thread_yield_to). Additional GCC 10.3 / 11.2 / 12.2 / 13.3 / 14.2 checks with -march=cooperlake -mtune=generic gave the same result.
• x86-64 SysV: thread.cpp compiled with GCC 14.2 (x86_64-linux-gnu-g++) and Clang 20 (--target=x86_64-linux-gnu) for -march=x86-64, cooperlake -mtune=generic, x86-64-v4 and sapphirerapids.
• aarch64 (Linux): test-thread, test-socket, test-executor-std, test-exportfs, and test-pool passed in the local RelWithDebInfo build. Clang 20 also compiled thread.cpp with -march=armv8-a+sve.
• aarch64 (Linux), initial validation: GCC 15 and clang compile checks, including +sve and armv9-a, passed. The local RelWithDebInfo build succeeded; 66 of 70 selected tests passed. test-throttle, test-iouring, and test-rpc-message also failed on the unmodified base; cache_test lacked /mnt/tmp. This was not a fully passing test suite.
• Win64: compile-only. I compiled the extracted asm and switch functions with clang --target=x86_64-w64-windows-gnu and checked them with objdump. No runtime test.
• macOS / Apple arm64: not built locally; arm64 changes are clobber declarations.
• x86-64 runtime: relying on this PR's CI. Please re-run the gcc-10/11/12 jobs a few times so they land on class-A runners.

[lihuiba]

move to _photon_thread_stub and _photon_thread_die

[Coldwings]

Done in 9a3e754: the comment now sits right above DEF_ASM_FUNC(_photon_thread_stub), next to the sub $32, %rsp and the calls to the thread entry and _photon_thread_die it describes.