Repository navigation
[Backport][main to 0.9] | fix(thread): complete context-switch clobbers and reserve Win64 shadow space (#1720) - #1725
Merged
Coldwings merged 2 commits intoNov 4, 2026
Conversation
…w space (#1720) * fix(thread): complete clobbers of context switch, align defer stack, skip red zone switch_context() / switch_context_defer() hide a function call inside an asm statement, across which other photon threads run and may change any register or memory. The clobber lists only named GPRs (plus xmm6-15 on Win64), so the compiler was free to keep values in vector registers, in memory it had cached, or in the red zone across a switch. This is what makes GCC 10/11/12 CI jobs crash (SEGFAULT in test-thread, test-socket, test-pool, test-executor-std, test-exportfs) on part of the nodes: with -march=native on a CPU model unknown to these versions (most likely Emerald Rapids, 0xCF), the driver falls back to -march=cooperlake -mtune=generic, under which GCC keeps the thread pointer behind `return rq.current->error_number` in %xmm0 across the switch in thread_yield_to(). - x86_64 / aarch64: declare all vector registers (and AVX-512 opmask / SVE predicate registers when enabled) as clobbered, plus "memory"; plus "cc" on aarch64. - x86_64 SysV / Apple arm64: skip the 128-byte red zone before the call, as GCC's manual describes for asm containing calls or pushes. - x86_64: call the defer function with %rsp aligned to 16 bytes instead of jumping into it on whatever the switch-point %rsp happens to be. - Win64: reserve the 32-byte shadow space for the defer function and the thread entry; the latter otherwise overlaps the head of `struct thread`. - tools/photongdb.py: account for the skipped red zone when recovering %rsp of a switched-out thread. * fix(thread): clarify why volatile registers must be listed as clobbers The asm statement is not treated as a call, so the ABI's caller-saved registers are not implied clobbered; nor does the C++ call of the inline switch_context() help, as it is inlined, or subject to -fipa-ra. * fix(thread): remove speculative red-zone padding Restore the original saved-stack layout and photongdb offset. Keep vector/memory/cc clobbers, defer alignment and Win64 shadow space; no current Photon red-zone failure has been demonstrated. * fix(thread): preserve original defer jump trampolines * fix(thread): move Win64 shadow-space comment to _photon_thread_stub
lihuiba
approved these changes
Nov 3, 2026
lihuiba
marked this pull request as ready for review
November 3, 2026 01:29
photonlibos
deleted the
backport-pr-26e71f7d7680708aee0dc219b63ea53f334ce5bf-0.9
branch
November 4, 2026 04:17
lihuiba
pushed a commit
that referenced
this pull request
Nov 4, 2026
…rs and reserve Win64 shadow space (#1720) (#1725) (#1726) * fix(thread): complete context-switch clobbers and reserve Win64 shadow space (#1720) * fix(thread): complete clobbers of context switch, align defer stack, skip red zone switch_context() / switch_context_defer() hide a function call inside an asm statement, across which other photon threads run and may change any register or memory. The clobber lists only named GPRs (plus xmm6-15 on Win64), so the compiler was free to keep values in vector registers, in memory it had cached, or in the red zone across a switch. This is what makes GCC 10/11/12 CI jobs crash (SEGFAULT in test-thread, test-socket, test-pool, test-executor-std, test-exportfs) on part of the nodes: with -march=native on a CPU model unknown to these versions (most likely Emerald Rapids, 0xCF), the driver falls back to -march=cooperlake -mtune=generic, under which GCC keeps the thread pointer behind `return rq.current->error_number` in %xmm0 across the switch in thread_yield_to(). - x86_64 / aarch64: declare all vector registers (and AVX-512 opmask / SVE predicate registers when enabled) as clobbered, plus "memory"; plus "cc" on aarch64. - x86_64 SysV / Apple arm64: skip the 128-byte red zone before the call, as GCC's manual describes for asm containing calls or pushes. - x86_64: call the defer function with %rsp aligned to 16 bytes instead of jumping into it on whatever the switch-point %rsp happens to be. - Win64: reserve the 32-byte shadow space for the defer function and the thread entry; the latter otherwise overlaps the head of `struct thread`. - tools/photongdb.py: account for the skipped red zone when recovering %rsp of a switched-out thread. * fix(thread): clarify why volatile registers must be listed as clobbers The asm statement is not treated as a call, so the ABI's caller-saved registers are not implied clobbered; nor does the C++ call of the inline switch_context() help, as it is inlined, or subject to -fipa-ra. * fix(thread): remove speculative red-zone padding Restore the original saved-stack layout and photongdb offset. Keep vector/memory/cc clobbers, defer alignment and Win64 shadow space; no current Photon red-zone failure has been demonstrated. * fix(thread): preserve original defer jump trampolines * fix(thread): move Win64 shadow-space comment to _photon_thread_stub * fix(thread): resolve Win64 thread stub conflict for release/0.9 Take the 32-byte shadow-space reservation from #1720 and drop the Win64 call to _asan_start, as main did in #937. ASan builds are Linux-only (PHOTON_BUILD_WITH_ASAN fails the configure elsewhere), so on Win64 _asan_start is always an empty function. --------- Co-authored-by: Coldwings <coldwings@me.com>
lihuiba
added a commit
that referenced
this pull request
Nov 4, 2026
…ers and reserve Win64 shadow space (#1720) (#1725) (#1726) (#1727) * [Backport][main to 0.9] | fix(thread): complete context-switch clobbers and reserve Win64 shadow space (#1720) (#1725) (#1726) * fix(thread): complete context-switch clobbers and reserve Win64 shadow space (#1720) * fix(thread): complete clobbers of context switch, align defer stack, skip red zone switch_context() / switch_context_defer() hide a function call inside an asm statement, across which other photon threads run and may change any register or memory. The clobber lists only named GPRs (plus xmm6-15 on Win64), so the compiler was free to keep values in vector registers, in memory it had cached, or in the red zone across a switch. This is what makes GCC 10/11/12 CI jobs crash (SEGFAULT in test-thread, test-socket, test-pool, test-executor-std, test-exportfs) on part of the nodes: with -march=native on a CPU model unknown to these versions (most likely Emerald Rapids, 0xCF), the driver falls back to -march=cooperlake -mtune=generic, under which GCC keeps the thread pointer behind `return rq.current->error_number` in %xmm0 across the switch in thread_yield_to(). - x86_64 / aarch64: declare all vector registers (and AVX-512 opmask / SVE predicate registers when enabled) as clobbered, plus "memory"; plus "cc" on aarch64. - x86_64 SysV / Apple arm64: skip the 128-byte red zone before the call, as GCC's manual describes for asm containing calls or pushes. - x86_64: call the defer function with %rsp aligned to 16 bytes instead of jumping into it on whatever the switch-point %rsp happens to be. - Win64: reserve the 32-byte shadow space for the defer function and the thread entry; the latter otherwise overlaps the head of `struct thread`. - tools/photongdb.py: account for the skipped red zone when recovering %rsp of a switched-out thread. * fix(thread): clarify why volatile registers must be listed as clobbers The asm statement is not treated as a call, so the ABI's caller-saved registers are not implied clobbered; nor does the C++ call of the inline switch_context() help, as it is inlined, or subject to -fipa-ra. * fix(thread): remove speculative red-zone padding Restore the original saved-stack layout and photongdb offset. Keep vector/memory/cc clobbers, defer alignment and Win64 shadow space; no current Photon red-zone failure has been demonstrated. * fix(thread): preserve original defer jump trampolines * fix(thread): move Win64 shadow-space comment to _photon_thread_stub * fix(thread): resolve Win64 thread stub conflict for release/0.9 Take the 32-byte shadow-space reservation from #1720 and drop the Win64 call to _asan_start, as main did in #937. ASan builds are Linux-only (PHOTON_BUILD_WITH_ASAN fails the configure elsewhere), so on Win64 _asan_start is always an empty function. --------- Co-authored-by: Coldwings <coldwings@me.com> * fix(thread): resolve the Win64 thread stub conflict for release/0.7 Take the 32-byte shadow-space reservation from #1720. release/0.7 never had the Win64 `call _asan_start` that the upstream change replaced, so there is nothing to drop here. --------- Co-authored-by: Coldwings <coldwings@me.com> Co-authored-by: lihuiba <huiba.lhb@alibaba-inc.com>
lihuiba
added a commit
that referenced
this pull request
Nov 4, 2026
…ers and reserve Win64 shadow space (#1720) (#1725) (#1726) (#1727) (#1728) * [Backport][main to 0.9] | fix(thread): complete context-switch clobbers and reserve Win64 shadow space (#1720) (#1725) (#1726) * fix(thread): complete context-switch clobbers and reserve Win64 shadow space (#1720) * fix(thread): complete clobbers of context switch, align defer stack, skip red zone switch_context() / switch_context_defer() hide a function call inside an asm statement, across which other photon threads run and may change any register or memory. The clobber lists only named GPRs (plus xmm6-15 on Win64), so the compiler was free to keep values in vector registers, in memory it had cached, or in the red zone across a switch. This is what makes GCC 10/11/12 CI jobs crash (SEGFAULT in test-thread, test-socket, test-pool, test-executor-std, test-exportfs) on part of the nodes: with -march=native on a CPU model unknown to these versions (most likely Emerald Rapids, 0xCF), the driver falls back to -march=cooperlake -mtune=generic, under which GCC keeps the thread pointer behind `return rq.current->error_number` in %xmm0 across the switch in thread_yield_to(). - x86_64 / aarch64: declare all vector registers (and AVX-512 opmask / SVE predicate registers when enabled) as clobbered, plus "memory"; plus "cc" on aarch64. - x86_64 SysV / Apple arm64: skip the 128-byte red zone before the call, as GCC's manual describes for asm containing calls or pushes. - x86_64: call the defer function with %rsp aligned to 16 bytes instead of jumping into it on whatever the switch-point %rsp happens to be. - Win64: reserve the 32-byte shadow space for the defer function and the thread entry; the latter otherwise overlaps the head of `struct thread`. - tools/photongdb.py: account for the skipped red zone when recovering %rsp of a switched-out thread. * fix(thread): clarify why volatile registers must be listed as clobbers The asm statement is not treated as a call, so the ABI's caller-saved registers are not implied clobbered; nor does the C++ call of the inline switch_context() help, as it is inlined, or subject to -fipa-ra. * fix(thread): remove speculative red-zone padding Restore the original saved-stack layout and photongdb offset. Keep vector/memory/cc clobbers, defer alignment and Win64 shadow space; no current Photon red-zone failure has been demonstrated. * fix(thread): preserve original defer jump trampolines * fix(thread): move Win64 shadow-space comment to _photon_thread_stub * fix(thread): resolve Win64 thread stub conflict for release/0.9 Take the 32-byte shadow-space reservation from #1720 and drop the Win64 call to _asan_start, as main did in #937. ASan builds are Linux-only (PHOTON_BUILD_WITH_ASAN fails the configure elsewhere), so on Win64 _asan_start is always an empty function. --------- * fix(thread): resolve the Win64 thread stub conflict for release/0.7 Take the 32-byte shadow-space reservation from #1720. release/0.7 never had the Win64 `call _asan_start` that the upstream change replaced, so there is nothing to drop here. --------- Co-authored-by: Coldwings <coldwings@me.com> Co-authored-by: lihuiba <huiba.lhb@alibaba-inc.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
switch_context() / switch_context_defer() hide a function call inside an
asm statement, across which other photon threads run and may change any
register or memory. The clobber lists only named GPRs (plus xmm6-15 on
Win64), so the compiler was free to keep values in vector registers, in
memory it had cached, or in the red zone across a switch.
This is what makes GCC 10/11/12 CI jobs crash (SEGFAULT in test-thread,
test-socket, test-pool, test-executor-std, test-exportfs) on part of the
nodes: with -march=native on a CPU model unknown to these versions (most
likely Emerald Rapids, 0xCF), the driver falls back to -march=cooperlake
-mtune=generic, under which GCC keeps the thread pointer behind
return rq.current->error_numberin %xmm0 across the switch inthread_yield_to().
SVE predicate registers when enabled) as clobbered, plus "memory";
plus "cc" on aarch64.
as GCC's manual describes for asm containing calls or pushes.
of jumping into it on whatever the switch-point %rsp happens to be.
thread entry; the latter otherwise overlaps the head of
struct thread.%rsp of a switched-out thread.
The asm statement is not treated as a call, so the ABI's caller-saved
registers are not implied clobbered; nor does the C++ call of the inline
switch_context() help, as it is inlined, or subject to -fipa-ra.
Restore the original saved-stack layout and photongdb offset. Keep vector/memory/cc clobbers, defer alignment and Win64 shadow space; no current Photon red-zone failure has been demonstrated.
fix(thread): preserve original defer jump trampolines
fix(thread): move Win64 shadow-space comment to _photon_thread_stub
Generated by Backport Auto PR, by cherry-pick related commits.
Please review and decide whether to merge or close this backport PR.
Conflicts
Cherry-pick produced conflicts. Conflict markers are committed as-is; please resolve them manually before merging.
Conflict resolution
Resolved in a14f89f. The only conflict is the Win64
_photon_thread_stub, where release/0.9 still hascall _asan_start(main removed it from the Win64 stub in #937). The resolution keeps thesub $32, %rspshadow-space reservation from #1720 and dropscall _asan_start, so the Win64 stub is now identical to main. ASan builds are Linux-only (PHOTON_BUILD_WITH_ASANfails the CMake configure on other systems), so on Win64_asan_startwas always an empty function. The SysV and aarch64 stubs are unchanged.