Windows: pin Microsoft.ML.OnnxRuntime.DirectML back to 1.23.0 (1.24.x faults natively on session creation) - #2443
Merged
Conversation
…on creation The Windows DirectML build of ONNX Runtime 1.24.x can kill the process natively while an InferenceSession is being created. There is no managed exception and nothing is written to the log - the process simply disappears, leaving only a Windows ".NET Runtime" 1026 event with 0x80131506. It reproduces reliably when switching singer, i.e. when several sessions are created in quick succession. What was ruled out while tracking this down: - the CPU execution provider is unaffected; - the 1.23.0 DirectML build is unaffected; - no model in the voicebank is to blame: runtime profiling shows 252/258 nodes still executing on DmlExecutionProvider under both 1.23.0 and 1.24.4, and the per-node assignment is identical; - the DirectML EP sources are byte-identical between v1.23.0 and v1.24.4 apart from two added exports, and graph partitioning behaves the same way. 1.24.4 is a dead end for fixes. DirectML was moved to sustained engineering in 2025-09, so this is the last release of Microsoft.ML.OnnxRuntime.DirectML, and the robustness fixes made afterwards (upstream #28007 "per-instance mutexes to fix concurrent session crashes", #31719, #32745, ...) will never ship as a package upgrade. Only the Windows group is touched. The Linux group cannot move back - the package it uses (Microsoft.ML.OnnxRuntime.Gpu.Linux) has no 1.23.x release at all - and the macOS/CoreML path is unrelated to this crash.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes the native crash on Windows when the DirectML runner is used and a singer
is switched or loaded.
Symptom
The process disappears while an
InferenceSessionis being created. There is nomanaged exception and nothing reaches the log: the file ends on an ordinary
informational line, and
AppDomain.UnhandledExceptionprints nothing. The onlytrace left behind is a Windows ".NET Runtime" 1026 event with
0x80131506(
COR_E_EXECUTIONENGINE), i.e. a native fault that is fatalized at the P/Invokeboundary and never becomes a catchable exception.
Switching the runner to CPU makes it go away completely, which points at the
DirectML execution provider rather than at the rendering code.
Why the package version is the only variable
Every other candidate was checked:
session-creation path, no crash.
InferenceSessionprofiling on a realDiffSinger
dur.onnxshows 252 of 258 nodes still executing onDmlExecutionProviderunder both 1.23.0 and 1.24.4, and the per-nodeassignment is identical. Nothing silently falls back to CPU in a way that
would explain a difference between the two versions.
DirectML.dllis the same file. Both packages depend onMicrosoft.AI.DirectML 1.15.4, so the native DirectML runtime is not thevariable.
onnxruntime/core/providers/dmlbetweenv1.23.0andv1.24.4shows onlytwo changed blobs, and both are additive: two declarations in
DmlExecutionProvider/inc/DmlExecutionProvider.hand two exported APIs indml_provider_factory.cc. Kernel implementations, allocators, the executioncontext and
GraphDescBuilderare byte-identical, and graph partitioningbehaves identically.
AppendExecutionProvider(OrtEnv, OrtEpDevice[], opts)call introduced byfeat(onnx): drop Vortice.DXGI & update dml to 1.23.0 #1774 is byte-identical across the branches involved.
So the NuGet package version is what is left. This is the change that made it
move: #1722 ("Add CUDA rendering support on Linux") pushed the Windows group from
1.23.0 to 1.24.4 while adding the Linux CUDA package. The fix here simply
restores the version that Windows had been on since #1774 and for the ~10 months
in between, so it is not new territory for this codebase.
Scope
Only the Windows group is touched.
(
Microsoft.ML.OnnxRuntime.Gpu.Linux) has no 1.23.x release at all — it startsat 1.24.3 — and the CUDA detection added by Add CUDA rendering support on Linux #1722 expects the newer one.
Why this is a downgrade rather than a normal fix
To be explicit: this pins around an upstream regression, it does not fix it.
There is no upstream version to wait for — DirectML was moved to sustained
engineering in 2025-09,
Microsoft.ML.OnnxRuntime.DirectMLstops at 1.24.4,and the robustness fixes made after it (including
microsoft/onnxruntime#28007
"dml: add per-instance mutexes to fix concurrent session crashes", still open, as
well as #31719 and #32745) will never ship as a package upgrade. Given that, a
pin is the only mitigation that can actually reach users of the Windows DirectML
path. A proper fix means moving off DirectML — the CPU runner is known not to
crash — and that is a separate, larger change.
Testing
Windows 11 Pro 22H2 (build 22621), x64, NVIDIA RTX 2070, ONNX Runner set to
DirectML.
dotnet build OpenUtau -c Releaseon this branch: 0 errors.runtimes/win-x64/native/onnxruntime.dllis1.23.20250926.7and the managedMicrosoft.ML.OnnxRuntime.dllis1.23.0.0(previously1.24.20260316.9/1.24.4).runner and with the 1.23.0 build it does not reappear in the same usage.
I have only tested on Windows; the Linux and macOS groups are untouched and I did
not build or run them. Flagging this for the maintainers since it changes a
dependency version.