Mjcarroll/windows socket exhaustion - #1241
Conversation
Try to cure windows socket exhaustion issues we are seeing in CI Signed-off-by: Michael Carroll <mjcarroll.oss@gmail.com>
Signed-off-by: Michael Carroll <mjcarroll.oss@gmail.com>
…event socket exhaustion Check if daemon is already running in spawn_daemon() before trying to bind. This preserves the single-daemon constraint while allowing allow_reuse_address to be True and enabling SO_REUSEADDR socket option on all platforms (including Windows). This fixes WSAEADDRINUSE / WinError 10055 socket exhaustion during TCP TIME_WAIT states. Signed-off-by: Michael Carroll <mjcarroll.oss@gmail.com> Assisted-by: Gemini CLI:Gemini 3.5 Flash [view_file, grep_search, list_dir, search_web, run_command, replace_file_content]
91681f8 to
0b73f57
Compare
fujitatomoya
left a comment
fujitatomoya
left a comment
There was a problem hiding this comment.
fix looks good, but i am not 100% sure if this solves the problem for the windows...
starting CI to see if what is gonna happen 👀
|
Pulls: #1241
|
Yeah, at this point I'm kind of just attempting things to see if it makes a difference. It does seem that the real underlying cause is that we are running out of sockets on Windows. Based on my reading, sockets will still live for a few minutes after a process terminates, so I'm experimenting with more aggressively cleaning them up before we shutdown. |
The added teardown blocks in the daemon/strategy test fixtures left only one blank line before the following top-level def/decorator instead of the required two.
The receive thread only handled TimeoutError, so any other error became a pytest unhandled-thread-exception warning. test_api then failed as "assert b'test_api' == None", which says nothing about the actual cause, and test_port_mismatch could pass despite the receive never happening. Stash the error and re-raise it on the calling thread instead. Multicast joins and UDP binds draw on resources that are shared by every process on the machine, so these tests fail when something unrelated has exhausted them. Record the membership count, the number of open UDP endpoints, and the dynamic port range alongside the failure so a CI log is enough to tell an exhausted machine from a defect here. Also re-raise in test_group_mismatch. On Windows any OSError that was not 10051 fell through the conditional and was discarded, so the test passed while hiding the same failure its neighbours report. Refs: #1141
2 participants
Description
A few fixes to try to clear up Windows socket exhaustion issues.
Fixes #1229
ros2cli:spawn_daemonreturns early when a daemon is already running instead of racing to bind an address that is already taken.SO_REUSEADDRfor the daemon address check and forLocalXMLRPCServeron Windows too, so a socket inTIME_WAITis not reported as busy.local_nodefixture, and shut down any daemon left behind bytest_strategy, so tests stop leaking middleware sockets into the ones that follow.ros2multicast— better reporting for #1141:TimeoutError, so any other error became a pytest unhandled-thread-exception warning.test_apithen failed asassert b'test_api' == None, which says nothing about the cause, andtest_port_mismatchcould pass even though the receive never happened. The error is now re-raised on the calling thread.test_group_mismatchfell through its conditional and discarded anyOSErrorthat was not 10051, so it passed. That is why it is absent from the failure list in Win 11 - ros2_multicast test_api regressions on release jobs #1141 while its three neighbours fail on the same error.Is this user-facing behavior change?
No
Did you use Generative AI?
Assissted-by: Antigravity CLI (Gemini 3.5 Flash)
Assisted-by: Claude Code (Opus 5)
Additional Information
The
ros2multicasthalf does not fix #1141; it makes the next failure legible. Worth knowing while reading it: on Windows the UDP and TCP ephemeral pools are separate, and the UDP one defaults to 16384 ports from 49152. When it is exhausted,bind(('', 0))fails withWSAENOBUFS(10055) and an implicit bind fails withWSAEINVAL(10022), while TCP bind/listen/connect keep working. A socket bound to a fixed port still binds and joins fine in that state, so the 10055 reported in #1141 — which comes fromIP_ADD_MEMBERSHIPafter a successful fixed-port bind — is not plain port exhaustion.