Skip to content

fix(server): loopback-bind the TP rendezvous store, add --dist-port - #461

Open
shikhakath wants to merge 1 commit into
FlashML-org:mainfrom
shikhakath:fix-301-port-override
Open

shikhakath wants to merge 1 commit into
FlashML-org:mainfrom
shikhakath:fix-301-port-override

Conversation

@shikhakath

Copy link
Copy Markdown

Fixes #301.

The TP rendezvous port was hardcoded to server_port + 1, so two instances on adjacent ports fight over it. There's also a second issue the thread surfaced: even though the init URL says tcp://127.0.0.1:PORT, torch's TCPStore ignores the host you give it and binds every interface -- so today it's an unauthenticated store reachable off-box on any bare-metal or --network host setup.

This keeps TCP (per the thread's discussion, since a future multi-node rendezvous still needs it) but builds the store explicitly instead of trusting
init_method: rank 0 pre-binds a socket to 127.0.0.1 and hands the fd to
TCPStore via master_listen_fd. Other ranks connect as normal clients.
--dist-port makes the port configurable, defaulting to --port + 1 so nothing changes for existing users.

Testing

I don't have a GPU, and couldn't even install the package to run it locally (flashlib has no macOS wheel) -- so no checkpoint, no ft serve, no driver to report.

What I could check: I reproduced the exact logic standalone -- pre-bind a loopback socket, pass it to TCPStore via master_listen_fd, spawn two processes and run init_process_group + all_reduce over gloo. The socket only binds to 127.0.0.1, and the rendezvous still works. I've added that as two real tests
(tests/engine/test_distributed_store.py, tests/server/test_dist_port_arg.py),
but haven't run them against this actual package -- someone with an install should confirm they pass before merge.

Completely untested: the NCCL/TP>=2 path in _init_communication. Same gap nobody in the original thread could close either. Happy to test on a multi-GPU box if someone points me at one.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

fix(server): allow overriding the distributed port

1 participant