Skip to content

fix(runner): reuse connections to runtimes across executions - #259

Open
HarshMN2345 wants to merge 1 commit into
mainfrom
fix/reuse-runtime-connections
Open

HarshMN2345 wants to merge 1 commit into
mainfrom
fix/reuse-runtime-connections

Conversation

@HarshMN2345

Copy link
Copy Markdown
Member

The executor opened a new connection to the runtime for every execution (curl_init() per call). Under load the closed connections pile up in TIME_WAIT in the executor's network namespace, connect() gets expensive and eventually fails with EADDRNOTAVAIL.

This keeps idle curl handles per runtime hostname in each worker and reuses them (curl_reset() keeps the open connection), for v2 and v5 executions. A handle is only returned to the pool after a successful request, at most 128 are kept per runtime, and handles of runtimes not used for 60 s are closed. Runtime hostnames are random per runtime, so a recreated runtime never gets an old connection, and libcurl replaces connections that the runtime closed (tested with a runtime restart).

A shared connection cache (curl_share_init_persistent with CURL_LOCK_DATA_CONNECT) doesn't work here: Swoole's curl hook runs each curl_exec() in its own multi handle, which keeps at most 4 connections, and moving sockets between multi handles crashed Swoole under load.

Refs appwrite/appwrite#14094

Verification: real createExecution() against a keep-alive runtime stand-in, 20000 executions with 100 coroutines, 4 runs back to back:

before after
exec/s per run 3245, 332, 355, 1387 ~28000 each
new connections per run 20000 100
TIME_WAIT per run +20000 0
errors 300 per run in runs 2-3 (curl error 7) none

Unit tests, phpstan, pint and rector pass.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant