Skip to content

Firmware: a weak or lost Wi-Fi link no longer freezes the device - #105

Open
tozes wants to merge 1 commit into
Adolanium:mainfrom
tozes:firmware/websocket-link-loss
Open

tozes wants to merge 1 commit into
Adolanium:mainfrom
tozes:firmware/websocket-link-loss

Conversation

@tozes

@tozes tozes commented Oct 9, 2026

Copy link
Copy Markdown

What and why

On a Wi-Fi loss the device can freeze until a hard reset: the screen stops updating, touch does nothing, and Wi-Fi never reconnects. #103 records this on an AMOLED-1.75C.

The app task closes the WebSocket when the network goes down. WsTransport::close() called esp_websocket_client_close(), which in esp_websocket_client 1.8.0 sends its close frame with portMAX_DELAY whatever timeout it is given (esp_websocket_client_close_with_optional_body). With the link gone, the TLS socket never becomes writable, so the app task blocks in select() until lwIP gives up on the TCP connection, minutes later or never. Because the app task also drives the screen, touch and the Wi-Fi retries, everything stops.

The change:

  • Sends get their own task (hg-ws-tx). send_text/send_binary queue a copy and return; the task writes with a 2 s limit. A slow link therefore no longer stalls the app task, or the screen, for up to 2 s per send.
  • Closing goes through that task too. It sends the close frame with esp_websocket_client_send_with_opcode() under a 1 s limit (skipped when the link is already gone), then destroys the client after its queued messages.
  • Events carry their connection. Each client's events carry its own connection generation (user_context), so a client still being torn down can't be mistaken for the current one.
  • A lost connection can't go unnoticed. It is also recorded in the transport and picked up by the main loop: the library reports a failed send from the sending task, and the WsClosed event could be dropped when the event queue was full.

How you know it works

Tests, on this branch (based on a321be5):

  • hermes-gadget build-sim --test: core 113/113, including 15 new cases in tests/test_ws_link.cpp. The decisions live in drivers/ws_link.hpp: which connection is current, reporting a loss once, the send queue's reserved close slots, dropping stale messages, closing after queued messages, and a time limit on every network wait. Breaking each rule on purpose (no stale-message drop, an unlimited close frame, reporting a lost old connection) made the matching tests fail.
  • tests/test_firmware_source.py fails if firmware calls esp_websocket_client_close() again. The library bug itself can't be reproduced on the host.
  • Full pytest without the simulator-window tests (no virtual display here): 266 passed, 29 skipped.
  • pio run for every board in platformio.ini (11): all build, each app 1285–1302 KB (34–35% of the slot free).

Physical checks on a Waveshare AMOLED-1.75C (SKU 33691), separate from the tests:

  • Before: the freeze reproduced twice by weakening the signal (a hand over the board). A JTAG backtrace of the frozen board showed the app task in esp_vfs_select(timeout=NULL), under esp_transport_poll_write(-1), esp_websocket_client_send_close(timeout=portMAX_DELAY), esp_websocket_client_close(), WsTransport::close() and App::on_network(false).
  • After, the same test: Wi-Fi dropped and reconnected with no freeze. The screen and touch kept responding throughout (including while recording), and a stalled connection was noticed within milliseconds and reopened.
  • These ran on builds of a working branch that also carried other changes for this board, submitted separately; this branch's own image wasn't installed on its own.

Documentation and sources

The change is internal, so no guide changes; a changelog line under ### Firmware. The fix follows esp_websocket_client 1.8.0's source (esp_websocket_client.c); no code was copied.

Checklist

  • One change per PR, with a title that says what it does
  • Relevant docs updated, or the reason no changes apply is explained
  • Verification results and remaining limitations recorded
  • Applicable vendor sources, licenses, and contributor credit retained
  • For a new board: the checklist in CONTRIBUTING.md (not a new board)

On a Wi-Fi loss the app task closes the WebSocket. WsTransport::close()
called esp_websocket_client_close(), which in esp_websocket_client 1.8.0
sends its close frame with portMAX_DELAY whatever timeout it is given
(esp_websocket_client_close_with_optional_body). With the link gone the
TLS socket never becomes writable, so the app task blocked in select()
until lwIP gave up on the TCP connection, minutes later or never. The app
task also drives the screen, touch and Wi-Fi retries, so the device froze
until a hard reset. A JTAG backtrace of a frozen board showed exactly that
chain, from App::on_network(false) down to esp_vfs_select(timeout=NULL).

- Sends move to their own task (hg-ws-tx): send_text/send_binary queue a
  copy and return, and the task writes with a 2 s limit. A slow link no
  longer stalls the app task, and so the screen, for up to 2 s per send.
- close() hands the client to that task, which sends the close frame with
  esp_websocket_client_send_with_opcode() under a 1 s limit (skipped when
  the link is already gone) and destroys the client after its queued
  messages.
- Each client's events carry its own connection generation
  (user_context), so a client still being torn down can't be mistaken for
  the current one.
- A lost connection is also recorded in the transport and picked up by
  the main loop: the library reports a failed send from the sending task,
  and its WsClosed event could be dropped when the event queue was full.

The decisions live in drivers/ws_link.hpp with host tests
(tests/test_ws_link.cpp, 15 cases). The library bug itself can't be
reproduced on the host; tests/test_firmware_source.py fails if firmware
calls esp_websocket_client_close() again.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant