Service request timeouts missing in RCL → RMW interaction

[mini-1235]

Description

In the current RCL/RMW design, service requests are assumed stateless at the RMW layer. This is affecting non-dds implementations like Zenoh, see ros2/rmw_zenoh#783 (comment) for more information

Motivation

We are trying to add zenoh to Nav2's CI ros-navigation/navigation2#5516, which fails because Zenoh server will fail if the service call is longer than the queries_default_timeout set in the Zenoh config

Design / Implementation Considerations

No response

Additional Information

No response

[mini-1235]

@fujitatomoya @christophebedard I am not sure who to ping, would you be able to take a look? Thanks!

[fujitatomoya]

i am not familiar with the context for rmw_zenoh, but can we spin it until it times out and application can remove the pending request?

[JEnoch]

More context on the issue.

The current approach for sending a request with a timeout involves the user calling:

// Send the Request object - no timeout passed here
auto future_result = client->async_send_request(request);
// Wait for completion or timeout - managed in RCL, RMW is not called and the Request/Query is already sent anyway
if (spin_until_complete(future_result, timeout) != rclcpp::FutureReturnCode::SUCCESS) {
    // Pending request must be manually cleaned up if execution is interrupted or timed out
    client_->remove_pending_request(future_result);
    throw std::runtime_error(service_name_ + " service client: async_send_request failed");
}

The RCL architecture treats Service Requests like message publications, expecting the middleware to remain stateless regarding responses or timeout management.

• None of the Client::async_send_request() operations takes a timeout argument
• The timeout is managed in RCL, spining on the future until timeout, without involving RMW.
• The Client::remove_pending_request() operation cleans the pending requests state in RCL, but doesn't call RMW.

DDS-based RMWs may operate statelessly for pending requests, but rmw_zenoh cannot: its use of Zenoh’s Query/Queryable primitives means the session must maintain state to correlate replies, and routers (when not in peer-to-peer mode) retain "breadcrumb" state for reply routing. If no reply arrives, Zenoh must actively clear these states. Thus, every Query in Zenoh requires a timeout, falling back to a configurable default if omitted.

This default timeout is configured to 60 seconds by rmw_zenoh.
While I wasn't expecting any call to a ROS service to take more than 1 minute, that seems to be the case at least in Nav2 tests:
ros-navigation/navigation2#5516
More generaly the problem is that, while the code could wait for a response with a timeout >1 minute, the queries_default_timeout set in the configuration prevails and any response sent after this timeout is dropped.

While I agree that it would be ideal for the user-specified timeout to be enforced at the RMW level—if supported—this would necessitate modifying user code to pass the timeout alongside the Request, rather than in the spin_until_complete() call:

// Send the Request object, with a timeout - the timeout is passed to RMW
auto future_result = client->async_send_request(request, timeout);
// Wait for completion or timeout
if (spin_until_complete(future_result) != rclcpp::FutureReturnCode::SUCCESS) {
    // Manual cleanup might still be required by some RMWs
    client_->remove_pending_request(future_result);
    throw std::runtime_error(service_name_ + " service client: async_send_request failed");
}

It’s probably feasible to implement this change without breaking the API by adding the timeout as an optional parameter with a default value. However, for backward compatibility, the timeout argument would need to remain in spin_until_complete(). This raises the question: what happens if the two timeouts are inconsistent ?

---

Another, less disruptive option is to configure rmw_zenoh to use MAX_UINT64 as the timeout for Service calls and depend on Client::remove_pending_request() to call a new rmw_remove_pending_request(request_id) operation for cleanup. The downside is that, in rmw_zenoh, any failure to call this function or if a Service Client crash would leave the routers' pending state foreaver.

[ros-discourse]

This issue has been mentioned on Open Robotics Discourse. There might be relevant details there:

https://discourse.openrobotics.org/t/ros-pmc-minutes-for-september-30-2025/50379/1

[jmachowinski]

The missing timeout is a constant pain when dealing with the services. One needs to write awkward boiler plate with an additional timer as proposed in the examples. If there is a good reason now to add a timeout, I would personally go all in on it and add a proper API even if it is breaking.

Please note that the example you showed specifies the 'blocking' synchronous use case. We don't use that at all but rely on a callback driven interface. Therefore in order to support this use case you would need to add a timeout parameter and a timeout callback as well.

[JEnoch]

As discussed during the PMC meeting on 30/9, we will proceed with implementing the alternative option as a first step:

Action Plan

• Add a new function to the RMW interface to remove a pending request.
• Update RCL APIs to call this new RMW function when remove_pending_request(request_id) is invoked.
• Implement the new RMW function:
  • Ideally as a default no-op in ros2/rmw. That would avoid changes in DDS-based RMWs and ease backporting.
  • In ros2/rmw_zenoh: cancel the pending Zenoh query.
• Then the queries_default_timeout setting in rmw_zenoh default config will be increased to a very large duration.

Regarding Adding a Timeout for Sending Requests

The following points were highlighted during the PMC meeting:

• Adding a timeout parameter to send request operations is independent of adding a cleanup function in RMW and can be addressed later.
• Even if a timeout parameter is added, it does not prevent the RMW implementation or middleware from performing automatic cleanup (e.g., when the Request object is deleted or the Client crashes).
• The timeout used in spin_until_complete() is not directly related to the request timeout. Users may call spin_until_complete() multiple times with short timeouts to perform other tasks while waiting for a reply.
• It is possible for a Service Server to take an "arbitrarily long time" to respond (e.g., services requiring user interaction).
• For rmw_zenoh, a default timeout must be retained but should be large by default and configurable for users requiring longer timeouts. This is already the case:
  • The timeout is configurable via queries_default_timeout in the $ZENOH_SESSION_CONFIG_URI file.
  • The default value was recently increased to 10 minutes (Config: increase queries_default_timeout to 10min rmw_zenoh#820). Once the cancelation of pending request is implemented in rmw_zenoh, we can set this default value to a very large value (24 hours ?) and rely on the user to call remove_pending_request(request_id).

[JEnoch]

Hi @jmachowinski

The missing timeout is a constant pain when dealing with the services. One needs to write awkward boiler plate with an additional timer as proposed in the examples. If there is a good reason now to add a timeout, I would personally go all in on it and add a proper API even if it is breaking.

I agree. But we'll go first with the plan explained in comment above.

Changing the all the RCL APIs is more work for almost all packages and is a breaking change which cannot be backported.

Please note that the example you showed specifies the 'blocking' synchronous use case. We don't use that at all but rely on a callback driven interface. Therefore in order to support this use case you would need to add a timeout parameter and a timeout callback as well.

Wouldn’t a single callback with a signature that handles both cases (response or timeout) be more practical?