What causes "Hyperliquid WebSocket handler did not stop after abort" in nautechsystems/nautilus_trader?

Add read/send timeouts to the underlying WebSocket connection so a dead socket cannot block the handler indefinitely.

nautechsystems/nautilus_trader · error · anyhow::Error

Hyperliquid WebSocket handler did not stop after abort

Error message

Hyperliquid WebSocket handler did not stop after abort

What it means

Raised in disconnect_locked when, after sending an abort signal to the WebSocket handler task, the joined task reports TaskJoinOutcome::Incomplete: the task neither completed nor returned an abort outcome within the expected window. The handler ignored or missed the abort and may still be running, so disconnect cannot be confirmed complete. Rate-limit reservations are released before bailing.

Source

Thrown at crates/adapters/hyperliquid/src/websocket/client.rs:586

        if self.task_handle.is_empty() {
            log::debug!("No task handle to await");
        } else {
            log::debug!("Waiting for task handle to complete");

            if let Some(outcome) = self
                .task_handle
                .finish(Duration::from_secs(2), Duration::from_secs(2))
                .await
            {
                match outcome {
                    TaskJoinOutcome::Completed(()) | TaskJoinOutcome::Aborted => {}
                    TaskJoinOutcome::Failed(error) => {
                        self.release_limit_reservations();
                        anyhow::bail!("Hyperliquid WebSocket handler failed: {error}");
                    }
                    TaskJoinOutcome::Incomplete => {
                        self.release_limit_reservations();
                        anyhow::bail!("Hyperliquid WebSocket handler did not stop after abort");
                    }
                }
            }
        }
        self.release_limit_reservations();
        log::debug!("Disconnected");
        Ok(())
    }

    /// Requests a full transport reconnect.
    ///
    /// Transitions the connection from `Active` to `Reconnect`; the network
    /// layer re-establishes the socket with backoff and the handler replays all
    /// active subscriptions once reconnected. Returns `false` when the
    /// connection is not active (already reconnecting, disconnecting, or
    /// closed), leaving any in-flight transition untouched.
    pub fn request_reconnect(&self) -> bool {
        ConnectionMode::request_reconnect(&self.connection_mode.load())

View on GitHub (pinned to 18893faf8b)

Solutions

Add read/send timeouts to the underlying WebSocket connection so a dead socket cannot block the handler indefinitely.
Ensure the handler's main loop selects on the abort/shutdown signal (tokio::select!) so aborts take effect between awaits.
Retry disconnect() or drop the client; the task handle is aborted and reservations released, so a fresh client is safe.
If it recurs, use tokio-console to find the non-cancellable await and make it cancellable.

Example fix

// before: read can hang forever on dead socket / loop { let msg = ws.read().await?; handle(msg); } / // after: abort observed promptly / loop { tokio::select! { msg = ws.read() => handle(msg?), _ = shutdown_rx.changed() => break } }

Defensive patterns

Strategy: retry

Try / catch

match client.disconnect().await { Err(e) if e.to_string().contains("did not stop after abort") => { log::warn!("handler ignored abort; forcing teardown"); drop(client); client = HyperliquidWebSocketClient::new(url, ...); } Err(e) => return Err(e.into()), Ok(()) => {} }

Prevention

Ensure WebSocket reads/writes carry timeouts so dead sockets cannot stall the handler.
Write the handler loop with tokio::select! over the shutdown signal.
Treat incomplete shutdown as requiring a client rebuild.
Use tokio-console or tracing to find non-cancellable awaits if this recurs.

When it happens

Trigger: Calling disconnect() (or the reconnect path in connect_locked) when the handler task is blocked in a non-cancellation-safe await, e.g. stuck on a hung WebSocket read or a send that never resolves, so the abort signal is observed too late or never.

Common situations: Dead network connection where the TCP read hangs without timeout; handler loop without a select! on the abort/shutdown signal; half-open socket stalls delaying task exit; slow exchange responses during shutdown.

Understand the failure class

Background: Request timed out: what client-side request timeouts mean across libraries (Request timed out, TIMED_OUT, APITimeoutError) — this error's family across 39 libraries.

Related errors

AI-assisted analysis of nautechsystems/nautilus_trader@18893faf8b (2026-09-08). Data as JSON: /api/errors/8906ebf1c8481d15. Report an issue: GitHub.