nautechsystems/nautilus_trader · warning · anyhow::Error

Failed to terminate Hyperliquid data tasks: {e}

Error message

Failed to terminate Hyperliquid data tasks: {e}

What it means

During teardown, await_pending_tasks asks the Hyperliquid data client's pending task set to shut down with bounded grace (1s) and drain (2s) timeouts. If tasks do not terminate within those bounds, finish_shutdown returns an error which is wrapped as 'Failed to terminate Hyperliquid data tasks'. This indicates stuck background data tasks at disconnect time.

Source

Thrown at crates/adapters/hyperliquid/src/data.rs:260

        if let Err(e) = self.await_pending_tasks().await {
            self.shutdown_errors.push(e.to_string());
        }
        self.clear_stream_health();
        self.is_connected.store(false, Ordering::Release);

        if !self.shutdown_errors.is_empty() {
            anyhow::bail!(std::mem::take(&mut self.shutdown_errors).join("; "));
        }
        Ok(())
    }

    async fn await_pending_tasks(&self) -> anyhow::Result<()> {
        self.pending_tasks.begin_shutdown();
        self.pending_tasks
            .finish_shutdown(Duration::from_secs(1), Duration::from_secs(2))
            .await
            .map_err(|e| anyhow::anyhow!("Failed to terminate Hyperliquid data tasks: {e}"))?;
        Ok(())
    }

    async fn await_session_tasks(&self) -> anyhow::Result<()> {
        self.session_tasks.begin_shutdown();
        self.session_tasks
            .finish_shutdown(Duration::from_secs(1), Duration::from_secs(2))
            .await
            .map_err(|e| {
                anyhow::anyhow!("Failed to terminate Hyperliquid data session tasks: {e}")
            })?;
        Ok(())
    }

    fn clear_stream_health(&self) {
        self.stream_health.lock().clear();
    }

View on GitHub (pinned to 18893faf8b)

Solutions

1. Investigate which pending task hangs (enable task logging) and ensure it respects the shutdown signal and cancellation tokens
2. Add timeouts to individual in-flight requests so they cannot block teardown indefinitely
3. Increase finish_shutdown grace/drain durations if tasks legitimately need longer to wind down
4. Retry disconnect; if persistent, restart the client/node since a stuck task indicates a bug

Example fix

// before
.finish_shutdown(Duration::from_secs(1), Duration::from_secs(2))
// after
.finish_shutdown(Duration::from_secs(5), Duration::from_secs(10)) // or fix the hanging task to honor shutdown

// best-effort pre-check: ensure no task is mid-request before teardown
if !pending_tasks.is_empty() {
    tracing::warn!("tearing down with {} pending tasks", pending_tasks.len());
}

fn shutdown_within(d: Duration) -> impl Future<Output=bool> {
    async move {
        tokio::time::timeout(d, tasks.finish_shutdown(Duration::from_secs(1), Duration::from_secs(2)))
            .await.is_ok()
    }
}

if let Err(e) = await_pending_tasks().await {
    tracing::warn!("{e}; forcing abort of pending tasks");
    pending_tasks.abort_all();
}

When it happens

Trigger: Calling teardown_partial_connect -> await_pending_tasks while a pending task (e.g. an in-flight HTTP/WS request loop) hangs and does not finish within the 1s begin/finish shutdown grace plus 2s drain window.

Common situations: Network stalls blocking in-flight requests during disconnect; task loops not observing the shutdown signal promptly; slow exchange responses at teardown time; deadlocks or very long awaits inside spawned tasks.

Understand the failure class

Background: Request timed out: what client-side request timeouts mean across libraries (Request timed out, TIMED_OUT, APITimeoutError) — this error's family across 39 libraries.

Related errors

• Failed to finish Betfair data command tasks: {e}
• Failed to terminate Hyperliquid data session tasks: {e}
• Timeout waiting for account {account_id} to be registered af
• {shutdown_errors joined with "; "}
• Hyperliquid WebSocket handler did not stop after abort

AI-assisted analysis of nautechsystems/nautilus_trader@18893faf8b (2026-09-08). Data as JSON: /api/errors/5c9f9b0d25d4f706. Report an issue: GitHub.