nautechsystems/nautilus_trader · error · TransportError

Failed to send update command: {e}

Error message

Failed to send update command: {e}

What it means

After a reconnect establishes a new socket, the client sends a WriterCommand::Update to the writer task over an mpsc channel so it can swap in the new writer before the state machine returns to ACTIVE. If the writer task has already exited (its receiver dropped), the send fails and this BrokenPipe TransportError is returned. It means the internal writer task is gone, so the reconnect cannot complete through the normal path.

Source

Thrown at crates/network/src/websocket/client.rs:1348

                std::io::ErrorKind::TimedOut,
                format!(
                    "reconnection timed out after {}s",
                    self.connect_timeout.as_secs_f64()
                ),
            ))
        })??;

        if ConnectionMode::from_atomic(&self.connection_mode).is_disconnect() {
            log::debug!("Reconnect aborted mid-flight (after connect)");
            return Ok(ReconnectOutcome::Aborted);
        }

        // Use a oneshot channel to synchronize the writer swap before transitioning
        // back to ACTIVE. Buffered messages stay in the writer task and replay later.
        let (tx, rx) = tokio::sync::oneshot::channel();
        if let Err(e) = self.writer_tx.send(WriterCommand::Update(new_writer, tx)) {
            log::error!("{e}");
            return Err(TransportError::Io(std::io::Error::new(
                std::io::ErrorKind::BrokenPipe,
                format!("Failed to send update command: {e}"),
            )));
        }

        // Wait for writer to confirm it accepted the new socket
        let connection_epoch = match rx.await {
            Ok(connection_epoch) => {
                log::debug!("Writer confirmed socket update: epoch={connection_epoch}");
                connection_epoch
            }
            Err(e) => {
                log::error!("Writer dropped update channel: {e}");
                return Err(TransportError::Io(std::io::Error::new(
                    std::io::ErrorKind::BrokenPipe,
                    "Writer task dropped response channel",
                )));
            }

View on GitHub (pinned to 18893faf8b)

Solutions

1. Log and inspect writer-task lifetime; ensure the writer loop only exits on deliberate shutdown commands
2. Retry the reconnect from scratch (recreate client/writer) rather than reusing a half-dead client instance
3. Check writer task code for panics (e.g. on poisoned locks) and make the loop resilient to transient socket errors
4. If shutdown is concurrent with reconnect, gate reconnect on a connection-state check

fn is_broken_pipe(e: &TransportError) -> bool {
    matches!(e, TransportError::Io(io) if io.kind() == std::io::ErrorKind::BrokenPipe)
}

match reconnect_with_outcome().await {
    Err(e) if is_broken_pipe(&e) => {
        // writer task dead: rebuild client or rerun full connect
        rebuild_and_connect().await?;
    }
    r => r?,
}

When it happens

Trigger: writer_tx.send(WriterCommand::Update(..)) failing because the writer task was aborted, panicked, or its loop exited (e.g. after an earlier disconnect or fatal send error) while reconnect_with_outcome was in progress.

Common situations: Writer task killed by an unrelated panic during a network blip; client shutting down concurrently with reconnect; a prior unrecoverable writer error ended the task but reconnect was still attempted.

Related errors

• Writer task dropped response channel
• Binance Futures data teardown failed: {}
• Binance Spot data teardown failed: {}
• Lighter WebSocket handler failed: {error}
• Lighter WebSocket initial connection timeout after {} second

AI-assisted analysis of nautechsystems/nautilus_trader@18893faf8b (2026-09-08). Data as JSON: /api/errors/5252b1f7dc9bf0cb. Report an issue: GitHub.