nautechsystems/nautilus_trader · error · anyhow::Error

Live runner system command channel is closed

Error message

Live runner system command channel is closed

What it means

After building the ReconnectSocket command, DataActor::reconnect_socket sends it over the system command channel. If the receiver end has been dropped the send fails, and the method maps that to 'Live runner system command channel is closed', meaning the live runner's system command consumer no longer exists.

Source

Thrown at crates/common/src/actor/data_actor.rs:5767

    pub fn reconnect_socket(&self, client_id: ClientId, endpoint: &str) -> anyhow::Result<()> {
        let endpoint = socket_endpoint(endpoint)?;

        if !self.is_properly_registered() {
            anyhow::bail!(
                "Actor {} has not been registered with a Trader",
                self.actor_id
            );
        }

        let sender = try_get_system_command_sender()
            .ok_or_else(|| anyhow::anyhow!("Live runner system command channel is unavailable"))?;
        let trader_id = self
            .trader_id
            .ok_or_else(|| anyhow::anyhow!("Actor {} has no trader ID", self.actor_id))?;
        let command = ReconnectSocket::new(trader_id, client_id, endpoint, self.timestamp_ns());
        sender
            .send(SystemCommand::ReconnectSocket(command))
            .map_err(|_| anyhow::anyhow!("Live runner system command channel is closed"))?;
        Ok(())
    }

    #[cfg(test)]
    pub fn quote_handler_count(&self) -> usize {
        self.quote_handlers.len()
    }

    #[cfg(test)]
    pub fn trade_handler_count(&self) -> usize {
        self.trade_handlers.len()
    }

    #[cfg(test)]
    pub fn bar_handler_count(&self) -> usize {
        self.bar_handlers.len()
    }

View on GitHub (pinned to 18893faf8b)

Solutions

1. Check node lifecycle state before calling reconnect_socket; only invoke it while the live runner is running.
2. Restart the trading node if its system command loop has shut down — the channel cannot be reattached at runtime.
3. Inspect runner logs for why the system command consumer exited (panic, task cancellation) and fix the root cause.
4. Handle the error gracefully in actor code (log and schedule reconnect after the node is back up) instead of retrying immediately.

Example fix

// before
actor.reconnect_socket(client_id, endpoint)?; // fails after node stopped

// after
if node.is_running() {
    actor.reconnect_socket(client_id, endpoint)?;
} else {
    tracing::warn!("node not running; cannot send ReconnectSocket");
}

// Rust
if !node.is_running() {
    tracing::warn!("node not running; defer reconnect_socket");
    return Ok(());
}

match actor.reconnect_socket(client_id, endpoint) {
    Ok(()) => {},
    Err(e) if e.to_string().contains("channel is closed") => {
        tracing::warn!("system command channel closed; will retry after restart");
        schedule_reconnect_after_restart(client_id, endpoint);
    }
    Err(e) => return Err(e),
}

When it happens

Trigger: Calling reconnect_socket when the system command sender exists but its receiving side (the live runner's command loop) has been shut down or dropped — e.g. after node stop/disconnect or if the runner's command-handling task terminated.

Common situations: Requesting a socket reconnect during or after live-node shutdown; the live runner's system command task crashed or exited while actors keep running; holding an actor handle and calling reconnect after the TradingNode was stopped.

Related errors

• {FAILED_TX_CHANNEL}: {e}
• Close command should not be drained
• Betfair data shutdown failed: {}
• Betfair execution shutdown failed: {}
• failed to finish Binance Futures data command tasks: {e}

AI-assisted analysis of nautechsystems/nautilus_trader@18893faf8b (2026-09-08). Data as JSON: /api/errors/470730fec9d9c557. Report an issue: GitHub.