What causes "Failed to send to channel: {e}" in nautechsystems/nautilus_trader?

Verify the background Redis cache task is running and healthy before insert; restart/reinitialize it if it stopped.

nautechsystems/nautilus_trader · critical

Failed to send to channel: {e}

Error message

Failed to send to channel: {e}

What it means

RedisCache::insert enqueues a DatabaseCommand::Insert onto an mpsc channel serviced by a background Redis task. If that task has already terminated (receiver dropped), send fails and the error is surfaced as "Failed to send to channel: <send error>". The write never reaches Redis.

Source

Thrown at crates/infrastructure/src/redis/cache.rs:497

            let result = DatabaseQueries::load_custom_data(&con, &trader_key, &data_type).await;
            if let Err(e) = tx.send(result) {
                log::error!("Failed to send custom data result for '{data_type}': {e:?}");
            }
        });

        blocking_recv(&rx).map_err(|e| anyhow::anyhow!("load_custom_data channel closed: {e}"))?
    }

    /// Sends an insert command for `key` with optional `payload` to Redis via the background task.
    ///
    /// # Errors
    ///
    /// Returns an error if the command cannot be sent to the background task channel.
    pub fn insert(&self, key: String, payload: Option<Vec<Bytes>>) -> anyhow::Result<()> {
        let op = DatabaseCommand::new(DatabaseOperation::Insert, key, payload);
        match self.tx.send(op) {
            Ok(()) => Ok(()),
            Err(e) => anyhow::bail!("{FAILED_TX_CHANNEL}: {e}"),
        }
    }

    /// Stores custom data in Redis (key format: `custom:<ts_init_020>:<uuid>`, value: full JSON).
    ///
    /// # Errors
    ///
    /// Returns an error if serialization fails or the insert command cannot be sent.
    pub fn add_custom_data(&self, data: &CustomData) -> anyhow::Result<()> {
        let json_bytes = serde_json::to_vec(data)
            .map_err(|e| anyhow::anyhow!("CustomData serialization failed: {e}"))?;
        let ts_init = data.ts_init().as_u64();
        let key = format!(
            "{CUSTOM}{REDIS_DELIMITER}{:020}{REDIS_DELIMITER}{}",
            ts_init,
            UUID4::new()
        );
        self.insert(key, Some(vec![Bytes::from(json_bytes)]))

View on GitHub (pinned to 18893faf8b)

Solutions

Verify the background Redis cache task is running and healthy before insert; restart/reinitialize it if it stopped.
Check logs for a prior panic or error that killed the background task.
Guard shutdown ordering so all cache writes complete before the task/channel is dropped.

Example fix

// before
node.stop();
cache.insert(key, payload)?; // send fails: receiver dropped
// after
cache.insert(key, payload)?;
node.stop();

Defensive patterns

Strategy: try-catch

Try / catch

match cache.insert(key, payload) {
    Ok(()) => {}
    Err(e) if e.to_string().starts_with("Failed to send to channel") => {
        log::error!("redis cache background task is down: {e}");
        // reinitialize cache or buffer the write for retry
    }
    Err(e) => return Err(e.into()),
}

Prevention

Start the background Redis task before any component uses the cache.
Monitor task liveness and reconnect/respawn on failure.
Enforce shutdown ordering: flush cache writes before stopping the task.
Avoid holding cache handles beyond the runtime lifetime (e.g. in Python wrappers).

When it happens

Trigger: Calling cache.insert(...) (directly or via py_insert / add_custom_data) after the background cache task has shut down or the cache was constructed without a live receiver task.

Common situations: Live/trading node shutting down while a callback still tries to persist data; background task crashed earlier; cache object kept alive past its owning runtime's lifetime.

Related errors

AI-assisted analysis of nautechsystems/nautilus_trader@18893faf8b (2026-09-08). Data as JSON: /api/errors/78dce5ccd35643fb. Report an issue: GitHub.