nautechsystems/nautilus_trader · error

Failed to start blockchain task generation: {e}

Error message

Failed to start blockchain task generation: {e}

What it means

After terminating submissions during shutdown, the client restarts the pending_tasks generation via start_generation to reset the queue for a future reconnect. If the queue fails to start a new generation, the error is wrapped as "Failed to start blockchain task generation". This indicates the task queue could not be recycled and the client cannot safely accept new submissions.

Source

Thrown at crates/adapters/blockchain/src/execution/client.rs:5852

            log::warn!("Blockchain execution client already connected");
            return Ok(());
        }

        log::info!(
            "Connecting to blockchain execution client on chain {}",
            self.chain.name
        );

        if !self.pending_tasks.is_open() || !self.pending_tasks.is_empty() {
            self.pending_tasks.begin_shutdown();
            self.pending_tasks
                .finish_shutdown(Duration::from_secs(5), Duration::from_secs(2))
                .await
                .map_err(|e| anyhow::anyhow!("Failed to terminate blockchain submissions: {e}"))?;
            self.signer = None;
            self.pending_tasks
                .start_generation()
                .map_err(|e| anyhow::anyhow!("Failed to start blockchain task generation: {e}"))?;
        }
        release_preparing_slot(&self.in_flight);

        let setup_guard = TaskGroupGuard::new(&[&self.pending_tasks], || {});

        let payload_keys = PayloadKeySet::load(
            self.config.payload_key_env.as_deref(),
            &self.config.payload_key_retired_env,
            self.config.payload_deployment_id.as_deref(),
        )?
        .map(Arc::new);

        if self.cache.database.is_some() || self.config.postgres_cache_database_config.is_some() {
            let keys = payload_keys.as_deref().ok_or_else(|| {
                anyhow::anyhow!(
                    "Postgres execution requires an active payload key and deployment identity"
                )
            })?;

View on GitHub (pinned to 18893faf8b)

Solutions

1. Read the inner error in {e}; if the queue is already closed or invalid, fully drop/recreate the client instead of reusing it
2. Ensure disconnect/restart is not invoked concurrently — serialize lifecycle calls behind a lock or lifecycle state machine
3. Wait for the finish_shutdown phase to complete (await its future) before attempting restart
4. Retry the reconnect from a fresh client instance if the queue remains wedged

// ensure teardown completed before restart
assert!(client.shutdown_complete(), "cannot restart before finish_shutdown resolves");

match client.restart().await {
    Err(e) if e.to_string().contains("Failed to start blockchain task generation") => {
        log::error!("task generation restart failed: {e}; recreating client");
        drop(client);
        ExecutionClient::new(config).await // fresh queue
    }
    other => other,
}

When it happens

Trigger: Calling the disconnect/restart path where, after finish_shutdown, pending_tasks.start_generation() fails (queue already closed or in an invalid internal state).

Common situations: Reconnect attempted while the queue was not fully torn down; concurrent shutdown/restart calls racing; a bug or prior failure left the queue in a bad state so subsequent generations cannot be opened.

Understand the failure class

Background: "Invalid state transition" errors: "status must be X, actually Y", "already rejected/charging/uninstalled", "cannot ... while running" — what they mean when a library rejects your call — this error's family across 31 libraries.

Related errors

• Active execution intent {intent_id} was not found
• DataActor {} must be registered before calling `clock_mut()`
• DataActor {} must be registered before calling `clock()` - t
• DataActor {} must be registered before calling `clock_ref()`
• DataActor {} must be registered before calling `cache()` - t

AI-assisted analysis of nautechsystems/nautilus_trader@18893faf8b (2026-09-08). Data as JSON: /api/errors/3e2a9836e3e9796e. Report an issue: GitHub.