What causes "Failed to start execution status transition: {e}" in nautechsystems/nautilus_trader?

Verify database health and retry the observation - record_execution_status is idempotent via its transition keys

nautechsystems/nautilus_trader · error · anyhow::Error

Failed to start execution status transition: {e}

Error message

Failed to start execution status transition: {e}

What it means

Thrown when record_execution_status fails at pool.begin(), before the receipt observation is applied. The wrapped sqlx error means a connection could not be acquired: database unreachable, pool exhausted, or a dropped connection. The observation is lost unless the caller retries it.

Source

Thrown at crates/adapters/blockchain/src/cache/database.rs:3615

            .map(i64::try_from)
            .transpose()
            .with_context(|| {
                format!(
                    "Execution block number {} exceeds PostgreSQL BIGINT",
                    block_number.unwrap_or_default()
                )
            })?;
        let gas_used_db = gas_used.map(i64::try_from).transpose().with_context(|| {
            format!(
                "Execution gas used {} exceeds PostgreSQL BIGINT",
                gas_used.unwrap_or_default()
            )
        })?;
        let mut transaction = self
            .pool
            .begin()
            .await
            .map_err(|e| anyhow::anyhow!("Failed to start execution status transition: {e}"))?;
        let (current_status, fill_emitted, terminal_emitted) =
            sqlx::query_as::<_, (String, bool, bool)>(
            "SELECT status, fill_emitted, terminal_emitted FROM execution_intent WHERE id = $1 FOR UPDATE",
        )
        .bind(intent_id)
        .fetch_optional(&mut *transaction)
        .await
        .map_err(|e| anyhow::anyhow!("Failed to lock execution intent {intent_id}: {e}"))?
        .ok_or_else(|| anyhow::anyhow!("Execution intent {intent_id} was not found"))?;
        anyhow::ensure!(
            execution_transition_allowed(&current_status, status),
            "Invalid execution transition for intent {intent_id}: {current_status} -> {}",
            status.as_str()
        );

        let active = match status {
            TransactionStatus::Finalized | TransactionStatus::Reverted => {
                !fill_emitted && !terminal_emitted

View on GitHub (pinned to 2114cf6f76)

Solutions

Verify database health and retry the observation - record_execution_status is idempotent via its transition keys
Size the pool for peak watcher concurrency
Buffer and reorder observations rather than dropping them on acquisition failure

Defensive patterns

Strategy: retry

Try / catch

match db.record_execution_status(intent_id, &tx_hash, status, block, hash, success, gas, price).await {
    Err(e) if e.downcast_ref::<sqlx::Error>().is_some_and(|se| matches!(se, sqlx::Error::PoolTimedOut | sqlx::Error::Io(_))) => {
        // queue the observation and retry; record_execution_status is idempotent by transition key
    }
    other => other?,
}

Prevention

Queue receipt observations durably so pool exhaustion does not drop them
Size the pool for peak watcher concurrency, not average load
Exploit the idempotent transition keys: re-recording the same observation is safe

When it happens

Trigger: A burst of receipt observations exhausting the pool while broadcasting; database restart mid-observation; acquire_timeout shorter than the contention window.

Common situations: High transaction throughput with many concurrent watchers; undersized pools relative to watcher concurrency.

Related errors

AI-assisted analysis of nautechsystems/nautilus_trader@2114cf6f76 (2026-08-21). Data as JSON: /api/errors/70822522b3c9caf0. Report an issue: GitHub.