nautechsystems/nautilus_trader · error

Failed to start recoverable execution transition

Error message

Failed to start recoverable execution transition: {e}

What it means

`mark_execution_intent_recoverable` begins by opening a PostgreSQL transaction via `self.pool.begin()`. If acquiring a connection/starting the transaction fails (pool exhausted, database down, TLS/network error), the underlying sqlx error is wrapped into this anyhow error and returned before any intent state is touched.

Solutions

  1. Check database connectivity and credentials, then retry the operation — it is safe to retry since nothing was modified.
  2. Increase the sqlx pool size or add acquire-timeout headroom to handle pool exhaustion.
  3. Inspect the wrapped inner error (`{e}`) to distinguish network vs auth vs server errors.
  4. Add connection health checks / keepalives so stale connections are replaced before `begin()`.

Example fix

// before: no retry around transient pool errors
for id in intents { db.mark_execution_intent_recoverable(id).await?; }
// after: bounded retry on transient failures
for id in intents {
    for _ in 0..3 {
        match db.mark_execution_intent_recoverable(id).await {
            Ok(()) => break,
            Err(e) if is_transient_db_error(&e) => tokio::time::sleep(Duration::from_millis(200)).await,
            Err(e) => return Err(e),
        }
    }
}
Defensive patterns

Strategy: retry

Validate before calling

sqlx::query("SELECT 1").execute(&pool).await.map_err(|e| anyhow!("database unreachable: {e}"))?;

Try / catch

match db.mark_execution_intent_recoverable(id).await {
    Err(e) if is_transient_db_error(&e) => { backoff_and_retry(id, 3).await?; }
    Err(e) => return Err(e),
    Ok(()) => {},
}

Prevention

When it happens

Trigger: Any call to `mark_execution_intent_recoverable(intent_id)` while the database connection pool is exhausted, the database is unreachable, credentials are invalid, or the server rejects `BEGIN`.

Common situations: Connection pool saturation under load; PostgreSQL restart or failover; network partition between the adapter and the database; wrong DSN or expired credentials in the environment.

Understand the failure class

Background: Database query failed: Internal Server Error 500s wrapping SQL, Prisma, and connection failures — what to check first — this error's family across 16 libraries.

Related errors


AI-assisted analysis of nautechsystems/nautilus_trader@18893faf8b (2026-09-08). Data as JSON: /api/errors/6f128a0caefcf50e. Report an issue: GitHub.

Appendix: source

Thrown at crates/adapters/blockchain/src/cache/database.rs:6512

                .context("failed to record the verified replacement transition")?;
            }
        }

        transaction
            .commit()
            .await
            .context("failed to commit verified replacement scan")?;
        Ok(())
    }

    /// Releases an intent when no broadcast attempt can have occurred.
    ///
    /// # Errors
    ///
    /// Returns an error if the intent advanced to broadcast or persistence fails.
    pub async fn mark_execution_intent_recoverable(&self, intent_id: i64) -> anyhow::Result<()> {
        let mut transaction = self.pool.begin().await.map_err(|e| {
            anyhow::anyhow!("Failed to start recoverable execution transition: {e}")
        })?;
        let current_status = sqlx::query_scalar::<_, String>(
            "SELECT status FROM execution_intent WHERE id = $1 FOR UPDATE",
        )
        .bind(intent_id)
        .fetch_optional(&mut *transaction)
        .await
        .map_err(|e| anyhow::anyhow!("Failed to lock recoverable execution intent: {e}"))?
        .ok_or_else(|| anyhow::anyhow!("Execution intent {intent_id} was not found"))?;
        anyhow::ensure!(
            current_status == "prepared",
            "Execution intent {intent_id} is {current_status}, not recoverable before signing"
        );
        let result = sqlx::query(
            "
            UPDATE execution_intent
            SET status = 'recoverable', active = FALSE, updated_at = NOW()
            WHERE id = $1 AND status = 'prepared'

View on GitHub (pinned to 18893faf8b)