nautechsystems/nautilus_trader · error · anyhow::Error

Failed to start execution verification migration: {e}

Error message

Failed to start execution verification migration: {e}

What it means

This error wraps a failure from `self.pool.begin()` while starting the PostgreSQL transaction that performs the execution verification migration (`ensure_execution_verification_schema`). sqlx could not open a transaction on the connection pool, and the underlying sqlx error is embedded in the message as `{e}`. It is a pre-statement failure: none of the schema DDL has run yet.

Source

Thrown at crates/adapters/blockchain/src/cache/database.rs:3897

                && bootstrap.failure_domain_ids.len() >= 3
                && !bootstrap.decisions.is_empty(),
            "Connect verification evidence is incomplete"
        );
        let chain_id = i32::try_from(bootstrap.chain_id)
            .context("Verification chain ID exceeds PostgreSQL INTEGER")?;
        let checkpoint_number = i64::try_from(bootstrap.checkpoint_number)
            .context("Verification checkpoint exceeds PostgreSQL BIGINT")?;
        let checkpoint_timestamp = i64::try_from(bootstrap.checkpoint_timestamp)
            .context("Verification checkpoint timestamp exceeds PostgreSQL BIGINT")?;
        let next_canonical_nonce = i64::try_from(bootstrap.next_canonical_nonce)
            .context("Canonical nonce exceeds PostgreSQL BIGINT")?;
        let observed_canonical_nonce = i64::try_from(bootstrap.observed_canonical_nonce)
            .context("Observed canonical nonce exceeds PostgreSQL BIGINT")?;
        let base_fee = bootstrap
            .checkpoint_base_fee_per_gas
            .map(|value| value.to_string());
        let mut transaction = self.pool.begin().await.map_err(|e| {
            anyhow::anyhow!("Failed to start execution verification migration: {e}")
        })?;

        for statement in [
            "
            CREATE TABLE IF NOT EXISTS execution_verification_nonce (
                chain_id INTEGER NOT NULL REFERENCES chain(chain_id) ON DELETE RESTRICT,
                wallet_address TEXT NOT NULL,
                manifest_version TEXT NOT NULL CHECK (manifest_version <> ''),
                manifest_digest TEXT NOT NULL CHECK (manifest_digest <> ''),
                next_canonical_nonce BIGINT NOT NULL CHECK (next_canonical_nonce >= 0),
                revision BIGINT NOT NULL DEFAULT 0 CHECK (revision >= 0),
                updated_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
                PRIMARY KEY (chain_id, wallet_address)
            )
            ",
            "
            CREATE TABLE IF NOT EXISTS execution_verified_finalized_header (
                chain_id INTEGER NOT NULL,

View on GitHub (pinned to 18893faf8b)

Solutions

  1. Read the wrapped sqlx error in the message: if it says connection refused/timeout, verify PostgreSQL is running and `DATABASE_URL` (host, port, user, password, dbname) is correct.
  2. Increase the sqlx pool limits (`max_connections`, `acquire_timeout`) or reduce concurrent startup work that starves the pool.
  3. Check network reachability and TLS configuration between the adapter and the database (firewall rules, security groups, CA certificates).
  4. If errors are intermittent after idle periods, enable pool idle timeouts/recycling so stale connections are not reused.

Example fix

// before
let pool = PgPool::connect(&database_url).await?; // default pool, small acquire timeout

// after
let pool = PgPoolOptions::new()
    .max_connections(10)
    .acquire_timeout(std::time::Duration::from_secs(30))
    .idle_timeout(std::time::Duration::from_secs(300))
    .connect(&database_url)
    .await?;
Defensive patterns

Strategy: retry

Validate before calling

async fn assert_pool_ready(pool: &sqlx::PgPool) -> Result<(), sqlx::Error> {
    let mut tx = pool.begin().await?; // fail early with the raw driver error
    tx.rollback().await?;
    Ok(())
}

Try / catch

match database.ensure_execution_verification_schema(&bootstrap).await {
    Err(e) if e.to_string().contains("Failed to start execution verification migration") => {
        // inspect wrapped sqlx error; backoff and retry for transient connect/pool-acquire failures
        tokio::time::sleep(Duration::from_secs(5)).await;
        database.ensure_execution_verification_schema(&bootstrap).await?;
    }
    Err(e) => return Err(e),
    Ok(()) => {}
}

Prevention

When it happens

Trigger: Calling `ensure_execution_verification_schema` when the connection pool cannot begin a transaction — pool exhausted (all connections busy), database unreachable, TLS/auth rejected, or the connection dropped mid-handshake.

Common situations: PostgreSQL is down or restarting during application startup; wrong host/port/password in `DATABASE_URL`; connection pool `max_connections` too small for concurrent startup tasks; firewall or idle timeout severing pooled connections; certificate/TLS misconfiguration.

Understand the failure class

Background: Database query failed: Internal Server Error 500s wrapping SQL, Prisma, and connection failures — what to check first — this error's family across 16 libraries.

Related errors

AI-assisted analysis of nautechsystems/nautilus_trader@18893faf8b (2026-09-08). Data as JSON: /api/errors/b4076d3f2d196ab0. Report an issue: GitHub.