What causes "Failed to start execution schema migration: {e}" in nautechsystems/nautilus_trader?

Verify Postgres is reachable with the exact DSN used by the node (psql or SELECT 1) and fix host/port/credentials

nautechsystems/nautilus_trader · error · anyhow::Error

Failed to start execution schema migration: {e}

Error message

Failed to start execution schema migration: {e}

What it means

Thrown when BlockchainCacheDatabase::ensure_execution_transaction_schema fails at pool.begin(), before any migration SQL runs. The wrapped sqlx error means a connection could not be acquired: Postgres unreachable, pool exhausted, or the connection was dropped. The schema is untouched when this fires, so the operation is safely retryable.

Source

Thrown at crates/adapters/blockchain/src/cache/database.rs:3049

        Ok(())
    }

    /// Installs execution schema version 2 without changing existing transaction rows.
    ///
    /// The migration locks the legacy transaction table, refuses unresolved version 1 rows,
    /// installs the versioned intent and hash-history tables, and fences older writers before
    /// releasing the lock. This prevents a mixed-version process from bypassing the new signer
    /// ownership constraints.
    ///
    /// # Errors
    ///
    /// Returns an error if the database operation fails.
    pub async fn ensure_execution_transaction_schema(&self) -> anyhow::Result<()> {
        let mut transaction = self
            .pool
            .begin()
            .await
            .map_err(|e| anyhow::anyhow!("Failed to start execution schema migration: {e}"))?;

        sqlx::query("LOCK TABLE execution_transaction IN ACCESS EXCLUSIVE MODE")
            .execute(&mut *transaction)
            .await
            .map_err(|e| anyhow::anyhow!("Failed to lock legacy execution transactions: {e}"))?;

        for statement in [
            "
            ALTER TABLE execution_transaction
            ADD COLUMN IF NOT EXISTS client_order_id TEXT
            ",
            "
            ALTER TABLE execution_transaction
            ADD COLUMN IF NOT EXISTS wallet_address TEXT
            ",
            "
            ALTER TABLE execution_transaction
            ALTER COLUMN wallet_address DROP NOT NULL

View on GitHub (pinned to 2114cf6f76)

Solutions

Verify Postgres is reachable with the exact DSN used by the node (psql or SELECT 1) and fix host/port/credentials
Run the migration once at startup, before workers take connections, instead of concurrently with trading activity
Size the sqlx pool (max_connections/acquire_timeout) within Postgres max_connections
Add startup retry with backoff so the node waits for database readiness instead of aborting

Defensive patterns

Strategy: retry

Validate before calling

// Gate startup on database readiness before invoking the migration
let ready = sqlx::query("SELECT 1").execute(&pool).await.is_ok();
if !ready {
    // wait / re-probe with backoff instead of calling ensure_execution_transaction_schema
}

Try / catch

fn is_transient(e: &sqlx::Error) -> bool {
    matches!(
        e,
        sqlx::Error::Io(_)
            | sqlx::Error::PoolTimedOut
            | sqlx::Error::ConnectionClosed(_)
            | sqlx::Error::Database(_)
    ) && !matches!(e, sqlx::Error::Database(_))
}
match db.ensure_execution_transaction_schema().await {
    Ok(()) => {}
    Err(e) if e.downcast_ref::<sqlx::Error>().is_some_and(is_transient) => {
        // backoff and retry the idempotent migration
    }
    Err(e) => return Err(e),
}

Prevention

Run ensure_execution_transaction_schema once at startup, before any worker acquires pool connections
Keep the sqlx pool max_connections below Postgres max_connections with headroom for admin sessions
Add a database readiness gate (SELECT 1 loop) to deployment scripts and container startup orders
Alert on pool acquire_timeout metrics so exhaustion is visible before begin() calls fail

When it happens

Trigger: Calling ensure_execution_transaction_schema at startup while Postgres is down or DATABASE_URL/DSN points at the wrong host; a sqlx pool sized above Postgres max_connections; every pooled connection checked out by trading workers so begin() times out on acquire.

Common situations: Service starts before the Postgres container passes readiness checks; misconfigured credentials or port; a connection leak elsewhere exhausts the pool; failover happens mid-startup.

Related errors

AI-assisted analysis of nautechsystems/nautilus_trader@2114cf6f76 (2026-08-21). Data as JSON: /api/errors/a8912bee5da3312e. Report an issue: GitHub.