nautechsystems/nautilus_trader · error · anyhow::Error

Failed to lock execution verification state: {e}

Error message

Failed to lock execution verification state: {e}

What it means

During EVM execution-verification bootstrap, this code takes an ACCESS EXCLUSIVE lock on the verification tables inside a transaction before inspecting or mutating schema state. The query failed at the database level (deadlock, lock_timeout, connection loss, or a table not yet existing), so the whole bootstrap transaction aborts with this wrapped error.

Source

Thrown at crates/adapters/blockchain/src/cache/database.rs:3989

                UNIQUE (chain_id, wallet_address, nonce)
            )
            ",
        ] {
            sqlx::query(statement)
                .execute(&mut *transaction)
                .await
                .map_err(|e| anyhow::anyhow!("Failed to install verification schema: {e}"))?;
        }

        sqlx::query(
            "LOCK TABLE execution_intent, execution_transaction_hash, \
             execution_verification_nonce, execution_verified_finalized_header, \
             execution_verification_decision, execution_replacement_scan \
             IN ACCESS EXCLUSIVE MODE",
        )
        .execute(&mut *transaction)
        .await
        .map_err(|e| anyhow::anyhow!("Failed to lock execution verification state: {e}"))?;

        let installed_version = sqlx::query_scalar::<_, i16>(
            "SELECT version FROM execution_schema_version \
             WHERE component = 'evm_execution_verification'",
        )
        .fetch_optional(&mut *transaction)
        .await
        .map_err(|e| anyhow::anyhow!("Failed to read verification schema version: {e}"))?;
        if let Some(installed_version) = installed_version {
            anyhow::ensure!(
                installed_version <= VERIFICATION_SCHEMA_VERSION,
                "Execution verification schema version {installed_version} is newer than supported version {VERIFICATION_SCHEMA_VERSION}"
            );
        }

        let current = sqlx::query_as::<_, (String, String, i64, i64)>(
            "
            SELECT manifest_version, manifest_digest, next_canonical_nonce, revision

View on GitHub (pinned to 18893faf8b)

Solutions

  1. Check pg_locks / pg_stat_activity for the session holding conflicting locks and wait for or terminate it
  2. Ensure only one bootstrap/migration process runs at a time (leader election, advisory lock, or single deployment owner)
  3. Confirm the verification schema was installed first (the CREATE TABLE statements just above must have succeeded)
  4. Retry the bootstrap after transient connection/lock-timeout failures
  5. Verify connectivity and pool settings for the Postgres instance

Example fix

// before: bootstrap runs on every node at startup
run_verification_bootstrap(pool).await?;
// after: only the lock holder bootstraps
let mut conn = pool.acquire().await?;
let got = sqlx::query("SELECT pg_try_advisory_lock($1)").bind(LOCK_ID).fetch_one(&mut *conn).await?;
anyhow::ensure!(got.get::<_, bool>(0), "another node is bootstrapping");
run_verification_bootstrap(pool).await?;
Defensive patterns

Strategy: retry

Validate before calling

// Before bootstrap, check for blocking locks
let blockers = sqlx::query_scalar::<_, i64>(
  "SELECT COUNT(*) FROM pg_locks l JOIN pg_class c ON c.oid=l.relation
   WHERE c.relname IN ('execution_intent','execution_verification_nonce')
   AND NOT l.granted"
).fetch_one(&mut *conn).await?;
if blockers > 0 { /* defer bootstrap */ }

Try / catch

match run_bootstrap(&mut tx).await {
    Err(e) if e.to_string().contains("Failed to lock execution verification state") =>
        warn!(%e, "bootstrap lock contention; retrying with backoff"),
    Err(e) => return Err(e),
    Ok(v) => v,
}

Prevention

When it happens

Trigger: Running the verification bootstrap while another session holds conflicting locks on execution_intent / execution_transaction_hash / execution_verification_nonce / execution_verified_finalized_header / execution_verification_decision / execution_replacement_scan; lock_timeout or deadlock_timeout firing; connection dropped mid-transaction; one of the tables missing (schema install failed earlier).

Common situations: Two service instances bootstrapping the same database concurrently; a long-running migration or manual psql session holding locks; DB connection pool exhaustion or network blip; deploying a build against a database where the verification schema was never installed.

Understand the failure class

Background: Database query failed: Internal Server Error 500s wrapping SQL, Prisma, and connection failures — what to check first — this error's family across 16 libraries.

Related errors

AI-assisted analysis of nautechsystems/nautilus_trader@18893faf8b (2026-09-08). Data as JSON: /api/errors/0e354e162795118e. Report an issue: GitHub.