What causes "Failed to lock finality nonce ledger: {e}" in nautechsystems/nautilus_trader?

Inspect the wrapped sqlx error (`{e}`) to identify the root cause (lock timeout vs missing table vs permissions vs connection loss)

nautechsystems/nautilus_trader · error · anyhow::Error

Failed to lock finality nonce ledger: {e}

Error message

Failed to lock finality nonce ledger: {e}

What it means

Thrown when the `SELECT ... FROM execution_verification_nonce` query inside the verified finality transaction fails at the database level. The sqlx error is wrapped with this context so failures can be attributed to locking/reading the canonical nonce ledger row (selected FOR UPDATE style to serialize nonce advancement). The transaction is rolled back, so no partial state is written.

Source

Thrown at crates/adapters/blockchain/src/cache/database.rs:6946

            .context("Finality gas used exceeds PostgreSQL BIGINT")?;
        let mut transaction =
            self.pool.begin().await.map_err(|e| {
                anyhow::anyhow!("Failed to start verified finality transition: {e}")
            })?;
        let (manifest_version, manifest_digest, stored_nonce, revision) =
            sqlx::query_as::<_, (String, String, i64, i64)>(
                "
                SELECT manifest_version, manifest_digest, next_canonical_nonce, revision
                FROM execution_verification_nonce
                WHERE chain_id = $1 AND wallet_address = $2
                FOR UPDATE
                ",
            )
            .bind(chain_id)
            .bind(finality.wallet_address)
            .fetch_optional(&mut *transaction)
            .await
            .map_err(|e| anyhow::anyhow!("Failed to lock finality nonce ledger: {e}"))?
            .ok_or_else(|| anyhow::anyhow!("Canonical nonce ledger is not initialized"))?;
        anyhow::ensure!(
            manifest_version == finality.manifest_version
                && manifest_digest == finality.manifest_digest,
            "Verified finality manifest identity changed"
        );
        anyhow::ensure!(
            stored_nonce == nonce,
            "Finalized nonce {} does not match canonical nonce {stored_nonce}",
            finality.nonce
        );
        let stored_tip = sqlx::query_as::<_, (i64, String, String, i64, Option<String>, String)>(
            "
            SELECT number, hash, parent_hash, timestamp, base_fee_per_gas, manifest_digest
            FROM execution_verified_finalized_header
            WHERE chain_id = $1 AND wallet_address = $2
            ORDER BY number DESC
            LIMIT 1

View on GitHub (pinned to 18893faf8b)

Solutions

Inspect the wrapped sqlx error (`{e}`) to identify the root cause (lock timeout vs missing table vs permissions vs connection loss)
Retry the finality recording with backoff if it was a transient lock/deadlock conflict
Verify the execution_verification_nonce table exists and migrations ran (missing table would surface here as a query error)
Check database grants for the application role on execution_verification_nonce
Reduce contention by serializing finality recordings per wallet upstream

Example fix

// before
// concurrent transitions for same wallet hit lock timeout
let _ = tokio::join!(record(&db, f1), record(&db, f2));
// after
// serialize per wallet and retry on transient failure
let res = retry_backoff(3, || record(&db, f1)).await;

Defensive patterns

Strategy: retry

Validate before calling

// Rust: verify table exists and grants before recording finality
sqlx::query(
    "SELECT 1 FROM information_schema.tables WHERE table_name = 'execution_verification_nonce'",
).fetch_one(&db.pool).await
    .context("execution_verification_nonce table missing; run migrations")?;

Try / catch

// Distinguish transient lock errors (retry) from permanent ones (fail fast)
match db.record_execution_finality_verified(&finality).await {
    Err(e) if is_transient_db_error(&e) => retry_with_backoff(3, || record(&db, &finality)).await?,
    Err(e) if e.to_string().contains("does not exist") => run_migrations()?,
    other => other?,
}

Prevention

Apply migrations (including execution_verification_nonce) as part of deployment health checks
Grant the app role SELECT/UPDATE on execution_verification_nonce and verify after role changes
Serialize finality recordings per wallet to avoid lock contention on the ledger row
Set explicit lock_timeout/statement_timeout and treat those as retryable transient errors

When it happens

Trigger: Calling record_execution_finality_verified when the ledger SELECT fails: connection dropped mid-transaction, statement timeout, deadlock/lock timeout on the ledger row, permission denied on execution_verification_nonce, or malformed query parameters (e.g. wallet binding type error).

Common situations: Two concurrent finality recordings for the same wallet contend on the ledger row and hit a lock timeout under load; DB failover killed the in-flight transaction; migrations not applied so execution_verification_nonce does not exist; role lacks SELECT grants after a permissions change.

Understand the failure class

Background: Database query failed: Internal Server Error 500s wrapping SQL, Prisma, and connection failures — what to check first — this error's family across 16 libraries.

Related errors

AI-assisted analysis of nautechsystems/nautilus_trader@18893faf8b (2026-09-08). Data as JSON: /api/errors/1bba3767d07831f6. Report an issue: GitHub.