What causes "Failed to lock finality nonce ledger: {e}" in nautechsystems/nautilus_trader?

Inspect the wrapped sqlx error (`{e}`) to identify the root cause (lock timeout vs missing table vs permissions vs connection loss)

nautechsystems/nautilus_trader · error · anyhow::Error

Failed to lock finality nonce ledger

Error message

Failed to lock finality nonce ledger: {e}

What it means

Thrown when the `SELECT ... FROM execution_verification_nonce` query inside the verified finality transaction fails at the database level. The sqlx error is wrapped with this context so failures can be attributed to locking/reading the canonical nonce ledger row (selected FOR UPDATE style to serialize nonce advancement). The transaction is rolled back, so no partial state is written.

Solutions

Inspect the wrapped sqlx error (`{e}`) to identify the root cause (lock timeout vs missing table vs permissions vs connection loss)
Retry the finality recording with backoff if it was a transient lock/deadlock conflict
Verify the execution_verification_nonce table exists and migrations ran (missing table would surface here as a query error)
Check database grants for the application role on execution_verification_nonce
Reduce contention by serializing finality recordings per wallet upstream

Example fix

// before
// concurrent transitions for same wallet hit lock timeout
let _ = tokio::join!(record(&db, f1), record(&db, f2));
// after
// serialize per wallet and retry on transient failure
let res = retry_backoff(3, || record(&db, f1)).await;

Defensive patterns

Strategy: retry

Validate before calling

// Rust: verify table exists and grants before recording finality
sqlx::query(
    "SELECT 1 FROM information_schema.tables WHERE table_name = 'execution_verification_nonce'",
).fetch_one(&db.pool).await
    .context("execution_verification_nonce table missing; run migrations")?;

Try / catch

// Distinguish transient lock errors (retry) from permanent ones (fail fast)
match db.record_execution_finality_verified(&finality).await {
    Err(e) if is_transient_db_error(&e) => retry_with_backoff(3, || record(&db, &finality)).await?,
    Err(e) if e.to_string().contains("does not exist") => run_migrations()?,
    other => other?,
}

Prevention

Apply migrations (including execution_verification_nonce) as part of deployment health checks
Grant the app role SELECT/UPDATE on execution_verification_nonce and verify after role changes
Serialize finality recordings per wallet to avoid lock contention on the ledger row
Set explicit lock_timeout/statement_timeout and treat those as retryable transient errors

When it happens

Trigger: Calling record_execution_finality_verified when the ledger SELECT fails: connection dropped mid-transaction, statement timeout, deadlock/lock timeout on the ledger row, permission denied on execution_verification_nonce, or malformed query parameters (e.g. wallet binding type error).

Common situations: Two concurrent finality recordings for the same wallet contend on the ledger row and hit a lock timeout under load; DB failover killed the in-flight transaction; migrations not applied so execution_verification_nonce does not exist; role lacks SELECT grants after a permissions change.

Understand the failure class

Background: Database query failed: Internal Server Error 500s wrapping SQL, Prisma, and connection failures — what to check first — this error's family across 16 libraries.

Related errors

AI-assisted analysis of nautechsystems/nautilus_trader@18893faf8b (2026-09-08). Data as JSON: /api/errors/1bba3767d07831f6. Report an issue: GitHub.

Appendix: source

Thrown at crates/adapters/blockchain/src/cache/database.rs:6946

            .context("Finality gas used exceeds PostgreSQL BIGINT")?;
        let mut transaction =
            self.pool.begin().await.map_err(|e| {
                anyhow::anyhow!("Failed to start verified finality transition: {e}")
            })?;
        let (manifest_version, manifest_digest, stored_nonce, revision) =
            sqlx::query_as::<_, (String, String, i64, i64)>(
                "
                SELECT manifest_version, manifest_digest, next_canonical_nonce, revision
                FROM execution_verification_nonce
                WHERE chain_id = $1 AND wallet_address = $2
                FOR UPDATE
                ",
            )
            .bind(chain_id)
            .bind(finality.wallet_address)
            .fetch_optional(&mut *transaction)
            .await
            .map_err(|e| anyhow::anyhow!("Failed to lock finality nonce ledger: {e}"))?
            .ok_or_else(|| anyhow::anyhow!("Canonical nonce ledger is not initialized"))?;
        anyhow::ensure!(
            manifest_version == finality.manifest_version
                && manifest_digest == finality.manifest_digest,
            "Verified finality manifest identity changed"
        );
        anyhow::ensure!(
            stored_nonce == nonce,
            "Finalized nonce {} does not match canonical nonce {stored_nonce}",
            finality.nonce
        );
        let stored_tip = sqlx::query_as::<_, (i64, String, String, i64, Option<String>, String)>(
            "
            SELECT number, hash, parent_hash, timestamp, base_fee_per_gas, manifest_digest
            FROM execution_verified_finalized_header
            WHERE chain_id = $1 AND wallet_address = $2
            ORDER BY number DESC
            LIMIT 1

View on GitHub (pinned to 18893faf8b)