What causes "Failed to lock canonical nonce ledger: {e}" in nautechsystems/nautilus_trader?

Inspect the wrapped `{e}` error to identify whether it is a connectivity, timeout, permission, or missing-table error.

nautechsystems/nautilus_trader · error · anyhow::Error

Failed to lock canonical nonce ledger: {e}

Error message

Failed to lock canonical nonce ledger: {e}

What it means

Inside the begun transaction, this code runs the locking SELECT against `execution_verification_nonce` for (chain_id, wallet_address); the sqlx query itself failed. The row lock is what serializes nonce assignment, so any query failure aborts the whole assignment transaction. The `{e}` carries the underlying sqlx/database error.

Source

Thrown at crates/adapters/blockchain/src/cache/database.rs:5926

        let mut transaction = self
            .pool
            .begin()
            .await
            .map_err(|e| anyhow::anyhow!("Failed to start verified nonce assignment: {e}"))?;
        let (manifest_version, manifest_digest, next_nonce, revision) =
            sqlx::query_as::<_, (String, String, i64, i64)>(
                "
                SELECT manifest_version, manifest_digest, next_canonical_nonce, revision
                FROM execution_verification_nonce
                WHERE chain_id = $1 AND wallet_address = $2
                FOR UPDATE
                ",
            )
            .bind(chain_id)
            .bind(assignment.wallet_address)
            .fetch_optional(&mut *transaction)
            .await
            .map_err(|e| anyhow::anyhow!("Failed to lock canonical nonce ledger: {e}"))?
            .ok_or_else(|| anyhow::anyhow!("Canonical nonce ledger is not initialized"))?;
        anyhow::ensure!(
            manifest_version == assignment.manifest_version
                && manifest_digest == assignment.manifest_digest,
            "Verified nonce assignment manifest identity changed"
        );
        anyhow::ensure!(
            next_nonce == nonce,
            "Execution nonce {} does not match canonical nonce {next_nonce}",
            assignment.nonce
        );

        let (intent_chain_id, intent_wallet, intent_nonce, intent_status, intent_active) =
            sqlx::query_as::<_, (i32, String, Option<i64>, String, bool)>(
                "
            SELECT chain_id, wallet_address, nonce, status, active
            FROM execution_intent
            WHERE id = $1

View on GitHub (pinned to 18893faf8b)

Solutions

Inspect the wrapped `{e}` error to identify whether it is a connectivity, timeout, permission, or missing-table error.
Run pending migrations so `execution_verification_nonce` exists.
Grant the database role SELECT/UPDATE on the nonce table.
If deadlock/serialization, retry the operation; the transaction is aborted so a fresh attempt is safe.
Check for long-running transactions holding the row lock and reduce lock contention.

Example fix

// before
 retry without backoff on lock failure
// after
 match err { e if is_deadlock(&e) => retry_with_backoff(op, 3), e => return Err(e) }

Defensive patterns

Strategy: retry

Validate before calling

let table_ok = sqlx::query("SELECT 1 FROM information_schema.tables WHERE table_name = 'execution_verification_nonce'").fetch_optional(&pool).await?.is_some();
if !table_ok { return Err(anyhow!("nonce ledger table missing; run migrations")); }

Try / catch

match result {
    Err(e) if is_transient_db_error(&e) => retry_with_backoff(op, 3), // deadlock, timeout, connection reset
    Err(e) => return Err(e.context("nonce ledger lock failed permanently")),
    Ok(v) => Ok(v),
}

Prevention

Apply all migrations as part of deployment before serving traffic.
Grant the app role the needed privileges on execution_verification_nonce.
Keep transactions short to avoid deadlocks and statement timeouts.
Set statement_timeout above your worst-case query time, not below.
Retry deadlock/serialization failures with jittered backoff — they are expected under contention.

When it happens

Trigger: The locking SELECT against `execution_verification_nonce` returns a database error: connection dropped mid-transaction, statement timeout, serialization/deadlock failure, permission denied on the table, or the table/schema missing.

Common situations: Migrations not applied (table absent) in a fresh environment; role lacking SELECT privilege; deadlock with another writer assigning nonces concurrently; statement_timeout set too low; DB failover severing the connection.

Understand the failure class

Background: Database query failed: Internal Server Error 500s wrapping SQL, Prisma, and connection failures — what to check first — this error's family across 16 libraries.

Related errors

AI-assisted analysis of nautechsystems/nautilus_trader@18893faf8b (2026-09-08). Data as JSON: /api/errors/a424806e7392a9e6. Report an issue: GitHub.