nautechsystems/nautilus_trader · error · anyhow::Error

Canonical nonce ledger changed during verification bootstrap

Error message

Canonical nonce ledger changed during verification bootstrap

What it means

During verification bootstrap for an already-initialized signer, the stored next_canonical_nonce in execution_verification_nonce no longer equals the caller-supplied next_canonical_nonce (crates/adapters/blockchain/src/cache/database.rs:4032). The canonical nonce ledger is the durable record of how many finalized transactions this wallet has sent; if it moved between the caller's snapshot and this bootstrap transaction, state changed underneath and the bootstrap refuses to proceed rather than verify against a stale ledger view.

Source

Thrown at crates/adapters/blockchain/src/cache/database.rs:4032

        .bind(bootstrap.wallet_address)
        .fetch_optional(&mut *transaction)
        .await
        .map_err(|e| anyhow::anyhow!("Failed to read canonical nonce ledger: {e}"))?;
        let initialized = current.is_some();

        let revision = if let Some((manifest_version, manifest_digest, stored_nonce, revision)) =
            current
        {
            anyhow::ensure!(
                bootstrap.migration.is_none(),
                "Verification migration was supplied for an initialized signer"
            );
            anyhow::ensure!(
                manifest_version == bootstrap.manifest_version
                    && manifest_digest == bootstrap.manifest_digest,
                "Execution verification manifest identity changed"
            );
            anyhow::ensure!(
                stored_nonce == next_canonical_nonce,
                "Canonical nonce ledger changed during verification bootstrap"
            );

            if observed_canonical_nonce != stored_nonce {
                let expected_observed_nonce = stored_nonce
                    .checked_add(1)
                    .ok_or_else(|| anyhow::anyhow!("Canonical nonce overflow"))?;
                anyhow::ensure!(
                    observed_canonical_nonce == expected_observed_nonce,
                    "Verified finalized transaction count is outside the owned recovery range"
                );
                let recovery = sqlx::query_as::<_, (Option<i64>, String, i64)>(
                    "
                    SELECT
                        intent.nonce,
                        intent.status,
                        COUNT(hash.id) FILTER (

View on GitHub (pinned to 18893faf8b)

Solutions

  1. Re-read the current canonical nonce (SELECT next_canonical_nonce FROM execution_verification_nonce WHERE chain_id=$1 AND wallet_address=$2) and retry bootstrap with the fresh value.
  2. Ensure only one instance performs verification bootstrap per signer at a time (leader election / advisory lock).
  3. Minimize the window between reading the nonce and bootstrapping; do not bootstrap from long-lived cached snapshots.
  4. If a stale snapshot was replayed after recovery, rebuild the snapshot from the current database state.

Example fix

// before
let next_canonical_nonce = cached_snapshot.next_canonical_nonce; // stale
bootstrap(chain_id, wallet, next_canonical_nonce, observed)?;

// after
let next_canonical_nonce = db.current_canonical_nonce(chain_id, wallet).await?; // read fresh, inside the same lock window
bootstrap(chain_id, wallet, next_canonical_nonce, observed).await?;
Defensive patterns

Strategy: validation

Validate before calling

let current = sqlx::query_scalar::<_, i64>(
    "SELECT next_canonical_nonce FROM execution_verification_nonce WHERE chain_id=$1 AND wallet_address=$2",
).bind(chain_id).bind(wallet).fetch_optional(&pool).await?;
if current != Some(next_canonical_nonce) {
    return Err(anyhow::anyhow!("stale canonical nonce snapshot; re-read before bootstrap"));
}

Try / catch

match bootstrap_verification(...).await {
    Err(e) if e.to_string().contains("Canonical nonce ledger changed") => {
        // re-read the fresh nonce and retry once; if it fails again another writer is active
    }
    other => other?,
}

Prevention

When it happens

Trigger: Calling verification bootstrap with a next_canonical_nonce argument captured from an earlier read while another process/instance concurrently finalized transactions and advanced the ledger row for the same chain_id and wallet_address; passing a nonce snapshot from a stale cache or a different environment.

Common situations: Two replicas of the execution engine bootstrapping against the same database; a long gap between snapshotting the nonce and running bootstrap while live trading finalized transactions; replaying an old process state after a crash.

Understand the failure class

Background: "Invalid state transition" errors: "status must be X, actually Y", "already rejected/charging/uninstalled", "cannot ... while running" — what they mean when a library rejects your call — this error's family across 31 libraries.

Related errors

AI-assisted analysis of nautechsystems/nautilus_trader@18893faf8b (2026-09-08). Data as JSON: /api/errors/d58809974cb026f1. Report an issue: GitHub.