{"record":{"id":"d58809974cb026f1","repo":"nautechsystems/nautilus_trader","slug":"canonical-nonce-ledger-changed-during-verification","errorCode":null,"errorMessage":"Canonical nonce ledger changed during verification bootstrap","messagePattern":"Canonical nonce ledger changed during verification bootstrap","errorType":"validation","errorClass":"anyhow::Error","httpStatus":null,"severity":"error","filePath":"crates/adapters/blockchain/src/cache/database.rs","lineNumber":4032,"sourceCode":"        .bind(bootstrap.wallet_address)\n        .fetch_optional(&mut *transaction)\n        .await\n        .map_err(|e| anyhow::anyhow!(\"Failed to read canonical nonce ledger: {e}\"))?;\n        let initialized = current.is_some();\n\n        let revision = if let Some((manifest_version, manifest_digest, stored_nonce, revision)) =\n            current\n        {\n            anyhow::ensure!(\n                bootstrap.migration.is_none(),\n                \"Verification migration was supplied for an initialized signer\"\n            );\n            anyhow::ensure!(\n                manifest_version == bootstrap.manifest_version\n                    && manifest_digest == bootstrap.manifest_digest,\n                \"Execution verification manifest identity changed\"\n            );\n            anyhow::ensure!(\n                stored_nonce == next_canonical_nonce,\n                \"Canonical nonce ledger changed during verification bootstrap\"\n            );\n\n            if observed_canonical_nonce != stored_nonce {\n                let expected_observed_nonce = stored_nonce\n                    .checked_add(1)\n                    .ok_or_else(|| anyhow::anyhow!(\"Canonical nonce overflow\"))?;\n                anyhow::ensure!(\n                    observed_canonical_nonce == expected_observed_nonce,\n                    \"Verified finalized transaction count is outside the owned recovery range\"\n                );\n                let recovery = sqlx::query_as::<_, (Option<i64>, String, i64)>(\n                    \"\n                    SELECT\n                        intent.nonce,\n                        intent.status,\n                        COUNT(hash.id) FILTER (","sourceCodeStart":4014,"sourceCodeEnd":4050,"githubUrl":"https://github.com/nautechsystems/nautilus_trader/blob/18893faf8b356be3320add8de2f861b0b647cf06/crates/adapters/blockchain/src/cache/database.rs#L4014-L4050","documentation":"During verification bootstrap for an already-initialized signer, the stored next_canonical_nonce in execution_verification_nonce no longer equals the caller-supplied next_canonical_nonce (crates/adapters/blockchain/src/cache/database.rs:4032). The canonical nonce ledger is the durable record of how many finalized transactions this wallet has sent; if it moved between the caller's snapshot and this bootstrap transaction, state changed underneath and the bootstrap refuses to proceed rather than verify against a stale ledger view.","triggerScenarios":"Calling verification bootstrap with a next_canonical_nonce argument captured from an earlier read while another process/instance concurrently finalized transactions and advanced the ledger row for the same chain_id and wallet_address; passing a nonce snapshot from a stale cache or a different environment.","commonSituations":"Two replicas of the execution engine bootstrapping against the same database; a long gap between snapshotting the nonce and running bootstrap while live trading finalized transactions; replaying an old process state after a crash.","solutions":["Re-read the current canonical nonce (SELECT next_canonical_nonce FROM execution_verification_nonce WHERE chain_id=$1 AND wallet_address=$2) and retry bootstrap with the fresh value.","Ensure only one instance performs verification bootstrap per signer at a time (leader election / advisory lock).","Minimize the window between reading the nonce and bootstrapping; do not bootstrap from long-lived cached snapshots.","If a stale snapshot was replayed after recovery, rebuild the snapshot from the current database state."],"exampleFix":"// before\nlet next_canonical_nonce = cached_snapshot.next_canonical_nonce; // stale\nbootstrap(chain_id, wallet, next_canonical_nonce, observed)?;\n\n// after\nlet next_canonical_nonce = db.current_canonical_nonce(chain_id, wallet).await?; // read fresh, inside the same lock window\nbootstrap(chain_id, wallet, next_canonical_nonce, observed).await?;","handlingStrategy":"validation","validationCode":"let current = sqlx::query_scalar::<_, i64>(\n    \"SELECT next_canonical_nonce FROM execution_verification_nonce WHERE chain_id=$1 AND wallet_address=$2\",\n).bind(chain_id).bind(wallet).fetch_optional(&pool).await?;\nif current != Some(next_canonical_nonce) {\n    return Err(anyhow::anyhow!(\"stale canonical nonce snapshot; re-read before bootstrap\"));\n}","typeGuard":null,"tryCatchPattern":"match bootstrap_verification(...).await {\n    Err(e) if e.to_string().contains(\"Canonical nonce ledger changed\") => {\n        // re-read the fresh nonce and retry once; if it fails again another writer is active\n    }\n    other => other?,\n}","preventionTips":["Read the canonical nonce immediately before bootstrap, never from a long-lived cache","Ensure a single writer/leader performs bootstrap per (chain_id, wallet)","Use advisory locks or leader election to serialize bootstrap across replicas"],"tags":["database","concurrency","blockchain","stale-state"],"backgroundTag":"invalid-state-transition","analyzedSha":"18893faf8b356be3320add8de2f861b0b647cf06","analyzedAt":"2026-09-08T20:49:34.690Z","contentChangedAt":"2026-09-08T20:49:34.690Z","schemaVersion":2},"datasetVersion":"2026-09-14T05:17:10.506Z"}