{"record":{"id":"0e354e162795118e","repo":"nautechsystems/nautilus_trader","slug":"failed-to-lock-execution-verification-state-e","errorCode":null,"errorMessage":"Failed to lock execution verification state: {e}","messagePattern":"Failed to lock execution verification state: (.+?)","errorType":"exception","errorClass":"anyhow::Error","httpStatus":null,"severity":"error","filePath":"crates/adapters/blockchain/src/cache/database.rs","lineNumber":3989,"sourceCode":"                UNIQUE (chain_id, wallet_address, nonce)\n            )\n            \",\n        ] {\n            sqlx::query(statement)\n                .execute(&mut *transaction)\n                .await\n                .map_err(|e| anyhow::anyhow!(\"Failed to install verification schema: {e}\"))?;\n        }\n\n        sqlx::query(\n            \"LOCK TABLE execution_intent, execution_transaction_hash, \\\n             execution_verification_nonce, execution_verified_finalized_header, \\\n             execution_verification_decision, execution_replacement_scan \\\n             IN ACCESS EXCLUSIVE MODE\",\n        )\n        .execute(&mut *transaction)\n        .await\n        .map_err(|e| anyhow::anyhow!(\"Failed to lock execution verification state: {e}\"))?;\n\n        let installed_version = sqlx::query_scalar::<_, i16>(\n            \"SELECT version FROM execution_schema_version \\\n             WHERE component = 'evm_execution_verification'\",\n        )\n        .fetch_optional(&mut *transaction)\n        .await\n        .map_err(|e| anyhow::anyhow!(\"Failed to read verification schema version: {e}\"))?;\n        if let Some(installed_version) = installed_version {\n            anyhow::ensure!(\n                installed_version <= VERIFICATION_SCHEMA_VERSION,\n                \"Execution verification schema version {installed_version} is newer than supported version {VERIFICATION_SCHEMA_VERSION}\"\n            );\n        }\n\n        let current = sqlx::query_as::<_, (String, String, i64, i64)>(\n            \"\n            SELECT manifest_version, manifest_digest, next_canonical_nonce, revision","sourceCodeStart":3971,"sourceCodeEnd":4007,"githubUrl":"https://github.com/nautechsystems/nautilus_trader/blob/18893faf8b356be3320add8de2f861b0b647cf06/crates/adapters/blockchain/src/cache/database.rs#L3971-L4007","documentation":"During EVM execution-verification bootstrap, this code takes an ACCESS EXCLUSIVE lock on the verification tables inside a transaction before inspecting or mutating schema state. The query failed at the database level (deadlock, lock_timeout, connection loss, or a table not yet existing), so the whole bootstrap transaction aborts with this wrapped error.","triggerScenarios":"Running the verification bootstrap while another session holds conflicting locks on execution_intent / execution_transaction_hash / execution_verification_nonce / execution_verified_finalized_header / execution_verification_decision / execution_replacement_scan; lock_timeout or deadlock_timeout firing; connection dropped mid-transaction; one of the tables missing (schema install failed earlier).","commonSituations":"Two service instances bootstrapping the same database concurrently; a long-running migration or manual psql session holding locks; DB connection pool exhaustion or network blip; deploying a build against a database where the verification schema was never installed.","solutions":["Check pg_locks / pg_stat_activity for the session holding conflicting locks and wait for or terminate it","Ensure only one bootstrap/migration process runs at a time (leader election, advisory lock, or single deployment owner)","Confirm the verification schema was installed first (the CREATE TABLE statements just above must have succeeded)","Retry the bootstrap after transient connection/lock-timeout failures","Verify connectivity and pool settings for the Postgres instance"],"exampleFix":"// before: bootstrap runs on every node at startup\nrun_verification_bootstrap(pool).await?;\n// after: only the lock holder bootstraps\nlet mut conn = pool.acquire().await?;\nlet got = sqlx::query(\"SELECT pg_try_advisory_lock($1)\").bind(LOCK_ID).fetch_one(&mut *conn).await?;\nanyhow::ensure!(got.get::<_, bool>(0), \"another node is bootstrapping\");\nrun_verification_bootstrap(pool).await?;","handlingStrategy":"retry","validationCode":"// Before bootstrap, check for blocking locks\nlet blockers = sqlx::query_scalar::<_, i64>(\n  \"SELECT COUNT(*) FROM pg_locks l JOIN pg_class c ON c.oid=l.relation\n   WHERE c.relname IN ('execution_intent','execution_verification_nonce')\n   AND NOT l.granted\"\n).fetch_one(&mut *conn).await?;\nif blockers > 0 { /* defer bootstrap */ }","typeGuard":null,"tryCatchPattern":"match run_bootstrap(&mut tx).await {\n    Err(e) if e.to_string().contains(\"Failed to lock execution verification state\") =>\n        warn!(%e, \"bootstrap lock contention; retrying with backoff\"),\n    Err(e) => return Err(e),\n    Ok(v) => v,\n}","preventionTips":["Run bootstrap from a single leader instance (advisory lock or leader election)","Set a sane lock_timeout and retry with exponential backoff","Avoid running manual DDL or long transactions on verification tables during deployments","Ensure schema install completes before the locking step"],"tags":["database","postgres","locking","concurrency"],"backgroundTag":"database-query-failed","analyzedSha":"18893faf8b356be3320add8de2f861b0b647cf06","analyzedAt":"2026-09-08T20:49:34.690Z","contentChangedAt":"2026-09-08T20:49:34.690Z","schemaVersion":2},"datasetVersion":"2026-09-14T00:17:10.932Z"}