{"record":{"id":"bfe59b045a72d2f7","repo":"nautechsystems/nautilus_trader","slug":"failed-to-lock-active-execution-intent-intent-id","errorCode":null,"errorMessage":"Failed to lock active execution intent {intent_id}: {e}","messagePattern":"Failed to lock active execution intent (.+?): (.+?)","errorType":"exception","errorClass":"anyhow::Error","httpStatus":null,"severity":"error","filePath":"crates/adapters/blockchain/src/cache/database.rs","lineNumber":3799,"sourceCode":"    /// Returns an error if the intent is not active, the hash conflicts, or persistence fails.\n    pub async fn add_execution_replacement_hash(\n        &self,\n        intent_id: i64,\n        chain_id: u32,\n        transaction_hash: &str,\n    ) -> anyhow::Result<ExecutionTransactionHashRow> {\n        let chain_id_db = i32::try_from(chain_id)\n            .with_context(|| format!(\"Chain ID {chain_id} exceeds PostgreSQL INTEGER\"))?;\n        let mut transaction = self.pool.begin().await.map_err(|e| {\n            anyhow::anyhow!(\"Failed to start replacement transaction persistence: {e}\")\n        })?;\n        let current_status = sqlx::query_scalar::<_, String>(\n            \"SELECT status FROM execution_intent WHERE id = $1 AND active FOR UPDATE\",\n        )\n        .bind(intent_id)\n        .fetch_optional(&mut *transaction)\n        .await\n        .map_err(|e| anyhow::anyhow!(\"Failed to lock active execution intent {intent_id}: {e}\"))?\n        .ok_or_else(|| anyhow::anyhow!(\"Active execution intent {intent_id} was not found\"))?;\n        anyhow::ensure!(\n            execution_transition_allowed(&current_status, TransactionStatus::Replaced),\n            \"Invalid execution transition for intent {intent_id}: {current_status} -> replaced\"\n        );\n\n        sqlx::query(\n            \"\n            UPDATE execution_transaction_hash\n            SET current = FALSE, status = 'replaced', updated_at = NOW()\n            WHERE intent_id = $1 AND current\n            \",\n        )\n        .bind(intent_id)\n        .execute(&mut *transaction)\n        .await\n        .map_err(|e| anyhow::anyhow!(\"Failed to retire replaced execution hash: {e}\"))?;\n","sourceCodeStart":3781,"sourceCodeEnd":3817,"githubUrl":"https://github.com/nautechsystems/nautilus_trader/blob/2114cf6f761429e0adb5ca9596fcd7b895b16011/crates/adapters/blockchain/src/cache/database.rs#L3781-L3817","documentation":"Wrapped sqlx error from the SELECT status FROM execution_intent WHERE id = $1 AND active FOR UPDATE statement inside add_execution_replacement_hash (database.rs:3795-3799). This both checks that the intent is still active and takes a row lock so the replacement is recorded atomically. The error means the statement itself failed - classically a lock wait timeout or deadlock while another transaction (record_execution_status, another replacement, event marking) holds the same intent row.","triggerScenarios":"statement_timeout expiring while blocked on the FOR UPDATE row lock; deadlock detected (40P01) when two sessions lock intent and hash rows in different orders; connection lost during the lock wait; the transaction begun in error 87 being killed by the server.","commonSituations":"A finality watcher calling record_execution_status on the same intent at the same moment the replacement watcher calls this; idle-in-transaction sessions from a crashed worker holding locks; lock_timeout/statement_timeout set aggressively low on the Postgres side.","solutions":["Read the SQLSTATE from the downcast sqlx::Error: 55P03/57014 lock timeouts and 40P01 deadlocks are retried by re-running the whole call","Keep the begin-to-commit window short so FOR UPDATE locks are held briefly","Set an explicit per-statement lock_timeout on the cache role so waits fail fast instead of piling up","Ensure all intent-mutating paths acquire locks in the same order (intent row first, as this function does) to avoid deadlocks","Investigate and kill idle-in-transaction sessions if locks are held by ghost workers"],"exampleFix":"// before: any failure on the FOR UPDATE path aborts replacement recording\nlet row = db.add_execution_replacement_hash(intent_id, chain_id, &hash).await?;\n\n// after: retry lock-wait classes with backoff; safe because the transaction rolled back\nlet row = retry_on_transient(3, || async {\n    db.add_execution_replacement_hash(intent_id, chain_id, &hash).await\n}).await?;\n\nasync fn retry_on_transient<F, Fut, T: Sized>(max: u32, mut f: F) -> anyhow::Result<T>\nwhere F: FnMut() -> Fut, Fut: std::future::Future<Output = anyhow::Result<T>> {\n    for attempt in 1..=max {\n        match f().await {\n            Ok(v) => return Ok(v),\n            Err(e) if attempt < max && is_transient_db_error(&e) =>\n                tokio::time::sleep(Duration::from_millis(50 * 2u64.pow(attempt))).await,\n            Err(e) => return Err(e),\n        }\n    }\n    unreachable!()\n}","handlingStrategy":"retry","validationCode":null,"typeGuard":"fn is_lock_contention(err: &anyhow::Error) -> bool {\n    err.downcast_ref::<sqlx::Error>()\n        .and_then(|e| e.as_database_error())\n        .and_then(|d| d.code())\n        .map_or(false, |c| matches!(c.as_ref(), \"40P01\" | \"55P03\" | \"57014\"))\n}","tryCatchPattern":"match db.add_execution_replacement_hash(intent_id, chain_id, &hash).await {\n    Ok(row) => Ok(row),\n    Err(e) if is_lock_contention(&e) => retry_with_backoff(e), // rollback makes replay safe\n    Err(e) => Err(e),\n}","preventionTips":["Keep the FOR UPDATE window short: no awaits on external I/O between begin and commit","Always lock the intent row first (as this function does) so all writers order locks identically and deadlocks cannot form","Set an explicit lock_timeout/statement_timeout on the cache role so waits fail fast and retriable","Reap idle-in-transaction sessions left by crashed workers"],"tags":["rust","sqlx","postgres","blockchain","row-lock","deadlock","pessimistic-locking"],"backgroundTag":"database-lock-timeout","analyzedSha":"2114cf6f761429e0adb5ca9596fcd7b895b16011","analyzedAt":"2026-08-21T11:28:30.864Z","schemaVersion":2},"datasetVersion":"2026-08-22T04:17:13.399Z"}