What causes "Failed to lock execution intent {intent_id}: {e}" in nautechsystems/nautilus_trader?

Retry with backoff on lock timeout or deadlock - both are transient for this access pattern

nautechsystems/nautilus_trader · error · anyhow::Error

Failed to lock execution intent {intent_id}: {e}

Error message

Failed to lock execution intent {intent_id}: {e}

What it means

Thrown when the SELECT status, fill_emitted, terminal_emitted ... FOR UPDATE that locks the intent inside record_execution_status fails. This is the hot receipt path's serialization point, so the wrapped error is typically 'lock timeout' or 'deadlock detected' against other transition writers, or a dropped connection.

Source

Thrown at crates/adapters/blockchain/src/cache/database.rs:3623

        let gas_used_db = gas_used.map(i64::try_from).transpose().with_context(|| {
            format!(
                "Execution gas used {} exceeds PostgreSQL BIGINT",
                gas_used.unwrap_or_default()
            )
        })?;
        let mut transaction = self
            .pool
            .begin()
            .await
            .map_err(|e| anyhow::anyhow!("Failed to start execution status transition: {e}"))?;
        let (current_status, fill_emitted, terminal_emitted) =
            sqlx::query_as::<_, (String, bool, bool)>(
            "SELECT status, fill_emitted, terminal_emitted FROM execution_intent WHERE id = $1 FOR UPDATE",
        )
        .bind(intent_id)
        .fetch_optional(&mut *transaction)
        .await
        .map_err(|e| anyhow::anyhow!("Failed to lock execution intent {intent_id}: {e}"))?
        .ok_or_else(|| anyhow::anyhow!("Execution intent {intent_id} was not found"))?;
        anyhow::ensure!(
            execution_transition_allowed(&current_status, status),
            "Invalid execution transition for intent {intent_id}: {current_status} -> {}",
            status.as_str()
        );

        let active = match status {
            TransactionStatus::Finalized | TransactionStatus::Reverted => {
                !fill_emitted && !terminal_emitted
            }
            TransactionStatus::Recoverable => false,
            _ => true,
        };
        let hash_result = sqlx::query(
            "
            UPDATE execution_transaction_hash
            SET status = $3,

View on GitHub (pinned to 2114cf6f76)

Solutions

Retry with backoff on lock timeout or deadlock - both are transient for this access pattern
Funnel status observations for one intent through a single ordered task (e.g. keyed by intent_id)
Raise lock_timeout modestly if confirmation bursts regularly exceed it

Defensive patterns

Strategy: retry

Try / catch

for attempt in 0..3 {
    match db.record_execution_status(intent_id, &tx_hash, status, block, hash, success, gas, price).await {
        Ok(()) => break,
        Err(e) => {
            let msg = e.to_string();
            if msg.contains("lock timeout") || msg.contains("deadlock detected") {
                tokio::time::sleep(Duration::from_millis(50u64 << attempt)).await;
                continue;
            }
            return Err(e);
        }
    }
}

Prevention

Route all status observations for a given intent through one ordered queue (key by intent_id)
Acquire intent locks in a consistent order when a task touches multiple intents
Retry deadlocks automatically - Postgres aborts only one side and the operation is idempotent

When it happens

Trigger: Two observers recording statuses for the same intent simultaneously; lock_timeout exceeded against a long-running transition; connection drop while waiting for the row lock.

Common situations: Multiple watchers (confirmations, reorg monitors, replacement pollers) racing on one intent; deadlock-prone lock orderings across intents.

Related errors

AI-assisted analysis of nautechsystems/nautilus_trader@2114cf6f76 (2026-08-21). Data as JSON: /api/errors/0e4486cf81afcb16. Report an issue: GitHub.