nautechsystems/nautilus_trader · error · anyhow::Error

Failed to lock active execution intent {intent_id}: {e}

Error message

Failed to lock active execution intent {intent_id}: {e}

What it means

Wrapped sqlx error from the SELECT status FROM execution_intent WHERE id = $1 AND active FOR UPDATE statement inside add_execution_replacement_hash (database.rs:3795-3799). This both checks that the intent is still active and takes a row lock so the replacement is recorded atomically. The error means the statement itself failed - classically a lock wait timeout or deadlock while another transaction (record_execution_status, another replacement, event marking) holds the same intent row.

Source

Thrown at crates/adapters/blockchain/src/cache/database.rs:3799

    /// Returns an error if the intent is not active, the hash conflicts, or persistence fails.
    pub async fn add_execution_replacement_hash(
        &self,
        intent_id: i64,
        chain_id: u32,
        transaction_hash: &str,
    ) -> anyhow::Result<ExecutionTransactionHashRow> {
        let chain_id_db = i32::try_from(chain_id)
            .with_context(|| format!("Chain ID {chain_id} exceeds PostgreSQL INTEGER"))?;
        let mut transaction = self.pool.begin().await.map_err(|e| {
            anyhow::anyhow!("Failed to start replacement transaction persistence: {e}")
        })?;
        let current_status = sqlx::query_scalar::<_, String>(
            "SELECT status FROM execution_intent WHERE id = $1 AND active FOR UPDATE",
        )
        .bind(intent_id)
        .fetch_optional(&mut *transaction)
        .await
        .map_err(|e| anyhow::anyhow!("Failed to lock active execution intent {intent_id}: {e}"))?
        .ok_or_else(|| anyhow::anyhow!("Active execution intent {intent_id} was not found"))?;
        anyhow::ensure!(
            execution_transition_allowed(&current_status, TransactionStatus::Replaced),
            "Invalid execution transition for intent {intent_id}: {current_status} -> replaced"
        );

        sqlx::query(
            "
            UPDATE execution_transaction_hash
            SET current = FALSE, status = 'replaced', updated_at = NOW()
            WHERE intent_id = $1 AND current
            ",
        )
        .bind(intent_id)
        .execute(&mut *transaction)
        .await
        .map_err(|e| anyhow::anyhow!("Failed to retire replaced execution hash: {e}"))?;

View on GitHub (pinned to 2114cf6f76)

Solutions

1. Read the SQLSTATE from the downcast sqlx::Error: 55P03/57014 lock timeouts and 40P01 deadlocks are retried by re-running the whole call
2. Keep the begin-to-commit window short so FOR UPDATE locks are held briefly
3. Set an explicit per-statement lock_timeout on the cache role so waits fail fast instead of piling up
4. Ensure all intent-mutating paths acquire locks in the same order (intent row first, as this function does) to avoid deadlocks
5. Investigate and kill idle-in-transaction sessions if locks are held by ghost workers

Example fix

// before: any failure on the FOR UPDATE path aborts replacement recording
let row = db.add_execution_replacement_hash(intent_id, chain_id, &hash).await?;

// after: retry lock-wait classes with backoff; safe because the transaction rolled back
let row = retry_on_transient(3, || async {
    db.add_execution_replacement_hash(intent_id, chain_id, &hash).await
}).await?;

async fn retry_on_transient<F, Fut, T: Sized>(max: u32, mut f: F) -> anyhow::Result<T>
where F: FnMut() -> Fut, Fut: std::future::Future<Output = anyhow::Result<T>> {
    for attempt in 1..=max {
        match f().await {
            Ok(v) => return Ok(v),
            Err(e) if attempt < max && is_transient_db_error(&e) =>
                tokio::time::sleep(Duration::from_millis(50 * 2u64.pow(attempt))).await,
            Err(e) => return Err(e),
        }
    }
    unreachable!()
}

fn is_lock_contention(err: &anyhow::Error) -> bool {
    err.downcast_ref::<sqlx::Error>()
        .and_then(|e| e.as_database_error())
        .and_then(|d| d.code())
        .map_or(false, |c| matches!(c.as_ref(), "40P01" | "55P03" | "57014"))
}

match db.add_execution_replacement_hash(intent_id, chain_id, &hash).await {
    Ok(row) => Ok(row),
    Err(e) if is_lock_contention(&e) => retry_with_backoff(e), // rollback makes replay safe
    Err(e) => Err(e),
}

When it happens

Trigger: statement_timeout expiring while blocked on the FOR UPDATE row lock; deadlock detected (40P01) when two sessions lock intent and hash rows in different orders; connection lost during the lock wait; the transaction begun in error 87 being killed by the server.

Common situations: A finality watcher calling record_execution_status on the same intent at the same moment the replacement watcher calls this; idle-in-transaction sessions from a crashed worker holding locks; lock_timeout/statement_timeout set aggressively low on the Postgres side.

Related errors

• Failed to lock recoverable execution intent: {e}
• Failed to lock execution intent {intent_id}: {e}
• Failed to update execution hash {transaction_hash}: {e}
• Execution transaction hash {transaction_hash} was not found
• Failed to update execution intent {intent_id}: {e}

AI-assisted analysis of nautechsystems/nautilus_trader@2114cf6f76 (2026-08-21). Data as JSON: /api/errors/bfe59b045a72d2f7. Report an issue: GitHub.