What causes "Failed to lock recoverable execution intent: {e}" in nautechsystems/nautilus_trader?

Retry with backoff - FOR UPDATE waits are transient by nature

nautechsystems/nautilus_trader · error · anyhow::Error

Failed to lock recoverable execution intent: {e}

Error message

Failed to lock recoverable execution intent: {e}

What it means

Thrown when the SELECT status ... FOR UPDATE that serializes the recoverable transition fails in mark_execution_intent_recoverable. That row lock contends with record_execution_status and add_execution_transaction_hash touching the same intent, so the wrapped error is commonly 'lock timeout' or 'deadlock detected', or a dropped connection.

Source

Thrown at crates/adapters/blockchain/src/cache/database.rs:3439

        Ok(())
    }

    /// Releases an intent when no broadcast attempt can have occurred.
    ///
    /// # Errors
    ///
    /// Returns an error if the intent advanced to broadcast or persistence fails.
    pub async fn mark_execution_intent_recoverable(&self, intent_id: i64) -> anyhow::Result<()> {
        let mut transaction = self.pool.begin().await.map_err(|e| {
            anyhow::anyhow!("Failed to start recoverable execution transition: {e}")
        })?;
        let current_status = sqlx::query_scalar::<_, String>(
            "SELECT status FROM execution_intent WHERE id = $1 FOR UPDATE",
        )
        .bind(intent_id)
        .fetch_optional(&mut *transaction)
        .await
        .map_err(|e| anyhow::anyhow!("Failed to lock recoverable execution intent: {e}"))?
        .ok_or_else(|| anyhow::anyhow!("Execution intent {intent_id} was not found"))?;
        anyhow::ensure!(
            matches!(current_status.as_str(), "prepared" | "signed"),
            "Execution intent {intent_id} is {current_status}, not recoverable before broadcast"
        );
        let result = sqlx::query(
            "
            UPDATE execution_intent
            SET status = 'recoverable', active = FALSE, updated_at = NOW()
            WHERE id = $1 AND status IN ('prepared', 'signed')
            ",
        )
        .bind(intent_id)
        .execute(&mut *transaction)
        .await
        .map_err(|e| anyhow::anyhow!("Failed to mark execution intent recoverable: {e}"))?;
        anyhow::ensure!(
            result.rows_affected() == 1,

View on GitHub (pinned to 2114cf6f76)

Solutions

Retry with backoff - FOR UPDATE waits are transient by nature
Take the recoverable transition before enabling watchers on the intent, or drain them first
Raise lock_timeout if the contention window is legitimately long

Defensive patterns

Strategy: retry

Try / catch

for attempt in 0..3 {
    match db.mark_execution_intent_recoverable(intent_id).await {
        Ok(()) => break,
        Err(e) => {
            let msg = e.to_string();
            if msg.contains("lock timeout") || msg.contains("deadlock") {
                tokio::time::sleep(Duration::from_millis(50u64 << attempt)).await;
            } else {
                return Err(e);
            }
        }
    }
}

Prevention

Serialize all writes for one intent through a single task or actor
Take recoverable transitions before watchers begin recording statuses on the intent
Set lock_timeout to a small value with retry rather than waiting indefinitely

When it happens

Trigger: Releasing an intent at the same moment a receipt observation locks it in record_execution_status; lock_timeout exceeded while waiting on FOR UPDATE; connection drop during the lock wait.

Common situations: Signer, broadcaster, and watcher actors operating on one intent concurrently; deadlock-prone access orderings.

Related errors

AI-assisted analysis of nautechsystems/nautilus_trader@2114cf6f76 (2026-08-21). Data as JSON: /api/errors/da87f634c57f9523. Report an issue: GitHub.