What causes "in-flight mutex poisoned" in nautechsystems/nautilus_trader?

Hunt the original panic: search the log above this message for the FIRST panic in the process; a poisoned mutex is always a secondary symptom of that earlier unwind while the lock was held.

nautechsystems/nautilus_trader · critical

in-flight mutex poisoned

Error message

in-flight mutex poisoned

What it means

release_preparing_slot locks the blockchain execution client's in_flight mutex (std::sync::Mutex<Option<InFlightSlot>>) to clear a Preparing claim left behind by an aborted or failed pre-signature preparation; because no signed transaction exists for a preparing slot, releasing it cannot strand a broadcastable signature. The .expect panics when the mutex is poisoned, which in Rust means an earlier panic unwound through a critical section of this same mutex. Poisoning is permanent for the process lifetime, so the panic repeats on every later attempt to release, claim, or inspect the slot.

Source

Thrown at crates/adapters/blockchain/src/execution/client.rs:159

            "A {} transaction is being prepared; at most one transaction can be in flight",
            purpose.as_str()
        ),
        InFlightSlot::AwaitingFinality(in_flight) => anyhow::anyhow!(
            "Transaction {} (intent {}, {}, nonce {}) is still awaiting finality; at most one transaction can be in flight",
            in_flight.tx_hash,
            in_flight.intent_id,
            in_flight.purpose.as_str(),
            in_flight.nonce
        ),
    }
}

/// Releases a pre-signature slot claim when the slot is still in the preparing state.
///
/// Aborted or failed preparation can leave a claim behind; because no signed transaction
/// exists for a preparing slot, releasing it cannot strand a broadcastable signature.
fn release_preparing_slot(in_flight: &Mutex<Option<InFlightSlot>>) {
    let mut slot = in_flight.lock().expect("in-flight mutex poisoned");
    if matches!(*slot, Some(InFlightSlot::Preparing(_))) {
        *slot = None;
    }
}

#[derive(Debug)]
struct TransactionLimits {
    allowed_token_pairs: HashSet<(Address, Address)>,
    slippage_bps: u32,
    max_slippage_bps: u32,
    max_order_amount: u64,
    deadline_seconds: u64,
    max_quote_age_blocks: u64,
    receipt_timeout_secs: u64,
}

/// Execution client for blockchain interactions including balance tracking and order execution.
#[derive(Debug)]

View on GitHub (pinned to 2114cf6f76)

Solutions

Hunt the original panic: search the log above this message for the FIRST panic in the process; a poisoned mutex is always a secondary symptom of that earlier unwind while the lock was held.
Restart the process: a poisoned std::sync::Mutex never recovers in-process, and this client rebuilds in-flight state from its durable store on startup (intent, current transaction hash, nonce), so restart is the safe recovery for a capital-touching component.
If you maintain this code and can prove the guarded state is consistent, replace .expect with .lock().unwrap_or_else(|e| e.into_inner()); for live trading prefer a restart over blind recovery.
Audit every critical section of this mutex so it contains only infallible moves (assignment, retain, clone) with no RPCs, awaits, or panicking conversions, so future bugs cannot poison it.

Example fix

// before
let mut slot = in_flight.lock().expect("in-flight mutex poisoned");

// after (recover the inner state; prefer a process restart for live trading)
let mut slot = in_flight.lock().unwrap_or_else(|e| e.into_inner());

Defensive patterns

Strategy: fallback

Try / catch

let handle = tokio::spawn(execution_client_task);
if let Err(join_err) = handle.await {
    if join_err.is_panic() {
        // mutex poison: alert, then restart the process so the durable store
        // restores in-flight state on startup
        restart_execution_host();
    }
}

Prevention

Treat any panic in a process holding shared trading state as fatal: alert and restart instead of logging and continuing.
Search for the first panic above the poison message in logs; poison is always a secondary failure.
Never hold a std::sync::Mutex guard across an .await point; scope locks to infallible data moves only.
Supervise the execution client task so a panic triggers a clean restart; startup restores the in-flight slot from the durable store.

When it happens

Trigger: A prepare path aborts or fails after claim_slot and calls release_preparing_slot while the in_flight mutex was already poisoned by an earlier panic in any critical section: claim_slot, ensure_transaction_ready, fill_and_persist, release_slot, or the startup restore.

Common situations: An earlier unwrap/index/expect panic elsewhere in the execution client was logged but a task supervisor kept the process running; a custom fork added a panicking conversion inside a critical section; a version upgrade shipped a panic bug in the signing or preparation pipeline.

Related errors

AI-assisted analysis of nautechsystems/nautilus_trader@2114cf6f76 (2026-08-21). Data as JSON: /api/errors/f6a5bd050cf99c89. Report an issue: GitHub.