nautechsystems/nautilus_trader · error

wallet balance mutex poisoned

Error message

wallet balance mutex poisoned

What it means

refresh_wallet_balances locks the shared wallet_balance mutex (std::sync::Mutex<WalletBalance>) to publish a freshly fetched balance set after generating an account state event. The .expect panics if that mutex is poisoned, meaning an earlier panic unwound while holding it. Because poisoning never clears within a process, every later balance refresh that reaches this write dies here.

Source

Thrown at crates/adapters/blockchain/src/execution/client.rs:399

        // then use token_balance.set_amount_usd(amount_usd) to set the amount_usd value.

        Ok(token_balance)
    }

    /// Refreshes and publishes all native currency and tracked ERC-20 balances.
    async fn refresh_wallet_balances(&mut self) -> anyhow::Result<()> {
        let (wallet_balance, balances) = self.fetch_wallet_balances().await?;
        self.generate_account_state(
            balances,
            vec![],
            true,
            get_atomic_clock_realtime().get_time_ns(),
            None,
        )?;
        *self
            .wallet_balance
            .lock()
            .expect("wallet balance mutex poisoned") = wallet_balance;
        Ok(())
    }

    async fn fetch_wallet_balances(
        &mut self,
    ) -> anyhow::Result<(WalletBalance, Vec<AccountBalance>)> {
        let native_currency_balance = self.fetch_native_currency_balance().await?;
        let token_universe = self
            .wallet_balance
            .lock()
            .expect("wallet balance mutex poisoned")
            .token_universe
            .clone();
        let mut token_addresses = token_universe.iter().copied().collect::<Vec<_>>();
        token_addresses.sort_unstable();

        let mut token_balances = Vec::with_capacity(token_addresses.len());
        for token_address in token_addresses {

View on GitHub (pinned to 2114cf6f76)

Solutions

1. Search the log above this panic for the FIRST panic in the process; the poison message is a secondary symptom of that earlier unwind while wallet_balance was held.
2. Restart the process or rebuild the execution client; std::sync::Mutex poisoning is permanent in-process and balance state is re-fetched from the node on startup.
3. As a maintainer, recover with .lock().unwrap_or_else(|e| e.into_inner()) only if the balance state is provably consistent; prefer restart for production.
4. Keep every wallet_balance critical section to infallible data moves (clone, assignment) and never hold the guard across an await or RPC.

Example fix

// before
*self.wallet_balance.lock().expect("wallet balance mutex poisoned") = wallet_balance;

// after (recover inner state; prefer a process restart for production)
*self.wallet_balance.lock().unwrap_or_else(|e| e.into_inner()) = wallet_balance;

let handle = tokio::spawn(balance_refresh_loop);
if let Err(join_err) = handle.await {
    if join_err.is_panic() {
        // poisoned wallet_balance mutex: restart the process and let the
        // first refresh repopulate balances from RPC
        restart_execution_host();
    }
}

When it happens

Trigger: The periodic or on-demand wallet refresh completes its RPC fetches and generates the account state, then fails to acquire wallet_balance because an earlier panic in fetch_wallet_balances, the post-fill refresh, or query_account poisoned the mutex.

Common situations: A supervisor logged-and-continued after an earlier panic in the balance pipeline; a fork added a panicking parse of RPC balance responses inside a critical section; long-running gateways that survived an earlier unrelated panic on the same mutex.

Related errors

• in-flight mutex poisoned
• OCM state lock poisoned
• {e}
• {e}
• Execution schema version {} is newer than supported version

AI-assisted analysis of nautechsystems/nautilus_trader@2114cf6f76 (2026-08-21). Data as JSON: /api/errors/f17856255464617e. Report an issue: GitHub.