{"record":{"id":"f17856255464617e","repo":"nautechsystems/nautilus_trader","slug":"wallet-balance-mutex-poisoned","errorCode":null,"errorMessage":"wallet balance mutex poisoned","messagePattern":"wallet balance mutex poisoned","errorType":"panic","errorClass":null,"httpStatus":null,"severity":"error","filePath":"crates/adapters/blockchain/src/execution/client.rs","lineNumber":399,"sourceCode":"        // then use token_balance.set_amount_usd(amount_usd) to set the amount_usd value.\n\n        Ok(token_balance)\n    }\n\n    /// Refreshes and publishes all native currency and tracked ERC-20 balances.\n    async fn refresh_wallet_balances(&mut self) -> anyhow::Result<()> {\n        let (wallet_balance, balances) = self.fetch_wallet_balances().await?;\n        self.generate_account_state(\n            balances,\n            vec![],\n            true,\n            get_atomic_clock_realtime().get_time_ns(),\n            None,\n        )?;\n        *self\n            .wallet_balance\n            .lock()\n            .expect(\"wallet balance mutex poisoned\") = wallet_balance;\n        Ok(())\n    }\n\n    async fn fetch_wallet_balances(\n        &mut self,\n    ) -> anyhow::Result<(WalletBalance, Vec<AccountBalance>)> {\n        let native_currency_balance = self.fetch_native_currency_balance().await?;\n        let token_universe = self\n            .wallet_balance\n            .lock()\n            .expect(\"wallet balance mutex poisoned\")\n            .token_universe\n            .clone();\n        let mut token_addresses = token_universe.iter().copied().collect::<Vec<_>>();\n        token_addresses.sort_unstable();\n\n        let mut token_balances = Vec::with_capacity(token_addresses.len());\n        for token_address in token_addresses {","sourceCodeStart":381,"sourceCodeEnd":417,"githubUrl":"https://github.com/nautechsystems/nautilus_trader/blob/2114cf6f761429e0adb5ca9596fcd7b895b16011/crates/adapters/blockchain/src/execution/client.rs#L381-L417","documentation":"refresh_wallet_balances locks the shared wallet_balance mutex (std::sync::Mutex<WalletBalance>) to publish a freshly fetched balance set after generating an account state event. The .expect panics if that mutex is poisoned, meaning an earlier panic unwound while holding it. Because poisoning never clears within a process, every later balance refresh that reaches this write dies here.","triggerScenarios":"The periodic or on-demand wallet refresh completes its RPC fetches and generates the account state, then fails to acquire wallet_balance because an earlier panic in fetch_wallet_balances, the post-fill refresh, or query_account poisoned the mutex.","commonSituations":"A supervisor logged-and-continued after an earlier panic in the balance pipeline; a fork added a panicking parse of RPC balance responses inside a critical section; long-running gateways that survived an earlier unrelated panic on the same mutex.","solutions":["Search the log above this panic for the FIRST panic in the process; the poison message is a secondary symptom of that earlier unwind while wallet_balance was held.","Restart the process or rebuild the execution client; std::sync::Mutex poisoning is permanent in-process and balance state is re-fetched from the node on startup.","As a maintainer, recover with .lock().unwrap_or_else(|e| e.into_inner()) only if the balance state is provably consistent; prefer restart for production.","Keep every wallet_balance critical section to infallible data moves (clone, assignment) and never hold the guard across an await or RPC."],"exampleFix":"// before\n*self.wallet_balance.lock().expect(\"wallet balance mutex poisoned\") = wallet_balance;\n\n// after (recover inner state; prefer a process restart for production)\n*self.wallet_balance.lock().unwrap_or_else(|e| e.into_inner()) = wallet_balance;","handlingStrategy":"fallback","validationCode":null,"typeGuard":null,"tryCatchPattern":"let handle = tokio::spawn(balance_refresh_loop);\nif let Err(join_err) = handle.await {\n    if join_err.is_panic() {\n        // poisoned wallet_balance mutex: restart the process and let the\n        // first refresh repopulate balances from RPC\n        restart_execution_host();\n    }\n}","preventionTips":["Alert on the first panic anywhere in the client process, not just on task failure.","Keep wallet_balance lock scopes to clone/assignment; never across awaits or RPCs.","Supervise and restart the whole process on panic so balance state is rebuilt from the node.","In forks, avoid unwrap/indexing inside wallet_balance critical sections."],"tags":["rust","mutex","panic","blockchain","wallet-balance"],"backgroundTag":"mutex-poisoned","analyzedSha":"2114cf6f761429e0adb5ca9596fcd7b895b16011","analyzedAt":"2026-08-21T11:28:30.864Z","schemaVersion":2},"datasetVersion":"2026-08-22T04:17:13.399Z"}