{"record":{"id":"77313898eca83162","repo":"nautechsystems/nautilus_trader","slug":"retrying-thread-panicked","errorCode":null,"errorMessage":"retrying thread panicked","messagePattern":"retrying thread panicked","errorType":"panic","errorClass":null,"httpStatus":null,"severity":"error","filePath":"crates/event_store/src/writer/mod.rs","lineNumber":1429,"sourceCode":"\n        // This submit stalls past the threshold and latches the halt\n        let stalled = writer.submit(entry_draft(12)).expect_err(\"must stall\");\n        assert!(\n            matches!(stalled, SubmitError::HaltSignaled { .. }),\n            \"was {stalled:?}\",\n        );\n\n        // A second submitter now waits in the retry loop while the channel stays full\n        let writer_for_thread = Arc::clone(&writer);\n        let retrying = std::thread::spawn(move || writer_for_thread.submit(entry_draft(13)));\n        std::thread::sleep(Duration::from_millis(20));\n\n        // Release the gate: the freed slot must not rescue the retrying submit\n        let (lock, cvar) = &*gate;\n        *lock.lock() = true;\n        cvar.notify_all();\n\n        let result = retrying.join().expect(\"retrying thread panicked\");\n        assert!(matches!(result, Err(SubmitError::Closed)), \"was {result:?}\");\n\n        // The refused entry never commits\n        let mut waited = Duration::ZERO;\n        while writer.high_watermark() < 2 && waited < Duration::from_secs(2) {\n            std::thread::sleep(Duration::from_millis(5));\n            waited += Duration::from_millis(5);\n        }\n        assert_eq!(writer.high_watermark(), 2);\n        assert_eq!(\n            captured.lock().len(),\n            1,\n            \"halt must not refire for the refused submit\",\n        );\n    }\n\n    #[rstest]\n    fn record_snapshot_anchor_signals_halt_when_ack_stalls(","sourceCodeStart":1411,"sourceCodeEnd":1447,"githubUrl":"https://github.com/nautechsystems/nautilus_trader/blob/18893faf8b356be3320add8de2f861b0b647cf06/crates/event_store/src/writer/mod.rs#L1411-L1447","documentation":"Test panic from `retrying.join().expect(\"retrying thread panicked\")` at crates/event_store/src/writer/mod.rs:1429. `std::thread::JoinHandle::join` returns `Err` if the spawned thread panicked; the expect propagates it with the message `retrying thread panicked`. Here the retrying thread runs `writer.submit(entry_draft(13))`, so any panic inside `submit` — or an assertion panic in that closure — surfaces here. The test expects join to succeed and yield `Err(SubmitError::Closed)` because the stall halt latched while the thread waited in the retry loop.","triggerScenarios":"The spawned closure panics: an internal `expect`/`unwrap`/index-out-of-bounds inside `EventStoreWriter::submit`'s retry loop, a poisoned mutex accessed during retry, or `submit` panicking when the channel closes mid-wait.","commonSituations":"Concurrency regressions where the retry loop unwraps a closed channel or a poisoned `parking_lot`/`std` mutex; deadlocks converted into panics by timeout wrappers; debugging the `retrying_submit_returns_closed_after_stall_halt_latches` test after refactoring the submit retry logic.","solutions":["Print the panic payload via `match retrying.join() { Err(e) => panic!(\"thread panicked: {e:?}\"), ... }` to find the root cause inside `submit`.","Fix the panic source in the submit retry loop: handle channel-disconnect and poisoned-lock cases by returning `SubmitError::Closed` instead of unwrapping.","Verify no other thread poisoned shared state (the gate or captured mutexes) before the retrying thread runs.","Ensure the gate release (`lock = true; cvar.notify_all()`) happens before joining, so the retry loop exits deterministically rather than hitting an unexpected state."],"exampleFix":"// before\nlet result = retrying.join().expect(\"retrying thread panicked\");\n// after\nlet result = retrying\n    .join()\n    .unwrap_or_else(|p| panic!(\"retrying thread panicked: {p:?}\"));","handlingStrategy":"try-catch","validationCode":"// Before joining, ensure shared state is not poisoned and the gate is released\n*gate.0.lock() = true;\ngate.1.notify_all();","typeGuard":"match retrying.join() {\n    Ok(result) => match result {\n        Err(SubmitError::Closed) => {}, // expected\n        other => panic!(\"unexpected submit result: {other:?}\"),\n    },\n    Err(panic_payload) => panic!(\"retrying thread panicked: {panic_payload:?}\"),\n}","tryCatchPattern":"let result = retrying.join().unwrap_or_else(|p| panic!(\"worker panicked: {p:?}\"));\nassert!(matches!(result, Err(SubmitError::Closed)));","preventionTips":["Handle channel-disconnect and poisoned-lock paths inside submit with typed errors instead of unwrap/expect.","Release condvar gates before joining worker threads so retry loops exit deterministically.","Propagate panic payloads with their content rather than a static message.","Test concurrency paths under loom/sanitizer-style stress to catch panics before CI."],"tags":["rust","panic","threads","concurrency"],"backgroundTag":"thread-panicked","analyzedSha":"18893faf8b356be3320add8de2f861b0b647cf06","analyzedAt":"2026-09-08T20:49:34.690Z","contentChangedAt":"2026-09-08T20:49:34.690Z","schemaVersion":2},"datasetVersion":"2026-09-14T05:17:10.506Z"}