nautechsystems/nautilus_trader · error

retrying thread panicked

Error message

retrying thread panicked

What it means

Test panic from `retrying.join().expect("retrying thread panicked")` at crates/event_store/src/writer/mod.rs:1429. `std::thread::JoinHandle::join` returns `Err` if the spawned thread panicked; the expect propagates it with the message `retrying thread panicked`. Here the retrying thread runs `writer.submit(entry_draft(13))`, so any panic inside `submit` — or an assertion panic in that closure — surfaces here. The test expects join to succeed and yield `Err(SubmitError::Closed)` because the stall halt latched while the thread waited in the retry loop.

Source

Thrown at crates/event_store/src/writer/mod.rs:1429

        // This submit stalls past the threshold and latches the halt
        let stalled = writer.submit(entry_draft(12)).expect_err("must stall");
        assert!(
            matches!(stalled, SubmitError::HaltSignaled { .. }),
            "was {stalled:?}",
        );

        // A second submitter now waits in the retry loop while the channel stays full
        let writer_for_thread = Arc::clone(&writer);
        let retrying = std::thread::spawn(move || writer_for_thread.submit(entry_draft(13)));
        std::thread::sleep(Duration::from_millis(20));

        // Release the gate: the freed slot must not rescue the retrying submit
        let (lock, cvar) = &*gate;
        *lock.lock() = true;
        cvar.notify_all();

        let result = retrying.join().expect("retrying thread panicked");
        assert!(matches!(result, Err(SubmitError::Closed)), "was {result:?}");

        // The refused entry never commits
        let mut waited = Duration::ZERO;
        while writer.high_watermark() < 2 && waited < Duration::from_secs(2) {
            std::thread::sleep(Duration::from_millis(5));
            waited += Duration::from_millis(5);
        }
        assert_eq!(writer.high_watermark(), 2);
        assert_eq!(
            captured.lock().len(),
            1,
            "halt must not refire for the refused submit",
        );
    }

    #[rstest]
    fn record_snapshot_anchor_signals_halt_when_ack_stalls(

View on GitHub (pinned to 18893faf8b)

Solutions

1. Print the panic payload via `match retrying.join() { Err(e) => panic!("thread panicked: {e:?}"), ... }` to find the root cause inside `submit`.
2. Fix the panic source in the submit retry loop: handle channel-disconnect and poisoned-lock cases by returning `SubmitError::Closed` instead of unwrapping.
3. Verify no other thread poisoned shared state (the gate or captured mutexes) before the retrying thread runs.
4. Ensure the gate release (`lock = true; cvar.notify_all()`) happens before joining, so the retry loop exits deterministically rather than hitting an unexpected state.

Example fix

// before
let result = retrying.join().expect("retrying thread panicked");
// after
let result = retrying
    .join()
    .unwrap_or_else(|p| panic!("retrying thread panicked: {p:?}"));

// Before joining, ensure shared state is not poisoned and the gate is released
*gate.0.lock() = true;
gate.1.notify_all();

match retrying.join() {
    Ok(result) => match result {
        Err(SubmitError::Closed) => {}, // expected
        other => panic!("unexpected submit result: {other:?}"),
    },
    Err(panic_payload) => panic!("retrying thread panicked: {panic_payload:?}"),
}

let result = retrying.join().unwrap_or_else(|p| panic!("worker panicked: {p:?}"));
assert!(matches!(result, Err(SubmitError::Closed)));

When it happens

Trigger: The spawned closure panics: an internal `expect`/`unwrap`/index-out-of-bounds inside `EventStoreWriter::submit`'s retry loop, a poisoned mutex accessed during retry, or `submit` panicking when the channel closes mid-wait.

Common situations: Concurrency regressions where the retry loop unwraps a closed channel or a poisoned `parking_lot`/`std` mutex; deadlocks converted into panics by timeout wrappers; debugging the `retrying_submit_returns_closed_after_stall_halt_latches` test after refactoring the submit retry logic.

Related errors

• in-flight mutex poisoned
• wallet balance mutex poisoned
• instrument update lock poisoned
• rate limiter decision lock poisoned
• Unsupported blockchain {blockchain} for RPC connection

AI-assisted analysis of nautechsystems/nautilus_trader@18893faf8b (2026-09-08). Data as JSON: /api/errors/77313898eca83162. Report an issue: GitHub.