What causes "fence lock poisoned" in block/buzz?

Find and fix the original panic that poisoned the mutex — inspect the first panic backtrace in the test log, not this expect message.

block/buzz · error

fence lock poisoned

Error message

fence lock poisoned

What it means

ReplicaFence::close clears the retained proof ring under a std Mutex. The panic message "fence lock poisoned" comes from lock().expect(...) — it fires when the mutex is poisoned, i.e. another thread panicked while holding the fence lock, leaving it permanently poisoned for all subsequent lock() calls. The close operation itself is trivial; the real failure is the earlier panic.

Source

Thrown at crates/buzz-db/src/runtime/replica_fence.rs:199

    }
}

/// Shared fence state. `Db` holds an `Arc` of this; the probe task records
/// entries and per-request routing resolves proofs against it.
#[derive(Debug, Default)]
pub struct ReplicaFence {
    inner: Mutex<FenceInner>,
}

impl ReplicaFence {
    /// A new fence, initially closed (empty ring).
    pub fn new() -> Self {
        Self::default()
    }

    /// Close the fence: drop all retained proofs; reads route to the writer.
    pub fn close(&self) {
        let mut inner = self.inner.lock().expect("fence lock poisoned");
        inner.ring.clear();
    }

    /// Record one probe sample. Epoch changes (re-seed) clear the ring and
    /// start a new one under the observed epoch — sound, because an entry
    /// only proves commits on its own timeline and readers must match the
    /// epoch to cite it. A same-epoch token regression is the unsafe case:
    /// see [`RecordOutcome::TokenRegression`].
    pub fn record(
        &self,
        token: i64,
        epoch: Uuid,
        committed_at: Instant,
        fence_wall: DateTime<Utc>,
    ) -> RecordOutcome {
        let mut inner = self.inner.lock().expect("fence lock poisoned");
        if inner.epoch != Some(epoch) {
            inner.ring.clear();

View on GitHub (pinned to dad5a33865)

Solutions

Find and fix the original panic that poisoned the mutex — inspect the first panic backtrace in the test log, not this expect message.
If the fence must survive internal panics, switch to parking_lot::Mutex (no poisoning) or handle lock().unwrap_or_else(|e| e.into_inner()) for lock-free recovery.
Ensure ring-mutation code paths cannot panic (avoid indexing/unwrap inside the locked region).
Re-run the failing test in isolation with --nocapture to capture the root panic.

Example fix

// before
let mut inner = self.inner.lock().expect("fence lock poisoned");
// after
let mut inner = self.inner.lock().unwrap_or_else(|poisoned| poisoned.into_inner());

Defensive patterns

Strategy: try-catch

Try / catch

let mut inner = match self.inner.lock() {
    Ok(g) => g,
    Err(poisoned) => {
        tracing::error!("fence lock poisoned by earlier panic");
        poisoned.into_inner() // recover fence state; fix root panic separately
    }
};

Prevention

Never panic while holding a std Mutex — return Results inside the locked region.
Prefer parking_lot::Mutex for locks whose invariants survive internal errors.
When this message appears, hunt the ORIGINAL panic in the log; it is a symptom, not the cause.

When it happens

Trigger: Calling close(), record(), or any lock-taking method after some other thread panicked while mutating the fence's inner state (e.g. during a probe sample recording or ring mutation).

Common situations: A panic inside a probe worker (run_probe) or scratch-db teardown (drop_scratch_db) poisons the lock; subsequent fence tests like fence_starts_closed_and_opens_on_record then fail with this message, masking the original panic.

Related errors

AI-assisted analysis of block/buzz@dad5a33865 (2026-08-30). Data as JSON: /api/errors/8ab67658d424289c. Report an issue: GitHub.