vectordotdev/vector · critical

poisoned locks are dumb

Error message

poisoned locks are dumb

What it means

`get_trace_subscriber_list` locks the static `SUBSCRIBERS` mutex holding waiting oneshot senders for early trace events. A poisoned lock (prior panic while holding it) makes `.lock()` return `Err`, and the `expect("poisoned locks are dumb")` panics. Like the BUFFER panic, it indicates corrupted internal tracing state.

Solutions

1. Locate and fix the original panic that poisoned SUBSCRIBERS (appears earlier in logs)
2. Restart the process — poisoned mutexes are unrecoverable
3. Keep panics out of code that runs while the subscriber list is locked
4. Replace `.expect` with `unwrap_or_else(|p| p.into_inner())` if a fork tolerates recovery

// catch panics around subscriber registration paths
let result = std::panic::catch_unwind(try_register_for_early_events);
if result.is_err() { eprintln!("trace subscriber state poisoned"); }

When it happens

Trigger: Any call to `get_trace_subscriber_list` (via `try_register_for_early_events` or `stop_early_buffering`) after a thread panicked while holding the SUBSCRIBERS lock.

Common situations: Cascading failure after an earlier panic in the trace-subscriber registration path; typically seen in tests or shutdown code where one thread panicked while registering subscribers.

Understand the failure class

Background: "Invalid state transition" errors: "status must be X, actually Y", "already rejected/charging/uninstalled", "cannot ... while running" — what they mean when a library rejects your call — this error's family across 31 libraries.

Related errors

• Couldn't acquire lock on internal logs buffer
• early buffer was already consumed
• span must already exist!
• a record with a next ID must have an event count
• a valid HTTP/1 URI is valid as an HTTP URI

AI-assisted analysis of vectordotdev/vector@bdb87aeaa4 (2026-09-16). Data as JSON: /api/errors/e794a48a038f59dc. Report an issue: GitHub.

Appendix: source

Thrown at src/trace.rs:217

}

/// Attempts to get the trace sender for sending internal log events.
///
/// If the trace sender has not yet been created, `None` is returned.
fn maybe_get_trace_sender() -> Option<&'static broadcast::Sender<LogEvent>> {
    SENDER.get()
}

/// Creates a trace receiver that receives internal log events.
///
/// This will create a trace sender if one did not already exist.
fn get_trace_receiver() -> broadcast::Receiver<LogEvent> {
    get_trace_sender().subscribe()
}

/// Gets a mutable reference to the list of waiting subscribers, if it exists.
fn get_trace_subscriber_list() -> MutexGuard<'static, Option<Vec<oneshot::Sender<Vec<LogEvent>>>>> {
    SUBSCRIBERS.lock().expect("poisoned locks are dumb")
}

/// Attempts to register for early buffered events.
///
/// If early buffering has not yet been stopped, `Some(receiver)` is returned. The given receiver will resolve to a
/// vector of all early buffered events once early buffering has been stopped. Otherwise, if early buffering is already
/// stopped, `None` is returned.
fn try_register_for_early_events() -> Option<oneshot::Receiver<Vec<LogEvent>>> {
    if SHOULD_BUFFER.load(Ordering::Acquire) {
        // We're still in early buffering mode. Attempt to subscribe by adding a oneshot sender
        // to SUBSCRIBERS. If it's already been consumed, then we've gotten beaten out by a
        // caller that is disabling early buffering, so we just go with the flow either way.
        get_trace_subscriber_list().as_mut().map(|subscribers| {
            let (tx, rx) = oneshot::channel();
            subscribers.push(tx);
            rx
        })
    } else {

View on GitHub (pinned to bdb87aeaa4)