What causes "mutex poisoned" in vectordotdev/vector?

Find and fix the original panic that poisoned the mutex (search logs for the first panic before this one).

mutex poisoned

Error message

mutex poisoned

What it means

The websocket_server sink run loop locks the message_buffering Mutex and unwraps the result with expect("mutex poisoned"). A Mutex is poisoned when another thread panicked while holding the lock; every subsequent lock attempt then returns PoisonError and this expect panics in turn, cascading the original panic.

Solutions

Find and fix the original panic that poisoned the mutex (search logs for the first panic before this one).
Use lock().unwrap_or_else(|e| e.into_inner()) to recover the buffer data instead of propagating poisoning.
Reduce panic sources inside the critical section (avoid unwrap/expect on message formatting there).
Switch to a parking_lot::Mutex, which is not poisoned.

Example fix

// before
let mut buffer = message_buffer.lock().expect("mutex poisoned");
// after
let mut buffer = message_buffer.lock().unwrap_or_else(|e| e.into_inner());

Defensive patterns

Strategy: fallback

Try / catch

let mut buffer = message_buffer.lock().unwrap_or_else(|e| e.into_inner());

Prevention

Avoid panics inside critical sections guarded by std Mutex.
Prefer parking_lot::Mutex in hot sink paths.
Investigate the first logged panic; poisoning is always secondary.

When it happens

Trigger: Any panic occurring in a task/thread that previously held message_buffer while it was locked (e.g. during buffer manipulation), followed by the sink run loop trying to lock the same buffer when should_buffer() is true.

Common situations: An earlier panic in message serialization/formatting inside the buffered section; long-running Vector instances where one poisoned lock cascades shutdown of the whole sink task.

Related errors

AI-assisted analysis of vectordotdev/vector@bdb87aeaa4 (2026-09-16). Data as JSON: /api/errors/081192db04570028. Report an issue: GitHub.

Appendix: source

Thrown at src/sinks/websocket_server/sink.rs:411

                .message_buffering
                .add_replay_message_id_to_event(&mut event);

            let event_byte_size = event.estimated_json_encoded_size_of();

            let mut bytes = BytesMut::new();
            match self.encoder.encode(event, &mut bytes) {
                Ok(()) => {
                    finalizers.update_status(EventStatus::Delivered);

                    let message = if encode_as_binary {
                        Message::binary(bytes)
                    } else {
                        Message::text(String::from_utf8_lossy(&bytes))
                    };
                    let message_len = message.len();

                    if self.message_buffering.should_buffer() {
                        let mut buffer = message_buffer.lock().expect("mutex poisoned");
                        if buffer.len() + 1 >= buffer.capacity() {
                            buffer.pop_front();
                        }
                        buffer.push_back((message_id, message.clone()));
                    }

                    let peers = peers.lock().expect("mutex poisoned");
                    let broadcast_recipients = peers.values();
                    for recp in broadcast_recipients {
                        if let Err(error) = recp.unbounded_send(message.clone()) {
                            emit!(WebSocketListenerSendError {
                                error: Box::new(error)
                            });
                        } else {
                            events_sent.emit(CountByteSize(1, event_byte_size));
                            bytes_sent.emit(ByteSize(message_len));
                        }
                    }

View on GitHub (pinned to bdb87aeaa4)