What causes "Couldn't acquire lock on internal logs buffer" in vectordotdev/vector?

Find and fix the original panic that poisoned the BUFFER mutex (it precedes this message in logs)

Couldn't acquire lock on internal logs buffer

Error message

Couldn't acquire lock on internal logs buffer

What it means

`get_early_buffer` locks the process-wide static `BUFFER` mutex holding the early internal-log buffer. If the mutex is poisoned (another thread panicked while holding it), `.lock()` returns `Err` and this `expect` panics with "Couldn't acquire lock on internal logs buffer". It signals unrecoverable corruption of tracing's internal state.

Solutions

Find and fix the original panic that poisoned the BUFFER mutex (it precedes this message in logs)
Restart the process; poisoning is not recoverable at runtime
Avoid panicking in code paths that emit internal tracing events
If reproducible without a root cause, file a Vector issue with a reproducer

Defensive patterns

Strategy: try-catch

Try / catch

// poisoning is not catchable per-lock; isolate the panic and treat as fatal
let guard = std::panic::catch_unwind(|| {
    // code path that emits internal trace events
});
if guard.is_err() { eprintln!("trace buffer poisoned; restarting"); }

Prevention

Never panic in code that emits internal log/trace events
Watch logs for earlier panics (root cause precedes this)
Treat mutex-poisoning panics as fatal and restart
Keep internal tracing paths panic-free in forks

When it happens

Trigger: Any call to `get_early_buffer` (via `reset_early_buffer`, `should_process_tracing_event`, `try_buffer_event`, or `consume_early_buffer`) after a previous panic occurred inside a closure that held the BUFFER lock.

Common situations: A panic inside tracing event processing left BUFFER poisoned; subsequent internal trace calls then cascade-panic. Rare in production; usually follows another bug (e.g. a panicking user transform that emits internal log events).

Understand the failure class

Background: "Invalid state transition" errors: "status must be X, actually Y", "already rejected/charging/uninstalled", "cannot ... while running" — what they mean when a library rejects your call — this error's family across 31 libraries.

Related errors

AI-assisted analysis of vectordotdev/vector@bdb87aeaa4 (2026-09-16). Data as JSON: /api/errors/b2fc8c8ad5b464d4. Report an issue: GitHub.

Appendix: source

Thrown at src/trace.rs:153

        let rate_limited =
            RateLimitedLayer::new(formatter).with_default_limit(internal_log_rate_limit);
        let subscriber = subscriber.with(rate_limited.with_filter(fmt_filter));

        _ = subscriber.try_init();
    }
}

#[cfg(test)]
pub fn reset_early_buffer() -> Option<Vec<LogEvent>> {
    get_early_buffer().replace(Vec::new())
}

/// Gets a  mutable reference to the early buffer.
fn get_early_buffer() -> MutexGuard<'static, Option<Vec<LogEvent>>> {
    BUFFER
        .lock()
        .expect("Couldn't acquire lock on internal logs buffer")
}

/// Determines whether tracing events should be processed (e.g. converted to log
/// events) to avoid unnecessary performance overhead.
///
/// Checks if [`BUFFER`] is set or if a trace sender exists
fn should_process_tracing_event() -> bool {
    get_early_buffer().is_some() || maybe_get_trace_sender().is_some()
}

/// Attempts to buffer an event into the early buffer.
fn try_buffer_event(log: &LogEvent) -> bool {
    if SHOULD_BUFFER.load(Ordering::Acquire)
        && let Some(buffer) = get_early_buffer().as_mut()
    {
        buffer.push(log.clone());
        return true;
    }

View on GitHub (pinned to bdb87aeaa4)