vectordotdev/vector · error

mutex poisoned

Error message

mutex poisoned

What it means

`intern_alpn_protocols` interns ALPN protocol byte strings into a leaked static table guarded by a `Mutex`; `.expect("mutex poisoned")` panics if another thread panicked while holding the lock, poisoning it. After poisoning, every subsequent TLS context build with ALPN configuration on this process will panic.

Solutions

1. Find and fix the original panic that occurred while the mutex was held — the poisoning is secondary
2. Upgrade/patch so the critical section cannot panic (allocation failures aside, it is panic-free)
3. Restart the process — poisoning is permanent for the process lifetime
4. If resilience matters, replace `Mutex::lock().expect` with `lock().unwrap_or_else(|p| p.into_inner())` since the map tolerates a torn insert

Example fix

// before
let mut interned = INTERNED.lock().expect("mutex poisoned");
// after
let mut interned = INTERNED
    .lock()
    .unwrap_or_else(|poisoned| poisoned.into_inner());

// Cannot pre-validate; ensure no panics occur while the intern lock is held
assert!(!protocols.is_empty() && protocols.len() % 2 == 0, "ALPN wire format is length-prefixed");

// poisoning is permanent; detect at first use and restart the component
match std::panic::catch_unwind(|| settings.apply_context(&mut builder)) {
    Ok(_) => {},
    Err(_) => restart_tls_subsystem(), // 'mutex poisoned' observed
}

When it happens

Trigger: A thread panics while holding the INTERNED mutex lock (e.g. during allocation), then any later call to `intern_alpn_protocols` (via `apply_context_base`, e.g. on certificate reload) panics on the poisoned lock.

Common situations: Panics inside TLS context setup cascading into persistent 'mutex poisoned' failures across cert reloads; long-running services doing repeated `TlsSettings::apply_context` calls.

Related errors

• mutex poisoned
• poisoned lock
• Building HTTP client failed
• Failed to get entry for dir
• HTTPS initialization failed

AI-assisted analysis of vectordotdev/vector@bdb87aeaa4 (2026-09-16). Data as JSON: /api/errors/924877ff5aaecf89. Report an issue: GitHub.

Appendix: source

Thrown at lib/vector-core/src/tls/settings.rs:414

            }
        } else {
            connection.set_verify_hostname(self.verify_hostname);
        }
        Ok(())
    }
}

/// Return a `'static` copy of a server ALPN protocol list, leaking each distinct list at most once.
///
/// `SslContextBuilder::set_alpn_select_callback` requires the protocol list to outlive the context
/// with a `'static` lifetime, so the bytes must be leaked. Interning by content means rebuilding an
/// acceptor with the same ALPN configuration (e.g. on every certificate reload) reuses the existing
/// allocation instead of leaking a fresh copy each time, keeping the leak bounded and one-time.
fn intern_alpn_protocols(protocols: &[u8]) -> &'static [u8] {
    static INTERNED: LazyLock<Mutex<HashMap<Vec<u8>, &'static [u8]>>> =
        LazyLock::new(|| Mutex::new(HashMap::new()));

    let mut interned = INTERNED.lock().expect("mutex poisoned");

    if let Some(existing) = interned.get(protocols).copied() {
        return existing;
    }
    let leaked: &'static [u8] = Box::leak(protocols.to_vec().into_boxed_slice());
    interned.insert(protocols.to_vec(), leaked);
    leaked
}

impl TlsConfig {
    fn load_authorities(&self) -> Result<Vec<X509>> {
        match &self.ca_file {
            None => Ok(vec![]),
            Some(filename) => {
                let (data, filename) = open_read(filename, "certificate")?;
                der_or_pem(
                    data,
                    |der| X509::from_der(&der).map(|x509| vec![x509]),

View on GitHub (pinned to bdb87aeaa4)