What causes "Poisoned lock" in firecracker-microvm/firecracker?

Find and fix the FIRST panic that poisoned the lock (check stderr/log for the earlier panic message)

Poisoned lock

Error message

Poisoned lock

What it means

Panic in `Vmm` config export (lib.rs:405): `mmds.lock().expect("Poisoned lock")` while building `MmdsConfig` from the MMDS service handle. The Mutex<VecDeque/Mmds> is poisoned because some thread panicked while holding it — every later `lock()` returns Err(PoisonError) and this serialization path aborts instead of degrading. The original panic is the true root cause; this is the follow-on failure.

Source

Thrown at src/vmm/src/lib.rs:405

                VirtioDeviceType::Vsock => {
                    if let Some(v) = device.as_any().downcast_ref::<Vsock<VsockUnixBackend>>() {
                        vsock = Some(VsockDeviceConfig::from(v));
                    }
                }
                VirtioDeviceType::Rng => {
                    if let Some(e) = device.as_any().downcast_ref::<Entropy>() {
                        entropy = Some(EntropyDeviceConfig::from(e));
                    }
                }
                VirtioDeviceType::Mem => {
                    if let Some(m) = device.as_any().downcast_ref::<VirtioMem>() {
                        memory_hotplug = Some(MemoryHotplugConfig::from(m));
                    }
                }
            });

        let mmds_config = mmds_ref.map(|mmds| {
            let mmds = mmds.lock().expect("Poisoned lock");
            MmdsConfig {
                version: mmds.version(),
                ipv4_address: mmds_ipv4_address,
                network_interfaces: net_with_mmds,
                imds_compat: mmds.imds_compat(),
            }
        });

        // This must match the From<&VmResources> for VmmConfig implementation
        // in resources.rs which is used to retrieve the config before the VM
        // is started.
        VmmConfig {
            balloon,
            drives: block,
            boot_source: self.boot_source_config.clone(),
            cpu_config: None,
            logger: None,
            machine_config: Some(self.machine_config.clone()),

View on GitHub (pinned to 0a745def42)

Solutions

Find and fix the FIRST panic that poisoned the lock (check stderr/log for the earlier panic message)
Make this read path poison-tolerant: `mmds.lock().unwrap_or_else(|e| e.into_inner())` — MMDS state is still readable after a poisoned unlock
Avoid holding the MMDS lock across code that can panic; scope guards tightly around data access only

Example fix

// before
let mmds = mmds.lock().expect("Poisoned lock");

// after
let mmds = mmds.lock().unwrap_or_else(|poisoned| poisoned.into_inner());

Defensive patterns

Strategy: fallback

Validate before calling

// Before serializing config, detect poisoning cheaply
if let Ok(mmds) = mmds.try_lock() {
    // healthy path: build MmdsConfig
} else if mmds.is_poisoned() {
    // recover or report degraded config
}

Try / catch

// Read through the poison: MMDS content is still consistent enough for a config dump
let mmds = mmds.lock().unwrap_or_else(|p| p.into_inner());
let cfg = MmdsConfig { version: mmds.version(), .. };

Prevention

Scope MMDS lock guards to data access only; never across request parsing or unwraps
Install a panic hook that logs which thread poisoned which lock
Surface poison state via a health endpoint so config queries can report degraded instead of aborting

When it happens

Trigger: Any earlier panic in a thread holding the MMDS lock (e.g. a request handler unwinding inside `mmds.lock()`), followed by a config dump / DescribeInstances-style API call that serializes VmmConfig including MmdsConfig.

Common situations: Firecracker API queries the microvm configuration after a background MMDS/HTTP thread crashed; long-lived microvms where a poisoned lock surfaces only on the first later MMDS interaction; tests that intentionally panic inside MMDS handlers and then read config.

Related errors

AI-assisted analysis of firecracker-microvm/firecracker@0a745def42 (2026-08-19). Data as JSON: /api/errors/af18d086bfc5060e. Report an issue: GitHub.