{"record":{"id":"746689da4610eed4","repo":"firecracker-microvm/firecracker","slug":"error-unlocking-vmm","errorCode":null,"errorMessage":"error unlocking vmm","messagePattern":"error unlocking vmm","errorType":"panic","errorClass":null,"httpStatus":null,"severity":"error","filePath":"src/vmm/src/gdb/target.rs","lineNumber":276,"sourceCode":"        }\n\n        self.paused_vcpu = None;\n\n        Ok(())\n    }\n\n    /// Resets all Vcpus to their base state\n    fn reset_all_vcpu_states(&mut self) {\n        for value in self.vcpu_state.iter_mut() {\n            value.reset_vcpu_state();\n        }\n    }\n\n    /// Shuts down the VMM\n    pub fn shutdown_vmm(&self) {\n        self.vmm\n            .lock()\n            .expect(\"error unlocking vmm\")\n            .stop(FcExitCode::Ok)\n    }\n\n    /// Pauses the requested Vcpu\n    pub fn pause_vcpu(&mut self, tid: Tid) -> Result<(), GdbTargetError> {\n        let vcpu_state = &mut self.vcpu_state[tid_to_vcpuid(tid)];\n\n        if vcpu_state.paused {\n            info!(\"Attempted to pause a vcpu already paused.\");\n            // Pausing an already paused vcpu is not considered an error case from GDB\n            return Ok(());\n        }\n\n        let vmm = self.vmm.lock()?;\n        let kvm_vm = vmm.vm.as_kvm().expect(\"GDB requires KVM\");\n        let mut handles = kvm_vm.vcpus_handles();\n        let cpu_handle = &mut handles[tid_to_vcpuid(tid)];\n","sourceCodeStart":258,"sourceCodeEnd":294,"githubUrl":"https://github.com/firecracker-microvm/firecracker/blob/0a745def42ddf4cc2a744d79a08a27ff50b5d27a/src/vmm/src/gdb/target.rs#L258-L294","documentation":"Panic in `FirecrackerTarget::shutdown_vmm` (target.rs:276): `self.vmm.lock().expect(\"error unlocking vmm\")` fires when the Vmm Mutex is poisoned — i.e. some other thread panicked while holding the lock, leaving it poisoned; every subsequent `lock()` returns Err(PoisonError). This is a secondary failure: the real fault is whatever panicked earlier while holding the Vmm lock.","triggerScenarios":"A GDB 'kill' command (or detach path that shuts down) calls shutdown_vmm after another thread already panicked inside `vmm.lock()` — e.g. one of the other `expect(\"GDB requires KVM\")` sites or a vcpu error path unwinding through a lock guard.","commonSituations":"Any earlier panic anywhere in the process while the Vmm mutex was held (device error, KVM ioctl failure, the as_kvm expects) followed by the GDB session issuing shutdown; long-running firecracker processes where a background thread hit a corner-case panic.","solutions":["Inspect logs/stderr for the FIRST panic — fixing that removes the poisoning; this expect is only a symptom","Make shutdown resilient: `self.vmm.lock().unwrap_or_else(|e| e.into_inner()).stop(FcExitCode::Ok)` since stopping a poisoned VMM is still correct","Audit code that holds the Vmm lock across fallible operations and move panics out of the critical section"],"exampleFix":"// before\nself.vmm\n    .lock()\n    .expect(\"error unlocking vmm\")\n    .stop(FcExitCode::Ok)\n\n// after\nself.vmm\n    .lock()\n    .unwrap_or_else(|poisoned| poisoned.into_inner())\n    .stop(FcExitCode::Ok)","handlingStrategy":"fallback","validationCode":"// Nothing to validate pre-call; instead ensure no panic ever holds the Vmm lock.\n// Audit every `vmm.lock()` critical section for unwrap/expect/index panics.","typeGuard":null,"tryCatchPattern":"// Treat a poisoned VMM as still stoppable — stop() is the terminal action anyway\nlet guard = self.vmm.lock().unwrap_or_else(|p| p.into_inner());\nguard.stop(FcExitCode::Ok);","preventionTips":["Never hold the Vmm mutex across fallible/panicking operations; lock, act, drop","Log the first panic explicitly (panic hook) so poisoning root causes are identifiable","Prefer channels over shared-mutex callbacks for cross-thread Vmm control"],"tags":["rust","panic","mutex","poisoned-lock","gdb","concurrency","firecracker"],"backgroundTag":"mutex-poisoned","analyzedSha":"0a745def42ddf4cc2a744d79a08a27ff50b5d27a","analyzedAt":"2026-08-19T05:27:02.517Z","contentChangedAt":"2026-08-19T05:27:02.517Z","schemaVersion":2},"datasetVersion":"2026-09-14T00:17:10.932Z"}