{"record":{"id":"ce13c254c0aefe5c","repo":"hashicorp/nomad","slug":"executor-shutdown-failed-v-ce13c2","errorCode":null,"errorMessage":"executor Shutdown failed: %v","messagePattern":"executor Shutdown failed: (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"drivers/qemu/driver.go","lineNumber":759,"sourceCode":"\t\t\t\t\t\tbreak out\n\t\t\t\t\t}\n\t\t\t\t}\n\t\t\t}\n\t\t} else {\n\t\t\td.logger.Debug(\"error sending graceful shutdown\", \"pid\", handle.pid, \"error\", err)\n\t\t}\n\t} else {\n\t\td.logger.Debug(\"monitor socket is empty, forcing shutdown\")\n\t}\n\n\t// TODO(preetha) we are calling shutdown on the executor here\n\t// after attempting a graceful qemu shutdown, qemu process may\n\t// not be around when we call exec.shutdown\n\tif err := handle.exec.Shutdown(signal, timeout); err != nil {\n\t\tif handle.pluginClient.Exited() {\n\t\t\treturn nil\n\t\t}\n\t\treturn fmt.Errorf(\"executor Shutdown failed: %v\", err)\n\t}\n\n\treturn nil\n}\n\nfunc (d *Driver) DestroyTask(taskID string, force bool) error {\n\thandle, ok := d.tasks.Get(taskID)\n\tif !ok {\n\t\treturn drivers.ErrTaskNotFound\n\t}\n\n\tif handle.IsRunning() && !force {\n\t\treturn fmt.Errorf(\"cannot destroy running task\")\n\t}\n\n\tif !handle.pluginClient.Exited() {\n\t\tif err := handle.exec.Shutdown(\"\", 0); err != nil {\n\t\t\thandle.logger.Error(\"destroying executor failed\", \"error\", err)","sourceCodeStart":741,"sourceCodeEnd":777,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/drivers/qemu/driver.go#L741-L777","documentation":"Returned by StopTask in the QEMU driver when the executor's Shutdown call fails and the plugin client is still alive. Since QEMU may already have exited gracefully, this error only surfaces when the executor plugin is still running but reported a shutdown failure.","triggerScenarios":"Calling StopTask on a QEMU task whose executor plugin has not exited but whose internal exec.Shutdown(signal, timeout) returns an error (e.g. the QEMU process died unexpectedly but the plugin process lingers).","commonSituations":"QEMU crashed or was killed externally (OOM, SIGKILL) so the executor cannot signal it; Nomad client restarting tasks; plugin/executor process in a bad state after client upgrade.","solutions":["Inspect the wrapped error (%v suffix) in client logs to find the underlying executor failure","Check if the QEMU process is still alive (ps aux | grep qemu); if it already exited, retry StopTask or DestroyTask with force=true","Restart the nomad client to clean up orphaned executor plugins","Upgrade Nomad; several executor shutdown races were fixed in later releases"],"exampleFix":"// before\nerr := driver.StopTask(taskID, 30*time.Second, \"SIGINT\") // -> executor Shutdown failed: ...\n// after\n// check plugin liveness / force-destroy path\nif err := driver.StopTask(taskID, 30*time.Second, \"SIGINT\"); err != nil {\n    _ = driver.DestroyTask(taskID, true) // force cleanup\n}","handlingStrategy":"try-catch","validationCode":"// Go: no pre-call validation; check handle state if accessible\n// if handle.pluginClient.Exited() { skip shutdown; task already gone }","typeGuard":null,"tryCatchPattern":"err := drv.StopTask(id, timeout, sig)\nif err != nil && strings.Contains(err.Error(), \"executor Shutdown failed\") {\n    // force cleanup\n    _ = drv.DestroyTask(id, true)\n}","preventionTips":["Always wrap StopTask errors with a forced DestroyTask fallback","Monitor executor plugin health on Nomad clients","Keep Nomad updated — shutdown races are patched over time"],"tags":["qemu","executor","shutdown","nomad-driver"],"backgroundTag":"executor-shutdown-failed","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}