{"record":{"id":"851af102523a7c10","repo":"hashicorp/nomad","slug":"executor-shutdown-failed-v-851af1","errorCode":null,"errorMessage":"executor Shutdown failed: %v","messagePattern":"executor Shutdown failed: (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"drivers/rawexec/driver.go","lineNumber":537,"sourceCode":"\tcase <-ctx.Done():\n\t\treturn\n\tcase <-d.ctx.Done():\n\t\treturn\n\tcase ch <- result:\n\t}\n}\n\nfunc (d *Driver) StopTask(taskID string, timeout time.Duration, signal string) error {\n\thandle, ok := d.tasks.Get(taskID)\n\tif !ok {\n\t\treturn drivers.ErrTaskNotFound\n\t}\n\n\tif err := handle.exec.Shutdown(signal, timeout); err != nil {\n\t\tif handle.pluginClient.Exited() {\n\t\t\treturn nil\n\t\t}\n\t\treturn fmt.Errorf(\"executor Shutdown failed: %v\", err)\n\t}\n\n\t// Wait for handle to finish\n\t<-handle.doneCh\n\n\t// Kill executor\n\thandle.pluginClient.Kill()\n\n\treturn nil\n}\n\nfunc (d *Driver) DestroyTask(taskID string, force bool) error {\n\thandle, ok := d.tasks.Get(taskID)\n\tif !ok {\n\t\treturn drivers.ErrTaskNotFound\n\t}\n\n\tif handle.IsRunning() && !force {","sourceCodeStart":519,"sourceCodeEnd":555,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/drivers/rawexec/driver.go#L519-L555","documentation":"raw_exec's StopTask asks the executor to gracefully shut down the task (signal + timeout). If exec.Shutdown returns an error and the executor plugin has NOT exited, the driver surfaces 'executor Shutdown failed'. Notably, if the plugin already exited the error is swallowed and nil is returned.","triggerScenarios":"Calling StopTask while the executor plugin is alive but Shutdown fails: the underlying process ignores the shutdown signal, the executor IPC call errors/times out, or the executor process is wedged.","commonSituations":"Tasks that ignore SIGTERM/SIGINT and hang past the kill_timeout; frozen or deadlocked task processes; executor plugin communication failures on a loaded client.","solutions":["Wait for the kill_timeout to elapse — Nomad will force-kill (SIGKILL) after graceful shutdown fails","Check whether the task handles the shutdown signal; add a signal handler or adjust the task's kill_signal/kill_timeout","Check client logs for executor plugin errors; restart the Nomad client agent if the executor is wedged","Upgrade Nomad if the executor repeatedly fails to signal the process"],"exampleFix":null,"handlingStrategy":"try-catch","validationCode":null,"typeGuard":null,"tryCatchPattern":"if err := driver.StopTask(id, timeout, sig); err != nil {\n    // plugin may have died; check handle.pluginClient.Exited() semantics\n    log.Warn(\"graceful shutdown failed, waiting for force kill\", \"err\", err)\n    <-doneCh // Nomad will SIGKILL after kill_timeout\n}","preventionTips":["Ensure tasks handle the configured kill_signal","Set an adequate kill_timeout in the job","Watch for executor plugin crashes in client logs"],"tags":["shutdown","signal","executor","nomad","rawexec"],"backgroundTag":"graceful-shutdown-failed","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}