{"record":{"id":"d7d81003ac03705b","repo":"hashicorp/nomad","slug":"executor-shutdown-failed-v-d7d810","errorCode":null,"errorMessage":"executor Shutdown failed: %v","messagePattern":"executor Shutdown failed: (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"drivers/java/driver.go","lineNumber":640,"sourceCode":"\tcase <-ctx.Done():\n\t\treturn\n\tcase <-d.ctx.Done():\n\t\treturn\n\tcase ch <- result:\n\t}\n}\n\nfunc (d *Driver) StopTask(taskID string, timeout time.Duration, signal string) error {\n\thandle, ok := d.tasks.Get(taskID)\n\tif !ok {\n\t\treturn drivers.ErrTaskNotFound\n\t}\n\n\tif err := handle.exec.Shutdown(signal, timeout); err != nil {\n\t\tif handle.pluginClient.Exited() {\n\t\t\treturn nil\n\t\t}\n\t\treturn fmt.Errorf(\"executor Shutdown failed: %v\", err)\n\t}\n\n\treturn nil\n}\n\nfunc (d *Driver) DestroyTask(taskID string, force bool) error {\n\thandle, ok := d.tasks.Get(taskID)\n\tif !ok {\n\t\treturn drivers.ErrTaskNotFound\n\t}\n\n\tif handle.IsRunning() && !force {\n\t\treturn fmt.Errorf(\"cannot destroy running task\")\n\t}\n\n\tif !handle.pluginClient.Exited() {\n\t\tif err := handle.exec.Shutdown(\"\", 0); err != nil {\n\t\t\thandle.logger.Error(\"destroying executor failed\", \"error\", err)","sourceCodeStart":622,"sourceCodeEnd":658,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/drivers/java/driver.go#L622-L658","documentation":"StopTask calls handle.exec.Shutdown(signal, timeout) to gracefully stop the task's executor, and this error wraps any failure from that shutdown. As a special case, if the plugin client has already exited (the executor process is gone), StopTask returns nil instead — so this error only surfaces when the executor is still connected but failed to shut down within the signal/timeout contract.","triggerScenarios":"Calling Driver.StopTask where exec.Shutdown returns an error AND handle.pluginClient.Exited() is false — i.e., the executor is alive but refused or failed to shut down (signal not handled, timeout expired while the JVM ignored SIGTERM, executor internal error).","commonSituations":"Java applications that trap SIGTERM and hang past the kill_timeout, a wedged JVM (thread dump deadlock, GC thrash), or executor plugin communication issues during client shutdown.","solutions":["Read the wrapped %v error to see whether it was a timeout or a shutdown RPC failure","Retry StopTask; if it still fails, DestroyTask with force=true will forcibly shut the executor down","Increase the task's kill_timeout so slow JVM shutdown hooks can complete","Inspect the executor logs for a hung or deadlocked JVM and fix the application's shutdown hooks","Verify the executor plugin connection is healthy (nomad node status, client logs)"],"exampleFix":"// before: 5s default may be too short for JVM shutdown hooks\nkill_timeout = \"5s\"\n// after: give the JVM time to run shutdown hooks\njob \"app\" {\n  group \"g\" {\n    task \"java-app\" {\n      kill_timeout = \"30s\"\n    }\n  }\n}","handlingStrategy":"try-catch","validationCode":"// ensure generous kill_timeout in the task spec so Shutdown has room\n// kill_timeout = \"30s\"","typeGuard":null,"tryCatchPattern":"err := driver.StopTask(taskID, \"SIGINT\", 30*time.Second)\nif err != nil {\n    if strings.Contains(err.Error(), \"executor Shutdown failed\") && !driver.IsRunning(taskID) {\n        return nil // executor actually stopped despite the error\n    }\n    // fall back to forced destroy\n    return driver.DestroyTask(taskID, true)\n}","preventionTips":["Set kill_timeout generously for JVM tasks with shutdown hooks","Avoid deadlocks/hangs in application shutdown hooks","Test task stop behavior after Nomad upgrades","Always follow a failed StopTask with a force DestroyTask"],"tags":["nomad","java-driver","task-stop","executor-shutdown","graceful-shutdown"],"backgroundTag":"executor-shutdown-failed","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}