{"record":{"id":"2b00900d45027ef6","repo":"hashicorp/nomad","slug":"process-failed-to-exit-after-15-seconds","errorCode":null,"errorMessage":"process failed to exit after 15 seconds","messagePattern":"process failed to exit after 15 seconds","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"warning","filePath":"drivers/shared/executor/executor_linux_cgo.go","lineNumber":406,"sourceCode":"\t\t\treturn nil\n\t\tcase <-time.After(grace):\n\t\t\tif err := l.container.Signal(os.Kill); err != nil {\n\t\t\t\treturn err\n\t\t\t}\n\t\t}\n\t} else {\n\t\terr := l.container.Signal(os.Kill)\n\t\tif err != nil {\n\t\t\tl.logger.Info(\"no grace fail\", \"error\", err)\n\t\t\treturn err\n\t\t}\n\t}\n\n\tselect {\n\tcase <-l.userProcExited:\n\t\treturn nil\n\tcase <-time.After(time.Second * 15):\n\t\treturn fmt.Errorf(\"process failed to exit after 15 seconds\")\n\t}\n}\n\n// UpdateResources updates the resource isolation with new values to be enforced\nfunc (l *LibcontainerExecutor) UpdateResources(resources *drivers.Resources) error {\n\treturn nil\n}\n\n// Version returns the api version of the executor\nfunc (l *LibcontainerExecutor) Version() (*ExecutorVersion, error) {\n\treturn &ExecutorVersion{Version: ExecutorVersionLatest}, nil\n}\n\n// Stats returns the resource statistics for processes managed by the executor\nfunc (l *LibcontainerExecutor) Stats(ctx context.Context, interval time.Duration) (<-chan *cstructs.TaskResourceUsage, error) {\n\tch := make(chan *cstructs.TaskResourceUsage)\n\tgo l.handleStats(ch, ctx, interval)\n\treturn ch, nil","sourceCodeStart":388,"sourceCodeEnd":424,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/drivers/shared/executor/executor_linux_cgo.go#L388-L424","documentation":"During graceful Shutdown, the executor sends the kill signal to the container's initial processes and then waits on the l.userProcExited channel. If the task's user process does not exit within 15 seconds of the signal, Shutdown gives up and returns this timeout error, after which Nomad proceeds with forceful cleanup.","triggerScenarios":"The container's initial process (and children) remain alive for >15s after receiving the configured shutdown signal — app ignores SIGTERM/SIGINT, or shutdown handler takes too long.","commonSituations":"Application traps SIGTERM but its cleanup exceeds 15s; app never installs a signal handler and ignores the default; blocked in I/O or a deadlock preventing exit; kill signal sent but the process actually listens on a different mechanism (e.g. child process holding the tty).","solutions":["Configure the app to exit promptly on the configured kill signal (install a SIGTERM handler that terminates) or change task kill_signal to one the app honors","Reduce app shutdown work or make it async so it finishes under 15 seconds","Use Nomad's kill_timeout plus driver force-kill behavior to ensure the task is eventually SIGKILLed","Investigate with ps/strace why the process remains after the signal (deadlock, D-state, ignored signal)"],"exampleFix":null,"handlingStrategy":"try-catch","validationCode":"// verify the app honors the configured signal before relying on graceful shutdown\n// e.g. ensure the process installs a SIGTERM handler:\n//   signal.Notify/sh, or test: kill -TERM <pid> and confirm exit within timeout","typeGuard":null,"tryCatchPattern":"if err := executor.Shutdown(); err != nil && strings.Contains(err.Error(), \"failed to exit after 15 seconds\") {\n    // rely on driver force-kill, and alert that the app ignores/handles signals slowly\n    log.Printf(\"graceful shutdown timed out: %v\", err)\n}","preventionTips":["Install prompt SIGTERM/SIGINT handlers in task applications","Keep shutdown cleanup under 15 seconds or move slow work to a pre-stop hook","Set an appropriate kill_signal the app actually listens to","Monitor tasks that regularly hit force-kill after shutdown timeouts"],"tags":["signals","timeout","shutdown","process-lifecycle"],"backgroundTag":"graceful-shutdown-timeout","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}