{"record":{"id":"899002361b304f3a","repo":"hashicorp/nomad","slug":"timed-out-waiting-for-script-checks-to-exit","errorCode":null,"errorMessage":"timed out waiting for script checks to exit","messagePattern":"timed out waiting for script checks to exit","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"warning","filePath":"client/allocrunner/taskrunner/script_check_hook.go","lineNumber":178,"sourceCode":"\t// Cancel scripts we no longer want\n\tfor id := range oldScriptChecks {\n\t\tif _, ok := h.scripts[id]; !ok {\n\t\t\tif oldScript, running := h.runningScripts[id]; running {\n\t\t\t\toldScript.cancel()\n\t\t\t}\n\t\t}\n\t}\n\treturn nil\n}\n\n// Stop implements interfaces.TaskStopHook and blocks waiting for running\n// scripts to finish (or for the shutdownWait timeout to expire).\nfunc (h *scriptCheckHook) Stop(ctx context.Context, req *interfaces.TaskStopRequest, resp *interfaces.TaskStopResponse) error {\n\th.mu.Lock()\n\tdefer h.mu.Unlock()\n\tclose(h.shutdownCh)\n\tdeadline := time.After(h.shutdownWait)\n\terr := fmt.Errorf(\"timed out waiting for script checks to exit\")\n\tfor _, script := range h.runningScripts {\n\t\tselect {\n\t\tcase <-script.wait():\n\t\tcase <-ctx.Done():\n\t\t\t// the caller is passing the background context, so\n\t\t\t// we should never really see this outside of testing\n\t\tcase <-deadline:\n\t\t\t// at this point the Consul client has been cleaned\n\t\t\t// up so we don't want to hang onto this.\n\t\t\treturn err\n\t\t}\n\t}\n\treturn nil\n}\n\nfunc (h *scriptCheckHook) newScriptChecks() map[string]*scriptCheck {\n\tscriptChecks := make(map[string]*scriptCheck)\n\tinterpolatedTaskServices := taskenv.InterpolateServices(h.taskEnv, h.task.Services)","sourceCodeStart":160,"sourceCodeEnd":196,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/client/allocrunner/taskrunner/script_check_hook.go#L160-L196","documentation":"On Stop, scriptCheckHook closes its shutdown channel and waits (up to h.shutdownWait) for each running script-ex check goroutine to exit. If any scripts are still running when the shutdownWait deadline expires, Stop returns this error indicating the checks did not terminate in time.","triggerScenarios":"Stop closes h.shutdownCh and iterates h.runningScripts selecting on script.wait(), the ctx (stop context), and the deadline timer. The error is returned when the deadline fires before all scripts' wait channels close — i.e., script check goroutines hang or run longer than shutdownWait (default in Nomad, 5m, configurable via consul.script_checks shutdown_wait).","commonSituations":"Consul script check commands hang on blocked subprocesses (waiting on network, stdin, or a locked file); a script ignores SIGTERM and has no timeout; shutdown_wait is set too low for long-running checks; many checks run concurrently with slow exec backends; a plugin/exec driver keeps the script's session alive.","solutions":["Increase shutdown_wait on the script check (or the agent's consul.script_checks config) so slow checks can finish","Fix the check script itself: add internal timeouts (timeout command), close stdin, avoid blocking network calls without deadlines","Kill/limit check commands: ensure the script is not waiting on child processes; use exec-based checks with timeouts where possible","Inspect the client logs for the check's command output to identify which script is hanging, then repair that script","If the task must stop immediately, treat this error as non-fatal — scripts are abandoned when the runner kills the task"],"exampleFix":"// before\ncheck {\n  type     = \"script\"\n  command  = \"/bin/healthcheck.sh\"\n  interval = \"30s\"\n}\n// after\ncheck {\n  type          = \"script\"\n  command       = \"/bin/healthcheck.sh\"\n  args          = [\"--timeout=5s\"]\n  interval      = \"30s\"\n  timeout       = \"10s\"\n  check_restart { limit = 3 }\n}","handlingStrategy":"try-catch","validationCode":"// bound the check script itself so Stop never outlives it\nif err := exec.Command(\"timeout\", \"10s\", \"/bin/healthcheck.sh\").Run(); err != nil {\n    log.Warnf(\"check timed out or failed: %v\", err)\n}","typeGuard":null,"tryCatchPattern":"if err := hook.Stop(ctx, req, resp); err != nil {\n    if strings.Contains(err.Error(), \"timed out waiting for script checks to exit\") {\n        log.Warn(\"script checks did not exit before shutdown deadline; proceeding\")\n        return nil // non-fatal: task teardown will kill lingering processes\n    }\n    return err\n}","preventionTips":["Set explicit timeout on script checks and internal timeouts inside check scripts","Size shutdown_wait to the slowest expected check runtime","Avoid check scripts that block on network/child processes without deadlines","Log check command output on the client to identify hanging scripts early"],"tags":["nomad","script-check","consul","shutdown-timeout","goroutine"],"backgroundTag":"script-check-shutdown-timeout","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}