hashicorp/nomad · warning

timed out waiting for script checks to exit

Error message

timed out waiting for script checks to exit

What it means

On Stop, scriptCheckHook closes its shutdown channel and waits (up to h.shutdownWait) for each running script-ex check goroutine to exit. If any scripts are still running when the shutdownWait deadline expires, Stop returns this error indicating the checks did not terminate in time.

Source

Thrown at client/allocrunner/taskrunner/script_check_hook.go:178

	// Cancel scripts we no longer want
	for id := range oldScriptChecks {
		if _, ok := h.scripts[id]; !ok {
			if oldScript, running := h.runningScripts[id]; running {
				oldScript.cancel()
			}
		}
	}
	return nil
}

// Stop implements interfaces.TaskStopHook and blocks waiting for running
// scripts to finish (or for the shutdownWait timeout to expire).
func (h *scriptCheckHook) Stop(ctx context.Context, req *interfaces.TaskStopRequest, resp *interfaces.TaskStopResponse) error {
	h.mu.Lock()
	defer h.mu.Unlock()
	close(h.shutdownCh)
	deadline := time.After(h.shutdownWait)
	err := fmt.Errorf("timed out waiting for script checks to exit")
	for _, script := range h.runningScripts {
		select {
		case <-script.wait():
		case <-ctx.Done():
			// the caller is passing the background context, so
			// we should never really see this outside of testing
		case <-deadline:
			// at this point the Consul client has been cleaned
			// up so we don't want to hang onto this.
			return err
		}
	}
	return nil
}

func (h *scriptCheckHook) newScriptChecks() map[string]*scriptCheck {
	scriptChecks := make(map[string]*scriptCheck)
	interpolatedTaskServices := taskenv.InterpolateServices(h.taskEnv, h.task.Services)

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Increase shutdown_wait on the script check (or the agent's consul.script_checks config) so slow checks can finish
2. Fix the check script itself: add internal timeouts (timeout command), close stdin, avoid blocking network calls without deadlines
3. Kill/limit check commands: ensure the script is not waiting on child processes; use exec-based checks with timeouts where possible
4. Inspect the client logs for the check's command output to identify which script is hanging, then repair that script
5. If the task must stop immediately, treat this error as non-fatal — scripts are abandoned when the runner kills the task

Example fix

// before
check {
  type     = "script"
  command  = "/bin/healthcheck.sh"
  interval = "30s"
}
// after
check {
  type          = "script"
  command       = "/bin/healthcheck.sh"
  args          = ["--timeout=5s"]
  interval      = "30s"
  timeout       = "10s"
  check_restart { limit = 3 }
}

// bound the check script itself so Stop never outlives it
if err := exec.Command("timeout", "10s", "/bin/healthcheck.sh").Run(); err != nil {
    log.Warnf("check timed out or failed: %v", err)
}

if err := hook.Stop(ctx, req, resp); err != nil {
    if strings.Contains(err.Error(), "timed out waiting for script checks to exit") {
        log.Warn("script checks did not exit before shutdown deadline; proceeding")
        return nil // non-fatal: task teardown will kill lingering processes
    }
    return err
}

When it happens

Trigger: Stop closes h.shutdownCh and iterates h.runningScripts selecting on script.wait(), the ctx (stop context), and the deadline timer. The error is returned when the deadline fires before all scripts' wait channels close — i.e., script check goroutines hang or run longer than shutdownWait (default in Nomad, 5m, configurable via consul.script_checks shutdown_wait).

Common situations: Consul script check commands hang on blocked subprocesses (waiting on network, stdin, or a locked file); a script ignores SIGTERM and has no timeout; shutdown_wait is set too low for long-running checks; many checks run concurrently with slow exec backends; a plugin/exec driver keeps the script's session alive.

Understand the failure class

• Timeouts: ETIMEDOUT, deadlines, and hung requests — what actually expires when a request times out.

Related errors

• task %q not found in updated alloc
• timed out waiting for socket proxy to exit
• error creating bootstrap configuration for Connect proxy sid
• client stopped and may not longer create config entries
• non-default Consul cluster requires Nomad Enterprise

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/899002361b304f3a. Report an issue: GitHub.