hashicorp/nomad · warning

executor shutdown error: %v

Error message

executor shutdown error: %v

What it means

shutdownProcess sends a signal (default SIGINT) to the task's OS process during Shutdown. If Signal returns any error other than the expected 'process already finished' (finishedErr), it is wrapped in this message. Commonly it means the process already exited in a different way or is not signalable (e.g. zombie/permission).

Source

Thrown at drivers/shared/executor/executor_unix.go:53

	if e.childCmd.SysProcAttr != nil && e.childCmd.SysProcAttr.Setpgid {
		e.logger.Trace("sending sigkill to process group", "pid", pid, "negative", negative, "signal", signal)
		if err := syscall.Kill(negative, signal); err != nil && err.Error() != noSuchProcessErr {
			return err
		}
		return nil
	}
	return process.Kill()
}

// Only send the process a shutdown signal (default INT), doesn't
// necessarily kill it.
func (e *UniversalExecutor) shutdownProcess(sig os.Signal, proc *os.Process) error {
	if sig == nil {
		sig = os.Interrupt
	}

	if err := proc.Signal(sig); err != nil && err.Error() != finishedErr {
		return fmt.Errorf("executor shutdown error: %v", err)
	}

	return nil
}

// setCmdUser takes a user id as a string and looks up the user, and sets the command
// to execute as that user.
func setCmdUser(cmd *exec.Cmd, userid string) error {
	u, err := users.Lookup(userid)
	if err != nil {
		return fmt.Errorf("failed to identify user %v: %v", userid, err)
	}

	// Get the groups the user is a part of
	gidStrings, err := u.GroupIds()
	if err != nil {
		return fmt.Errorf("unable to lookup user's group membership: %v", err)
	}

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Treat as benign if the task already exited — check task logs and let Nomad reconcile state
2. Use ForceStop/kill escalation if a stale process remains (kill the process group)
3. Check whether a wrapper/exec chain exited early leaving a stale PID, and fix the driver's process handling
4. Upgrade Nomad if a known mismatch in finishedErr comparison was fixed in newer versions

Example fix

// before
if err := proc.Signal(sig); err != nil && err.Error() != finishedErr {
    return fmt.Errorf("executor shutdown error: %v", err)
}
// after (caller-side tolerance)
if err := exec.Shutdown(); err != nil {
    if strings.Contains(err.Error(), "process already finished") {
        return nil // benign
    }
    return err
}

if err := executor.Shutdown(); err != nil {
    if strings.Contains(err.Error(), "executor shutdown error") {
        // likely process already exited; check state and continue teardown
        logger.Warn("shutdown signal failed", "err", err)
        return nil
    }
    return err
}

When it happens

Trigger: UniversalExecutor.Shutdown -> shutdownProcess calls proc.Signal(sig); signal fails with 'os: process already finished' variants not matching finishedErr, 'no such process', or EPERM.

Common situations: Task process crashed just before shutdown (already finished with slightly different error text); process reaped by another watcher; client stopping a task whose PID changed (exec'd wrapper exited); permission issues signaling a setuid'd user process.

Related errors

• error unknown signal given for shutdown: %s
• process failed to exit after 15 seconds
• monitorPath not set
• client stopped and may not longer create config entries
• failed to parse signal: %w

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/70ff7532be768a79. Report an issue: GitHub.