hashicorp/nomad · error

executor: error waiting on process: %v

Error message

executor: error waiting on process: %v

What it means

In Nomad's raw_exec driver, handleWait waits on the executor-managed process via handle.exec.Wait(ctx). If the executor returns an error while waiting on the underlying process (rather than a normal exit), the driver wraps it with the 'executor: error waiting on process' prefix and reports it as the task's ExitResult.Err.

Source

Thrown at drivers/rawexec/driver.go:502

func (d *Driver) WaitTask(ctx context.Context, taskID string) (<-chan *drivers.ExitResult, error) {
	handle, ok := d.tasks.Get(taskID)
	if !ok {
		return nil, drivers.ErrTaskNotFound
	}

	ch := make(chan *drivers.ExitResult)
	go d.handleWait(ctx, handle, ch)

	return ch, nil
}

func (d *Driver) handleWait(ctx context.Context, handle *taskHandle, ch chan *drivers.ExitResult) {
	defer close(ch)
	var result *drivers.ExitResult
	ps, err := handle.exec.Wait(ctx)
	if err != nil {
		result = &drivers.ExitResult{
			Err: fmt.Errorf("executor: error waiting on process: %v", err),
		}
		// if process state is nil, we've probably been killed, so return a reasonable
		// exit state to the handlers
		if ps == nil {
			result.ExitCode = -1
			result.OOMKilled = false
		}
	} else {
		result = &drivers.ExitResult{
			ExitCode:  ps.ExitCode,
			Signal:    ps.Signal,
			OOMKilled: ps.OOMKilled,
		}
	}

	select {
	case <-ctx.Done():
		return

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Check the client logs for executor plugin crashes or kill signals around the failure time
2. Inspect system logs (dmesg/journalctl) for OOM-killer or signal kills of the task process
3. Ensure no external supervisor (systemd, monit) is killing processes under Nomad's client user
4. Retry the task allocation; if persistent, check Nomad/executor version compatibility

// treat ExitResult.Err as task failure
result := <-exitCh
if result.Err != nil {
    log.Warn("task wait failed", "err", result.Err, "exitCode", result.ExitCode)
    // inspect OOMKilled/ExitCode -1 as 'killed externally' signal
}

When it happens

Trigger: handle.exec.Wait returns a non-nil error AND a nil process state, e.g. the executor plugin died, the process was killed externally, the wait was cancelled by context teardown, or the executor could not reap the child.

Common situations: Task processes killed by external signals (OOM killer, admin kill, cgroup teardown); executor plugin crash or IPC channel closing mid-task; Nomad client shutdown cancelling the wait context.

Related errors

• failed to reattach to executor: %v
• failed to create executor: %v
• failed to launch command with executor: %v
• executor Shutdown failed: %v
• only one of cgroups_v1_override and cgroups_v2_override may

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/39c181006f65bd2e. Report an issue: GitHub.