hashicorp/nomad · error

executor: error waiting on process: %v

Error message

executor: error waiting on process: %v

What it means

This error is produced by the exec driver's handleWait goroutine when the executor's call to wait on the task process fails. Instead of returning a normal exit code, the driver wraps the wait error in an ExitResult so the task's exit is reported as failed. It typically means the executor could not observe the process termination (e.g. the process was killed abruptly or the executor plugin died).

Source

Thrown at drivers/exec/driver.go:595

func (d *Driver) WaitTask(ctx context.Context, taskID string) (<-chan *drivers.ExitResult, error) {
	handle, ok := d.tasks.Get(taskID)
	if !ok {
		return nil, drivers.ErrTaskNotFound
	}

	ch := make(chan *drivers.ExitResult)
	go d.handleWait(ctx, handle, ch)

	return ch, nil
}

func (d *Driver) handleWait(ctx context.Context, handle *taskHandle, ch chan *drivers.ExitResult) {
	defer close(ch)
	var result *drivers.ExitResult
	ps, err := handle.exec.Wait(ctx)
	if err != nil {
		result = &drivers.ExitResult{
			Err: fmt.Errorf("executor: error waiting on process: %v", err),
		}
		// if process state is nil, we've probably been killed, so return a reasonable
		// exit state to the handlers
		if ps == nil {
			result.ExitCode = -1
			result.OOMKilled = false
		}
	} else {
		result = &drivers.ExitResult{
			ExitCode:  ps.ExitCode,
			Signal:    ps.Signal,
			OOMKilled: ps.OOMKilled,
		}
	}

	select {
	case <-ctx.Done():
		return

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Inspect the agent/executor logs around the failure to find the underlying wait error (%v) for the real cause.
2. Check for external SIGKILLs (OOM killer via dmesg/journalctl -k, manual kills, deployment scripts) targeting the task process.
3. Verify the executor plugin process is healthy and the client machine is not killing processes (cgroup, systemd-oomd, container runtime interference).
4. Re-run the task; if reproducible, reduce memory limits or fix the application crash that prevents a clean exit.

result, err := taskHandle.Wait(ctx)
if err != nil || result.Err != nil || result.ExitCode == -1 {
    // treat as abnormal termination: alert, inspect executor logs, reschedule
    log.Printf("abnormal task exit: %v", firstNonNil(err, result.Err))
}

When it happens

Trigger: handle.Wait(ctx) on the executor returns a non-nil error; ps (process state) is nil, in which case ExitCode is set to -1 and OOMKilled to false to give handlers a reasonable exit state.

Common situations: The task process was SIGKILLed from outside Nomad; the executor subprocess crashed or its plugin client exited; OOM killer or cgroup teardown interrupted the wait; Nomad client restarted mid-task.

Related errors

• default_pid_mode must be %q or %q, got %q
• default_ipc_mode must be %q or %q, got %q
• allow_caps configured with capabilities not supported by sys
• pid_mode must be %q or %q, got %q
• ipc_mode must be %q or %q, got %q

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/a45ed67ef955dc59. Report an issue: GitHub.