hashicorp/nomad · error

executor: error waiting on process: %v

Error message

executor: error waiting on process: %v

What it means

handleWait is the goroutine that blocks on handle.exec.Wait(ctx) for the Java task's executor process. If Wait returns an error, the driver reports it as the task's exit result error (and, when the process state is nil — meaning the process was likely killed externally — it sets ExitCode -1 and OOMKilled false). This error is attached to the task's ExitResult rather than returned from an API call.

Source

Thrown at drivers/java/driver.go:605

func (d *Driver) WaitTask(ctx context.Context, taskID string) (<-chan *drivers.ExitResult, error) {
	handle, ok := d.tasks.Get(taskID)
	if !ok {
		return nil, drivers.ErrTaskNotFound
	}

	ch := make(chan *drivers.ExitResult)
	go d.handleWait(ctx, handle, ch)

	return ch, nil
}

func (d *Driver) handleWait(ctx context.Context, handle *taskHandle, ch chan *drivers.ExitResult) {
	defer close(ch)
	var result *drivers.ExitResult
	ps, err := handle.exec.Wait(ctx)
	if err != nil {
		result = &drivers.ExitResult{
			Err: fmt.Errorf("executor: error waiting on process: %v", err),
		}
		// if process state is nil, we've probably been killed, so return a reasonable
		// exit state to the handlers
		if ps == nil {
			result.ExitCode = -1
			result.OOMKilled = false
		}
	} else {
		result = &drivers.ExitResult{
			ExitCode:  ps.ExitCode,
			Signal:    ps.Signal,
			OOMKilled: ps.OOMKilled,
		}
	}

	select {
	case <-ctx.Done():
		return

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Check the task's exit error in nomad alloc status and the executor stderr logs for the root cause
2. Inspect dmesg/journalctl for OOM-killer or SIGKILL evidence on the executor process
3. Verify Nomad client and plugin versions match (no mixed-version executor communication)
4. If the alloc was being stopped, this is expected noise — the exit code -1 with nil process state indicates external kill
5. Reschedule/restart the allocation once the underlying cause is addressed

Example fix

// before: killing executor manually during upgrades
sudo pkill -f nomad_java_driver
// after: let Nomad stop the task so Wait() returns a normal exit
nomad alloc stop <alloc_id>

result := <-ch
if result.Err != nil && strings.Contains(result.Err.Error(), "executor: error waiting on process") {
    if result.ExitCode == -1 && !result.OOMKilled {
        // process state nil => externally killed; check dmesg/journalctl
        logger.Warn("task executor was killed externally", "err", result.Err)
    }
}

When it happens

Trigger: The executor plugin's Wait call fails: executor process crashed, plugin connection to the executor died, context canceled during shutdown, or the OS process vanished without a normal exit (killed via signal, OOM-adjacent kills without cgroup OOM detection).

Common situations: Host OOM killer or admin SIGKILL on the executor, Nomad client shutdown mid-task, executor binary/plugin incompatibility after a Nomad version upgrade, or a JVM crash that takes the executor path down abnormally.

Related errors

• failed to reattach to executor: %v
• failed to create executor: %v
• failed to launch command with executor: %v
• ErrCgroupMustBeSet
• user name must contain domain

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/e809f39a8c31afab. Report an issue: GitHub.