{"record":{"id":"39c181006f65bd2e","repo":"hashicorp/nomad","slug":"executor-error-waiting-on-process-v-39c181","errorCode":null,"errorMessage":"executor: error waiting on process: %v","messagePattern":"executor: error waiting on process: (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"drivers/rawexec/driver.go","lineNumber":502,"sourceCode":"func (d *Driver) WaitTask(ctx context.Context, taskID string) (<-chan *drivers.ExitResult, error) {\n\thandle, ok := d.tasks.Get(taskID)\n\tif !ok {\n\t\treturn nil, drivers.ErrTaskNotFound\n\t}\n\n\tch := make(chan *drivers.ExitResult)\n\tgo d.handleWait(ctx, handle, ch)\n\n\treturn ch, nil\n}\n\nfunc (d *Driver) handleWait(ctx context.Context, handle *taskHandle, ch chan *drivers.ExitResult) {\n\tdefer close(ch)\n\tvar result *drivers.ExitResult\n\tps, err := handle.exec.Wait(ctx)\n\tif err != nil {\n\t\tresult = &drivers.ExitResult{\n\t\t\tErr: fmt.Errorf(\"executor: error waiting on process: %v\", err),\n\t\t}\n\t\t// if process state is nil, we've probably been killed, so return a reasonable\n\t\t// exit state to the handlers\n\t\tif ps == nil {\n\t\t\tresult.ExitCode = -1\n\t\t\tresult.OOMKilled = false\n\t\t}\n\t} else {\n\t\tresult = &drivers.ExitResult{\n\t\t\tExitCode:  ps.ExitCode,\n\t\t\tSignal:    ps.Signal,\n\t\t\tOOMKilled: ps.OOMKilled,\n\t\t}\n\t}\n\n\tselect {\n\tcase <-ctx.Done():\n\t\treturn","sourceCodeStart":484,"sourceCodeEnd":520,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/drivers/rawexec/driver.go#L484-L520","documentation":"In Nomad's raw_exec driver, handleWait waits on the executor-managed process via handle.exec.Wait(ctx). If the executor returns an error while waiting on the underlying process (rather than a normal exit), the driver wraps it with the 'executor: error waiting on process' prefix and reports it as the task's ExitResult.Err.","triggerScenarios":"handle.exec.Wait returns a non-nil error AND a nil process state, e.g. the executor plugin died, the process was killed externally, the wait was cancelled by context teardown, or the executor could not reap the child.","commonSituations":"Task processes killed by external signals (OOM killer, admin kill, cgroup teardown); executor plugin crash or IPC channel closing mid-task; Nomad client shutdown cancelling the wait context.","solutions":["Check the client logs for executor plugin crashes or kill signals around the failure time","Inspect system logs (dmesg/journalctl) for OOM-killer or signal kills of the task process","Ensure no external supervisor (systemd, monit) is killing processes under Nomad's client user","Retry the task allocation; if persistent, check Nomad/executor version compatibility"],"exampleFix":null,"handlingStrategy":"try-catch","validationCode":null,"typeGuard":null,"tryCatchPattern":"// treat ExitResult.Err as task failure\nresult := <-exitCh\nif result.Err != nil {\n    log.Warn(\"task wait failed\", \"err\", result.Err, \"exitCode\", result.ExitCode)\n    // inspect OOMKilled/ExitCode -1 as 'killed externally' signal\n}","preventionTips":["Monitor client logs for executor plugin deaths","Avoid external process supervisors managing Nomad's tasks","Check dmesg/OOM events when tasks report this error"],"tags":["process","executor","nomad","rawexec"],"backgroundTag":"process-wait-failed","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}