{"record":{"id":"e809f39a8c31afab","repo":"hashicorp/nomad","slug":"executor-error-waiting-on-process-v-e809f3","errorCode":null,"errorMessage":"executor: error waiting on process: %v","messagePattern":"executor: error waiting on process: (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"drivers/java/driver.go","lineNumber":605,"sourceCode":"func (d *Driver) WaitTask(ctx context.Context, taskID string) (<-chan *drivers.ExitResult, error) {\n\thandle, ok := d.tasks.Get(taskID)\n\tif !ok {\n\t\treturn nil, drivers.ErrTaskNotFound\n\t}\n\n\tch := make(chan *drivers.ExitResult)\n\tgo d.handleWait(ctx, handle, ch)\n\n\treturn ch, nil\n}\n\nfunc (d *Driver) handleWait(ctx context.Context, handle *taskHandle, ch chan *drivers.ExitResult) {\n\tdefer close(ch)\n\tvar result *drivers.ExitResult\n\tps, err := handle.exec.Wait(ctx)\n\tif err != nil {\n\t\tresult = &drivers.ExitResult{\n\t\t\tErr: fmt.Errorf(\"executor: error waiting on process: %v\", err),\n\t\t}\n\t\t// if process state is nil, we've probably been killed, so return a reasonable\n\t\t// exit state to the handlers\n\t\tif ps == nil {\n\t\t\tresult.ExitCode = -1\n\t\t\tresult.OOMKilled = false\n\t\t}\n\t} else {\n\t\tresult = &drivers.ExitResult{\n\t\t\tExitCode:  ps.ExitCode,\n\t\t\tSignal:    ps.Signal,\n\t\t\tOOMKilled: ps.OOMKilled,\n\t\t}\n\t}\n\n\tselect {\n\tcase <-ctx.Done():\n\t\treturn","sourceCodeStart":587,"sourceCodeEnd":623,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/drivers/java/driver.go#L587-L623","documentation":"handleWait is the goroutine that blocks on handle.exec.Wait(ctx) for the Java task's executor process. If Wait returns an error, the driver reports it as the task's exit result error (and, when the process state is nil — meaning the process was likely killed externally — it sets ExitCode -1 and OOMKilled false). This error is attached to the task's ExitResult rather than returned from an API call.","triggerScenarios":"The executor plugin's Wait call fails: executor process crashed, plugin connection to the executor died, context canceled during shutdown, or the OS process vanished without a normal exit (killed via signal, OOM-adjacent kills without cgroup OOM detection).","commonSituations":"Host OOM killer or admin SIGKILL on the executor, Nomad client shutdown mid-task, executor binary/plugin incompatibility after a Nomad version upgrade, or a JVM crash that takes the executor path down abnormally.","solutions":["Check the task's exit error in nomad alloc status and the executor stderr logs for the root cause","Inspect dmesg/journalctl for OOM-killer or SIGKILL evidence on the executor process","Verify Nomad client and plugin versions match (no mixed-version executor communication)","If the alloc was being stopped, this is expected noise — the exit code -1 with nil process state indicates external kill","Reschedule/restart the allocation once the underlying cause is addressed"],"exampleFix":"// before: killing executor manually during upgrades\nsudo pkill -f nomad_java_driver\n// after: let Nomad stop the task so Wait() returns a normal exit\nnomad alloc stop <alloc_id>","handlingStrategy":"try-catch","validationCode":null,"typeGuard":null,"tryCatchPattern":"result := <-ch\nif result.Err != nil && strings.Contains(result.Err.Error(), \"executor: error waiting on process\") {\n    if result.ExitCode == -1 && !result.OOMKilled {\n        // process state nil => externally killed; check dmesg/journalctl\n        logger.Warn(\"task executor was killed externally\", \"err\", result.Err)\n    }\n}","preventionTips":["Keep Nomad client and driver plugin versions in lockstep","Monitor host memory so the OOM killer doesn't reap executors","Never manually SIGKILL nomad/executor processes; use alloc stop","Correlate exit code -1 with dmesg kill events before escalating"],"tags":["nomad","java-driver","executor","process-wait","exit-result"],"backgroundTag":"executor-wait-failed","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}