{"record":{"id":"b8363e71d72a6f9a","repo":"hashicorp/nomad","slug":"failed-to-reattach-to-executor-v-b8363e","errorCode":null,"errorMessage":"failed to reattach to executor: %v","messagePattern":"failed to reattach to executor: (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"drivers/java/driver.go","lineNumber":412,"sourceCode":"\t\td.logger.Error(\"failed to decode taskConfig state from handle\", \"error\", err, \"task_id\", handle.Config.ID)\n\t\treturn fmt.Errorf(\"failed to decode taskConfig state from handle: %v\", err)\n\t}\n\n\tplugRC, err := pstructs.ReattachConfigToGoPlugin(taskState.ReattachConfig)\n\tif err != nil {\n\t\td.logger.Error(\"failed to build ReattachConfig from taskConfig state\", \"error\", err, \"task_id\", handle.Config.ID)\n\t\treturn fmt.Errorf(\"failed to build ReattachConfig from taskConfig state: %v\", err)\n\t}\n\n\texecImpl, pluginClient, err := executor.ReattachToExecutor(\n\t\tplugRC,\n\t\td.logger.With(\"task_name\", handle.Config.Name, \"alloc_id\", handle.Config.AllocID),\n\t\td.nomadConfig.Topology.Compute(),\n\t)\n\n\tif err != nil {\n\t\td.logger.Error(\"failed to reattach to executor\", \"error\", err, \"task_id\", handle.Config.ID)\n\t\treturn fmt.Errorf(\"failed to reattach to executor: %v\", err)\n\t}\n\n\th := &taskHandle{\n\t\texec:         execImpl,\n\t\tpid:          taskState.Pid,\n\t\tpluginClient: pluginClient,\n\t\ttaskConfig:   taskState.TaskConfig,\n\t\tprocState:    drivers.TaskStateRunning,\n\t\tstartedAt:    taskState.StartedAt,\n\t\texitResult:   &drivers.ExitResult{},\n\t\tlogger:       d.logger,\n\t}\n\n\td.tasks.Set(taskState.TaskConfig.ID, h)\n\n\tgo h.run()\n\treturn nil\n}","sourceCodeStart":394,"sourceCodeEnd":430,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/drivers/java/driver.go#L394-L430","documentation":"This error is returned by RecoverTask when executor.ReattachToExecutor fails to reconnect to the still-running executor plugin process. The reattach config was valid, but the underlying connection (gRPC/netrpc over a unix socket or TCP) could not be established, meaning the executor has likely exited or the socket is gone.","triggerScenarios":"Recovering a Java task after client restart when the executor process died (host reboot, OOM-kill of executor), the unix socket was removed, or a protocol handshake failure occurred.","commonSituations":"Host reboots leaving stale allocation state; executors killed by OOM; mixed Nomad versions in a rolling upgrade where the plugin protocol changed; socket permissions issues.","solutions":["Restart the affected allocation (nomad alloc stop) to launch a new executor","Confirm the executor process is still running and its socket exists for the alloc","Check client logs for the underlying reattach error (connection refused / protocol mismatch)","Ensure the whole cluster runs a compatible Nomad version"],"exampleFix":"// remediation: replace the orphaned allocation\n// nomad alloc stop <alloc_id>","handlingStrategy":"retry","validationCode":"// before recovering, verify the executor socket/process is alive for the alloc\n// e.g. check /var/lib/nomad/alloc/<id>/alloc/logs and the executor PID","typeGuard":null,"tryCatchPattern":"if err := driver.RecoverTask(handle); err != nil {\n    if strings.Contains(err.Error(), \"failed to reattach to executor\") {\n        // retry once, then replace the allocation\n        time.Sleep(retryDelay)\n        if err := driver.RecoverTask(handle); err != nil {\n            _ = client.Allocations().Stop(ctx, alloc, nil)\n        }\n    }\n}","preventionTips":["Set sensible resource limits so executors are not OOM-killed","Monitor executor process liveness and alloc socket health","Plan reboots with node drains so stale state is not reattached","Keep protocol versions consistent during upgrades"],"tags":["nomad","java-driver","recover-task","executor","reattach"],"backgroundTag":"failed-to-reattach-to-executor","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}