{"record":{"id":"0590bfd26fc5162f","repo":"hashicorp/nomad","slug":"failed-to-reattach-to-executor-v","errorCode":null,"errorMessage":"failed to reattach to executor: %v","messagePattern":"failed to reattach to executor: (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"drivers/exec/driver.go","lineNumber":438,"sourceCode":"\t\td.logger.Error(\"failed to decode task state from handle\", \"error\", err, \"task_id\", handle.Config.ID)\n\t\treturn fmt.Errorf(\"failed to decode task state from handle: %v\", err)\n\t}\n\n\t// Create client for reattached executor\n\tplugRC, err := pstructs.ReattachConfigToGoPlugin(taskState.ReattachConfig)\n\tif err != nil {\n\t\td.logger.Error(\"failed to build ReattachConfig from task state\", \"error\", err, \"task_id\", handle.Config.ID)\n\t\treturn fmt.Errorf(\"failed to build ReattachConfig from task state: %v\", err)\n\t}\n\n\texec, pluginClient, err := executor.ReattachToExecutor(\n\t\tplugRC,\n\t\td.logger.With(\"task_name\", handle.Config.Name, \"alloc_id\", handle.Config.AllocID),\n\t\td.compute,\n\t)\n\tif err != nil {\n\t\td.logger.Error(\"failed to reattach to executor\", \"error\", err, \"task_id\", handle.Config.ID)\n\t\treturn fmt.Errorf(\"failed to reattach to executor: %v\", err)\n\t}\n\n\th := &taskHandle{\n\t\texec:         exec,\n\t\tpid:          taskState.Pid,\n\t\tpluginClient: pluginClient,\n\t\ttaskConfig:   taskState.TaskConfig,\n\t\tprocState:    drivers.TaskStateRunning,\n\t\tstartedAt:    taskState.StartedAt,\n\t\texitResult:   &drivers.ExitResult{},\n\t\tlogger:       d.logger,\n\t}\n\n\td.tasks.Set(taskState.TaskConfig.ID, h)\n\n\tgo h.run()\n\treturn nil\n}","sourceCodeStart":420,"sourceCodeEnd":456,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/drivers/exec/driver.go#L420-L456","documentation":"After converting the persisted ReattachConfig, RecoverTask calls executor.ReattachToExecutor to reconnect to the live executor plugin process. If the reattach handshake fails, the driver returns this error and the task cannot be adopted by the restarted client. The executor process may have died, its unix socket may be gone, or the plugin handshake failed.","triggerScenarios":"Client restarts while the executor process has already exited; the executor's reattach socket was removed (e.g., tmp cleanup); go-plugin handshake/version mismatch between client and executor binary.","commonSituations":"Host rebooted and executor PIDs are gone but state still lists them; /var/lib/nomad or temp dirs cleaned by tmpfiles.d; Nomad binary upgraded mid-flight so plugin protocol versions differ.","solutions":["Check whether the executor PID from the task state is still alive (ps -p <pid>); if not, GC the allocation and reschedule.","Confirm the reattach socket path exists and is accessible by the nomad user.","Ensure the Nomad client binary version matches what launched the task; upgrade across versions may break protocol compatibility.","If the task is unrecoverable, stop it and allow the scheduler to reschedule it cleanly."],"exampleFix":null,"handlingStrategy":"retry","validationCode":"// check the executor process is still alive before attempting recovery\nps, err := os.FindProcess(pid)\nif err != nil || signalZero(ps) != nil {\n    // process gone: skip RecoverTask, stop and reschedule\n}","typeGuard":"func socketReachable(addr net.Addr) bool {\n    c, err := net.Dial(addr.Network(), addr.String())\n    if err != nil { return false }\n    c.Close()\n    return true\n}","tryCatchPattern":null,"preventionTips":["Do not clean /var/lib/nomad or plugin temp dirs while allocations are running.","Keep Nomad client binaries at consistent versions across restarts.","Exclude Nomad state/socket paths from tmpwatch/tmpfiles.d cleanup."],"tags":["nomad","exec-driver","go-plugin","task-recovery"],"backgroundTag":"plugin-reattach-failed","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}