{"record":{"id":"5da5ab414fa07446","repo":"hashicorp/nomad","slug":"failed-to-reattach-to-executor-v-5da5ab","errorCode":null,"errorMessage":"failed to reattach to executor: %v","messagePattern":"failed to reattach to executor: (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"drivers/rawexec/driver.go","lineNumber":349,"sourceCode":"\t\td.logger.Error(\"failed to decode task state from handle\", \"error\", err, \"task_id\", handle.Config.ID)\n\t\treturn fmt.Errorf(\"failed to decode task state from handle: %v\", err)\n\t}\n\n\tplugRC, err := pstructs.ReattachConfigToGoPlugin(taskState.ReattachConfig)\n\tif err != nil {\n\t\td.logger.Error(\"failed to build ReattachConfig from task state\", \"error\", err, \"task_id\", handle.Config.ID)\n\t\treturn fmt.Errorf(\"failed to build ReattachConfig from task state: %v\", err)\n\t}\n\n\t// Create client for reattached executor\n\texec, pluginClient, err := executor.ReattachToExecutor(\n\t\tplugRC,\n\t\td.logger.With(\"task_name\", handle.Config.Name, \"alloc_id\", handle.Config.AllocID),\n\t\td.compute,\n\t)\n\tif err != nil {\n\t\td.logger.Error(\"failed to reattach to executor\", \"error\", err, \"task_id\", handle.Config.ID)\n\t\treturn fmt.Errorf(\"failed to reattach to executor: %v\", err)\n\t}\n\n\th := &taskHandle{\n\t\texec:         exec,\n\t\tpid:          taskState.Pid,\n\t\tpluginClient: pluginClient,\n\t\ttaskConfig:   taskState.TaskConfig,\n\t\tprocState:    drivers.TaskStateRunning,\n\t\tstartedAt:    taskState.StartedAt,\n\t\texitResult:   &drivers.ExitResult{},\n\t\tlogger:       d.logger,\n\t\tdoneCh:       make(chan struct{}),\n\t}\n\n\td.tasks.Set(taskState.TaskConfig.ID, h)\n\n\tgo h.run()\n\treturn nil","sourceCodeStart":331,"sourceCodeEnd":367,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/drivers/rawexec/driver.go#L331-L367","documentation":"RecoverTask failed when executor.ReattachToExecutor could not reconnect to the still-running executor plugin process (go-plugin reattach via pid + gRPC address). This typically means the executor process is gone or its socket is unreachable, so the driver cannot regain control of the task.","triggerScenarios":"Client restart calls RecoverTask; the executor process died while nomad was down, the reattach socket/address is stale, or handshake with the plugin fails (version/plugin mismatch, killed PID).","commonSituations":"Host rebooted or OOM-killer killed the executor; data_dir moved between machines so pid/address are meaningless; plugin handshake failures after binary upgrades; long downtime exceeding executor lifetime.","solutions":["Confirm whether the task/executor process is still alive (ps) — if not, stop the allocation so Nomad relaunches it: nomad alloc stop <alloc_id>","Check executor logs in the alloc dir (alloc/logs and executor stdout/stderr) for why the plugin handshake failed","Ensure the client data_dir was not copied to another host (stale pids/sockets are unusable)","Verify Nomad client binaries were not upgraded mid-flight with a running executor of an incompatible plugin version"],"exampleFix":null,"handlingStrategy":"retry","validationCode":"// before relying on recovery, check executor process exists\nif out, err := exec.Command(\"kill\", \"-0\", strconv.Itoa(pid)).Run(); err != nil {\n  // executor is dead — recovery will fail; reschedule instead\n}","typeGuard":null,"tryCatchPattern":"if err := d.RecoverTask(handle); err != nil {\n  if strings.Contains(err.Error(), \"failed to reattach to executor\") {\n    // transient? retry with backoff once, else reschedule allocation\n    return recoverWithRetry(handle, 2)\n  }\n}","preventionTips":["Schedule reboots with node drains so executors shut down cleanly","Protect executors from OOM-killer (avoid tight memory limits on nomad process tree)","Never move data_dir between machines","Test plugin handshake after binary upgrades by draining and restarting clients"],"tags":["nomad","rawexec","plugin","executor","reattach"],"backgroundTag":"plugin-reattach-failed","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}