{"record":{"id":"8d153f61c5a28630","repo":"hashicorp/nomad","slug":"failed-to-reattach-to-executor-v-8d153f","errorCode":null,"errorMessage":"failed to reattach to executor: %v","messagePattern":"failed to reattach to executor: (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"drivers/qemu/driver.go","lineNumber":319,"sourceCode":"\tif err := handle.GetDriverState(&taskState); err != nil {\n\t\td.logger.Error(\"failed to decode taskConfig state from handle\", \"error\", err, \"task_id\", handle.Config.ID)\n\t\treturn fmt.Errorf(\"failed to decode taskConfig state from handle: %v\", err)\n\t}\n\n\tplugRC, err := pstructs.ReattachConfigToGoPlugin(taskState.ReattachConfig)\n\tif err != nil {\n\t\td.logger.Error(\"failed to build ReattachConfig from taskConfig state\", \"error\", err, \"task_id\", handle.Config.ID)\n\t\treturn fmt.Errorf(\"failed to build ReattachConfig from taskConfig state: %v\", err)\n\t}\n\n\texecImpl, pluginClient, err := executor.ReattachToExecutor(\n\t\tplugRC,\n\t\td.logger.With(\"task_name\", handle.Config.Name, \"alloc_id\", handle.Config.AllocID),\n\t\td.nomadConfig.Topology.Compute(),\n\t)\n\tif err != nil {\n\t\td.logger.Error(\"failed to reattach to executor\", \"error\", err, \"task_id\", handle.Config.ID)\n\t\treturn fmt.Errorf(\"failed to reattach to executor: %v\", err)\n\t}\n\n\t// Try to restore monitor socket path.\n\ttaskDir := filepath.Join(handle.Config.AllocDir, handle.Config.Name)\n\tpossiblePaths := []string{\n\t\tfilepath.Join(taskDir, qemuMonitorSocketName),\n\t\t// Support restoring tasks that used the old socket name.\n\t\tfilepath.Join(taskDir, \"qemu-monitor.sock\"),\n\t}\n\n\tvar monitorPath string\n\tfor _, path := range possiblePaths {\n\t\tif _, err := os.Stat(path); err == nil {\n\t\t\tmonitorPath = path\n\t\t\td.logger.Debug(\"found existing monitor socket\", \"monitor\", monitorPath)\n\t\t\tbreak\n\t\t}\n\t}","sourceCodeStart":301,"sourceCodeEnd":337,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/drivers/qemu/driver.go#L301-L337","documentation":"RecoverTask calls executor.ReattachToExecutor to re-establish the plugin connection to the still-running executor using the reattach config. If the connection cannot be made (process gone, socket unavailable, RPC handshake failure), this wrapped error is returned.","triggerScenarios":"executor.ReattachToExecutor fails during RecoverTask — typically because the executor process no longer exists, its plugin socket was removed, or the RPC handshake failed after a client restart.","commonSituations":"Host rebooted so the recorded PID is gone; /var/lib/nomad or plugin sockets cleaned up; executor crashed while the allocation was marked running; version mismatch in plugin handshake.","solutions":["Verify the executor process for the allocation is still running (ps -p <pid>); if not, allow Nomad to reschedule the task instead of recovering.","Restart the allocation on the client so a fresh executor is started.","Check the inner error for whether the plugin socket path exists and the handshake versions match; upgrade the client if versions diverged."],"exampleFix":null,"handlingStrategy":"retry","validationCode":null,"typeGuard":null,"tryCatchPattern":"var err error\nfor i := 0; i < 3; i++ {\n    err = d.RecoverTask(handle)\n    if err == nil || !strings.Contains(err.Error(), \"failed to reattach to executor\") {\n        break\n    }\n    time.Sleep(2 * time.Second)\n}\nif err != nil {\n    // executor likely gone: fall back to rescheduling\n    return reschedule(allocID)\n}","preventionTips":["Check the executor PID is alive before attempting recovery.","Avoid host reboots/cleanup of Nomad plugin sockets mid-allocation; use orderly client shutdown.","Prefer rescheduling over repeated recovery when the executor is confirmed dead."],"tags":["nomad","qemu-driver","executor","reattach","recover-task"],"backgroundTag":"failed-to-reattach-to-executor","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}