hashicorp/nomad · error

failed to reattach to executor: %v

Error message

failed to reattach to executor: %v

What it means

RecoverTask calls executor.ReattachToExecutor to re-establish the plugin connection to the still-running executor using the reattach config. If the connection cannot be made (process gone, socket unavailable, RPC handshake failure), this wrapped error is returned.

Source

Thrown at drivers/qemu/driver.go:319

	if err := handle.GetDriverState(&taskState); err != nil {
		d.logger.Error("failed to decode taskConfig state from handle", "error", err, "task_id", handle.Config.ID)
		return fmt.Errorf("failed to decode taskConfig state from handle: %v", err)
	}

	plugRC, err := pstructs.ReattachConfigToGoPlugin(taskState.ReattachConfig)
	if err != nil {
		d.logger.Error("failed to build ReattachConfig from taskConfig state", "error", err, "task_id", handle.Config.ID)
		return fmt.Errorf("failed to build ReattachConfig from taskConfig state: %v", err)
	}

	execImpl, pluginClient, err := executor.ReattachToExecutor(
		plugRC,
		d.logger.With("task_name", handle.Config.Name, "alloc_id", handle.Config.AllocID),
		d.nomadConfig.Topology.Compute(),
	)
	if err != nil {
		d.logger.Error("failed to reattach to executor", "error", err, "task_id", handle.Config.ID)
		return fmt.Errorf("failed to reattach to executor: %v", err)
	}

	// Try to restore monitor socket path.
	taskDir := filepath.Join(handle.Config.AllocDir, handle.Config.Name)
	possiblePaths := []string{
		filepath.Join(taskDir, qemuMonitorSocketName),
		// Support restoring tasks that used the old socket name.
		filepath.Join(taskDir, "qemu-monitor.sock"),
	}

	var monitorPath string
	for _, path := range possiblePaths {
		if _, err := os.Stat(path); err == nil {
			monitorPath = path
			d.logger.Debug("found existing monitor socket", "monitor", monitorPath)
			break
		}
	}

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Verify the executor process for the allocation is still running (ps -p <pid>); if not, allow Nomad to reschedule the task instead of recovering.
2. Restart the allocation on the client so a fresh executor is started.
3. Check the inner error for whether the plugin socket path exists and the handshake versions match; upgrade the client if versions diverged.

var err error
for i := 0; i < 3; i++ {
    err = d.RecoverTask(handle)
    if err == nil || !strings.Contains(err.Error(), "failed to reattach to executor") {
        break
    }
    time.Sleep(2 * time.Second)
}
if err != nil {
    // executor likely gone: fall back to rescheduling
    return reschedule(allocID)
}

When it happens

Trigger: executor.ReattachToExecutor fails during RecoverTask — typically because the executor process no longer exists, its plugin socket was removed, or the RPC handshake failed after a client restart.

Common situations: Host rebooted so the recorded PID is gone; /var/lib/nomad or plugin sockets cleaned up; executor crashed while the allocation was marked running; version mismatch in plugin handshake.

Related errors

• failed to reattach to executor: %v
• failed to build ReattachConfig from taskConfig state: %v
• failed to build ReattachConfig from taskConfig state: %v
• error: handle cannot be nil
• failed to decode taskConfig state from handle: %v

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/8d153f61c5a28630. Report an issue: GitHub.