hashicorp/nomad · error

failed to reattach to executor: %v

Error message

failed to reattach to executor: %v

What it means

After converting the persisted ReattachConfig, RecoverTask calls executor.ReattachToExecutor to reconnect to the live executor plugin process. If the reattach handshake fails, the driver returns this error and the task cannot be adopted by the restarted client. The executor process may have died, its unix socket may be gone, or the plugin handshake failed.

Source

Thrown at drivers/exec/driver.go:438

		d.logger.Error("failed to decode task state from handle", "error", err, "task_id", handle.Config.ID)
		return fmt.Errorf("failed to decode task state from handle: %v", err)
	}

	// Create client for reattached executor
	plugRC, err := pstructs.ReattachConfigToGoPlugin(taskState.ReattachConfig)
	if err != nil {
		d.logger.Error("failed to build ReattachConfig from task state", "error", err, "task_id", handle.Config.ID)
		return fmt.Errorf("failed to build ReattachConfig from task state: %v", err)
	}

	exec, pluginClient, err := executor.ReattachToExecutor(
		plugRC,
		d.logger.With("task_name", handle.Config.Name, "alloc_id", handle.Config.AllocID),
		d.compute,
	)
	if err != nil {
		d.logger.Error("failed to reattach to executor", "error", err, "task_id", handle.Config.ID)
		return fmt.Errorf("failed to reattach to executor: %v", err)
	}

	h := &taskHandle{
		exec:         exec,
		pid:          taskState.Pid,
		pluginClient: pluginClient,
		taskConfig:   taskState.TaskConfig,
		procState:    drivers.TaskStateRunning,
		startedAt:    taskState.StartedAt,
		exitResult:   &drivers.ExitResult{},
		logger:       d.logger,
	}

	d.tasks.Set(taskState.TaskConfig.ID, h)

	go h.run()
	return nil
}

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Check whether the executor PID from the task state is still alive (ps -p <pid>); if not, GC the allocation and reschedule.
2. Confirm the reattach socket path exists and is accessible by the nomad user.
3. Ensure the Nomad client binary version matches what launched the task; upgrade across versions may break protocol compatibility.
4. If the task is unrecoverable, stop it and allow the scheduler to reschedule it cleanly.

// check the executor process is still alive before attempting recovery
ps, err := os.FindProcess(pid)
if err != nil || signalZero(ps) != nil {
    // process gone: skip RecoverTask, stop and reschedule
}

func socketReachable(addr net.Addr) bool {
    c, err := net.Dial(addr.Network(), addr.String())
    if err != nil { return false }
    c.Close()
    return true
}

When it happens

Trigger: Client restarts while the executor process has already exited; the executor's reattach socket was removed (e.g., tmp cleanup); go-plugin handshake/version mismatch between client and executor binary.

Common situations: Host rebooted and executor PIDs are gone but state still lists them; /var/lib/nomad or temp dirs cleaned by tmpfiles.d; Nomad binary upgraded mid-flight so plugin protocol versions differ.

Related errors

• failed to decode task state from handle: %v
• failed to build ReattachConfig from task state: %v
• failed to create executor: %v
• failed to start plugin: %v
• default_pid_mode must be %q or %q, got %q

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/0590bfd26fc5162f. Report an issue: GitHub.