hashicorp/nomad · error

failed to reattach to executor: %v

Error message

failed to reattach to executor: %v

What it means

RecoverTask failed when executor.ReattachToExecutor could not reconnect to the still-running executor plugin process (go-plugin reattach via pid + gRPC address). This typically means the executor process is gone or its socket is unreachable, so the driver cannot regain control of the task.

Source

Thrown at drivers/rawexec/driver.go:349

		d.logger.Error("failed to decode task state from handle", "error", err, "task_id", handle.Config.ID)
		return fmt.Errorf("failed to decode task state from handle: %v", err)
	}

	plugRC, err := pstructs.ReattachConfigToGoPlugin(taskState.ReattachConfig)
	if err != nil {
		d.logger.Error("failed to build ReattachConfig from task state", "error", err, "task_id", handle.Config.ID)
		return fmt.Errorf("failed to build ReattachConfig from task state: %v", err)
	}

	// Create client for reattached executor
	exec, pluginClient, err := executor.ReattachToExecutor(
		plugRC,
		d.logger.With("task_name", handle.Config.Name, "alloc_id", handle.Config.AllocID),
		d.compute,
	)
	if err != nil {
		d.logger.Error("failed to reattach to executor", "error", err, "task_id", handle.Config.ID)
		return fmt.Errorf("failed to reattach to executor: %v", err)
	}

	h := &taskHandle{
		exec:         exec,
		pid:          taskState.Pid,
		pluginClient: pluginClient,
		taskConfig:   taskState.TaskConfig,
		procState:    drivers.TaskStateRunning,
		startedAt:    taskState.StartedAt,
		exitResult:   &drivers.ExitResult{},
		logger:       d.logger,
		doneCh:       make(chan struct{}),
	}

	d.tasks.Set(taskState.TaskConfig.ID, h)

	go h.run()
	return nil

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Confirm whether the task/executor process is still alive (ps) — if not, stop the allocation so Nomad relaunches it: nomad alloc stop <alloc_id>
2. Check executor logs in the alloc dir (alloc/logs and executor stdout/stderr) for why the plugin handshake failed
3. Ensure the client data_dir was not copied to another host (stale pids/sockets are unusable)
4. Verify Nomad client binaries were not upgraded mid-flight with a running executor of an incompatible plugin version

// before relying on recovery, check executor process exists
if out, err := exec.Command("kill", "-0", strconv.Itoa(pid)).Run(); err != nil {
  // executor is dead — recovery will fail; reschedule instead
}

if err := d.RecoverTask(handle); err != nil {
  if strings.Contains(err.Error(), "failed to reattach to executor") {
    // transient? retry with backoff once, else reschedule allocation
    return recoverWithRetry(handle, 2)
  }
}

When it happens

Trigger: Client restart calls RecoverTask; the executor process died while nomad was down, the reattach socket/address is stale, or handshake with the plugin fails (version/plugin mismatch, killed PID).

Common situations: Host rebooted or OOM-killer killed the executor; data_dir moved between machines so pid/address are meaningless; plugin handshake failures after binary upgrades; long downtime exceeding executor lifetime.

Related errors

• failed to build ReattachConfig from task state: %v
• failed to create executor: %v
• failed to reattach to executor: %v
• failed to reattach to executor: %v
• executor: error waiting on process: %v

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/5da5ab414fa07446. Report an issue: GitHub.