hashicorp/nomad · error

failed to reattach to executor: %v

Error message

failed to reattach to executor: %v

What it means

This error is returned by RecoverTask when executor.ReattachToExecutor fails to reconnect to the still-running executor plugin process. The reattach config was valid, but the underlying connection (gRPC/netrpc over a unix socket or TCP) could not be established, meaning the executor has likely exited or the socket is gone.

Source

Thrown at drivers/java/driver.go:412

		d.logger.Error("failed to decode taskConfig state from handle", "error", err, "task_id", handle.Config.ID)
		return fmt.Errorf("failed to decode taskConfig state from handle: %v", err)
	}

	plugRC, err := pstructs.ReattachConfigToGoPlugin(taskState.ReattachConfig)
	if err != nil {
		d.logger.Error("failed to build ReattachConfig from taskConfig state", "error", err, "task_id", handle.Config.ID)
		return fmt.Errorf("failed to build ReattachConfig from taskConfig state: %v", err)
	}

	execImpl, pluginClient, err := executor.ReattachToExecutor(
		plugRC,
		d.logger.With("task_name", handle.Config.Name, "alloc_id", handle.Config.AllocID),
		d.nomadConfig.Topology.Compute(),
	)

	if err != nil {
		d.logger.Error("failed to reattach to executor", "error", err, "task_id", handle.Config.ID)
		return fmt.Errorf("failed to reattach to executor: %v", err)
	}

	h := &taskHandle{
		exec:         execImpl,
		pid:          taskState.Pid,
		pluginClient: pluginClient,
		taskConfig:   taskState.TaskConfig,
		procState:    drivers.TaskStateRunning,
		startedAt:    taskState.StartedAt,
		exitResult:   &drivers.ExitResult{},
		logger:       d.logger,
	}

	d.tasks.Set(taskState.TaskConfig.ID, h)

	go h.run()
	return nil
}

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Restart the affected allocation (nomad alloc stop) to launch a new executor
2. Confirm the executor process is still running and its socket exists for the alloc
3. Check client logs for the underlying reattach error (connection refused / protocol mismatch)
4. Ensure the whole cluster runs a compatible Nomad version

Example fix

// remediation: replace the orphaned allocation
// nomad alloc stop <alloc_id>

// before recovering, verify the executor socket/process is alive for the alloc
// e.g. check /var/lib/nomad/alloc/<id>/alloc/logs and the executor PID

if err := driver.RecoverTask(handle); err != nil {
    if strings.Contains(err.Error(), "failed to reattach to executor") {
        // retry once, then replace the allocation
        time.Sleep(retryDelay)
        if err := driver.RecoverTask(handle); err != nil {
            _ = client.Allocations().Stop(ctx, alloc, nil)
        }
    }
}

When it happens

Trigger: Recovering a Java task after client restart when the executor process died (host reboot, OOM-kill of executor), the unix socket was removed, or a protocol handshake failure occurred.

Common situations: Host reboots leaving stale allocation state; executors killed by OOM; mixed Nomad versions in a rolling upgrade where the plugin protocol changed; socket permissions issues.

Related errors

• failed to build ReattachConfig from taskConfig state: %v
• failed to reattach to executor: %v
• handle cannot be nil
• failed to decode taskConfig state from handle: %v
• failed to create executor: %v

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/b8363e71d72a6f9a. Report an issue: GitHub.