hashicorp/nomad · error

failed to build ReattachConfig from task state

Error message

failed to build ReattachConfig from task state: %v

What it means

During RecoverTask, the exec driver rebuilds a go-plugin ReattachConfig from the task state persisted in the client state DB and hands it to pstructs.ReattachConfigToGoPlugin. If that conversion fails, Nomad cannot reattach to the still-running executor process and the recovery is aborted. This typically means the persisted ReattachConfig is corrupt, incomplete, or from an incompatible plugin protocol version.

Solutions

1. Inspect the task's persisted state; if the executor process is gone or state is corrupt, stop the task and reschedule it rather than recovering.
2. Verify the client was not restored from a state directory of a different Nomad version; align versions or wipe the stale state for that allocation.
3. Check Nomad client logs for the underlying error from ReattachConfigToGoPlugin to confirm which field is invalid.
4. As a last resort, GC the allocation and let the scheduler place a new one.

// before relying on recovery, verify the persisted state is usable
if taskState.ReattachConfig == nil || taskState.ReattachConfig.Pid <= 0 {
    // treat as unrecoverable: stop + reschedule instead of RecoverTask
}

func reattachConfigValid(rc *plugin.ReattachConfig) bool {
    return rc != nil && rc.Protocol != "" && rc.Addr != nil && rc.Pid > 0
}

When it happens

Trigger: RecoverTask is called after a Nomad client restart with a task whose persisted drivers.TaskState contains a ReattachConfig that fails conversion: nil ReattachConfig, invalid Addr/Pid, or a protocol version go-plugin refuses to parse.

Common situations: Corrupted or truncated bolt state DB after a crash; upgrading Nomad across incompatible go-plugin protocol versions so old state entries no longer parse; manually edited or restored client state directory.

Related errors

• failed to decode task state from handle
• failed to reattach to executor
• allow_caps configured with capabilities not supported by…
• cannot destroy running task
• cap_add configured with capabilities not supported by system

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/015db1f51802e5c3. Report an issue: GitHub.

Appendix: source

Thrown at drivers/exec/driver.go:428

		d.logger.Trace("nothing to recover; task already exists",
			"task_id", handle.Config.ID,
			"task_name", handle.Config.Name,
		)
		return nil
	}

	// Handle doesn't already exist, try to reattach
	var taskState TaskState
	if err := handle.GetDriverState(&taskState); err != nil {
		d.logger.Error("failed to decode task state from handle", "error", err, "task_id", handle.Config.ID)
		return fmt.Errorf("failed to decode task state from handle: %v", err)
	}

	// Create client for reattached executor
	plugRC, err := pstructs.ReattachConfigToGoPlugin(taskState.ReattachConfig)
	if err != nil {
		d.logger.Error("failed to build ReattachConfig from task state", "error", err, "task_id", handle.Config.ID)
		return fmt.Errorf("failed to build ReattachConfig from task state: %v", err)
	}

	exec, pluginClient, err := executor.ReattachToExecutor(
		plugRC,
		d.logger.With("task_name", handle.Config.Name, "alloc_id", handle.Config.AllocID),
		d.compute,
	)
	if err != nil {
		d.logger.Error("failed to reattach to executor", "error", err, "task_id", handle.Config.ID)
		return fmt.Errorf("failed to reattach to executor: %v", err)
	}

	h := &taskHandle{
		exec:         exec,
		pid:          taskState.Pid,
		pluginClient: pluginClient,
		taskConfig:   taskState.TaskConfig,
		procState:    drivers.TaskStateRunning,

View on GitHub (pinned to 482b49bf1a)