hashicorp/nomad · error

failed to build ReattachConfig from task state: %v

Error message

failed to build ReattachConfig from task state: %v

What it means

During RecoverTask, the exec driver rebuilds a go-plugin ReattachConfig from the task state persisted in the client state DB and hands it to pstructs.ReattachConfigToGoPlugin. If that conversion fails, Nomad cannot reattach to the still-running executor process and the recovery is aborted. This typically means the persisted ReattachConfig is corrupt, incomplete, or from an incompatible plugin protocol version.

Source

Thrown at drivers/exec/driver.go:428

		d.logger.Trace("nothing to recover; task already exists",
			"task_id", handle.Config.ID,
			"task_name", handle.Config.Name,
		)
		return nil
	}

	// Handle doesn't already exist, try to reattach
	var taskState TaskState
	if err := handle.GetDriverState(&taskState); err != nil {
		d.logger.Error("failed to decode task state from handle", "error", err, "task_id", handle.Config.ID)
		return fmt.Errorf("failed to decode task state from handle: %v", err)
	}

	// Create client for reattached executor
	plugRC, err := pstructs.ReattachConfigToGoPlugin(taskState.ReattachConfig)
	if err != nil {
		d.logger.Error("failed to build ReattachConfig from task state", "error", err, "task_id", handle.Config.ID)
		return fmt.Errorf("failed to build ReattachConfig from task state: %v", err)
	}

	exec, pluginClient, err := executor.ReattachToExecutor(
		plugRC,
		d.logger.With("task_name", handle.Config.Name, "alloc_id", handle.Config.AllocID),
		d.compute,
	)
	if err != nil {
		d.logger.Error("failed to reattach to executor", "error", err, "task_id", handle.Config.ID)
		return fmt.Errorf("failed to reattach to executor: %v", err)
	}

	h := &taskHandle{
		exec:         exec,
		pid:          taskState.Pid,
		pluginClient: pluginClient,
		taskConfig:   taskState.TaskConfig,
		procState:    drivers.TaskStateRunning,

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Inspect the task's persisted state; if the executor process is gone or state is corrupt, stop the task and reschedule it rather than recovering.
2. Verify the client was not restored from a state directory of a different Nomad version; align versions or wipe the stale state for that allocation.
3. Check Nomad client logs for the underlying error from ReattachConfigToGoPlugin to confirm which field is invalid.
4. As a last resort, GC the allocation and let the scheduler place a new one.

// before relying on recovery, verify the persisted state is usable
if taskState.ReattachConfig == nil || taskState.ReattachConfig.Pid <= 0 {
    // treat as unrecoverable: stop + reschedule instead of RecoverTask
}

func reattachConfigValid(rc *plugin.ReattachConfig) bool {
    return rc != nil && rc.Protocol != "" && rc.Addr != nil && rc.Pid > 0
}

When it happens

Trigger: RecoverTask is called after a Nomad client restart with a task whose persisted drivers.TaskState contains a ReattachConfig that fails conversion: nil ReattachConfig, invalid Addr/Pid, or a protocol version go-plugin refuses to parse.

Common situations: Corrupted or truncated bolt state DB after a crash; upgrading Nomad across incompatible go-plugin protocol versions so old state entries no longer parse; manually edited or restored client state directory.

Related errors

• failed to decode task state from handle: %v
• failed to reattach to executor: %v
• default_pid_mode must be %q or %q, got %q
• default_ipc_mode must be %q or %q, got %q
• allow_caps configured with capabilities not supported by sys

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/015db1f51802e5c3. Report an issue: GitHub.