hashicorp/nomad · critical

failed to build ReattachConfig from task state: %v

Error message

failed to build ReattachConfig from task state: %v

What it means

RecoverTask failed while converting the persisted plugin ReattachConfig (gRPC address/pid) into a go-plugin client config via pstructs.ReattachConfigToGoPlugin. The stored ReattachConfig is missing required fields (e.g. empty address or invalid protocol), so the driver cannot even attempt to reconnect to the executor plugin.

Source

Thrown at drivers/rawexec/driver.go:338

	if _, ok := d.tasks.Get(handle.Config.ID); ok {
		d.logger.Trace("nothing to recover; task already exists",
			"task_id", handle.Config.ID,
			"task_name", handle.Config.Name,
		)
		return nil
	}

	// Handle doesn't already exist, try to reattach
	var taskState TaskState
	if err := handle.GetDriverState(&taskState); err != nil {
		d.logger.Error("failed to decode task state from handle", "error", err, "task_id", handle.Config.ID)
		return fmt.Errorf("failed to decode task state from handle: %v", err)
	}

	plugRC, err := pstructs.ReattachConfigToGoPlugin(taskState.ReattachConfig)
	if err != nil {
		d.logger.Error("failed to build ReattachConfig from task state", "error", err, "task_id", handle.Config.ID)
		return fmt.Errorf("failed to build ReattachConfig from task state: %v", err)
	}

	// Create client for reattached executor
	exec, pluginClient, err := executor.ReattachToExecutor(
		plugRC,
		d.logger.With("task_name", handle.Config.Name, "alloc_id", handle.Config.AllocID),
		d.compute,
	)
	if err != nil {
		d.logger.Error("failed to reattach to executor", "error", err, "task_id", handle.Config.ID)
		return fmt.Errorf("failed to reattach to executor: %v", err)
	}

	h := &taskHandle{
		exec:         exec,
		pid:          taskState.Pid,
		pluginClient: pluginClient,
		taskConfig:   taskState.TaskConfig,

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Check Nomad version consistency across the cluster; reattach config formats changed between releases
2. Stop and reschedule the allocation (nomad alloc stop <alloc_id>) so a new executor is launched
3. Inspect the handle state in the client data_dir for empty/invalid reattach fields; treat as corrupt and clear it
4. Restart the nomad client after cleaning the corrupt state entry

// validate reattach config before conversion
func validReattach(rc *pstructs.ReattachConfig) bool {
  return rc != nil && rc.Pid > 0 && rc.Addr != nil && rc.Addr.String() != "" && rc.Protocol != ""
}

if err := d.RecoverTask(handle); err != nil {
  if strings.Contains(err.Error(), "failed to build ReattachConfig") {
    // abandon recovery; let scheduler place a new allocation
    return fmt.Errorf("unrecoverable handle: %w", err)
  }
}

When it happens

Trigger: Calling RecoverTask where taskState.ReattachConfig decoded but is invalid: empty Addr/Protocol, nil ReattachConfig, or a config written in a format the current Nomad version cannot convert.

Common situations: Version skew after upgrade/downgrade of Nomad clients; partially written driver state after a crash; tasks whose executor plugin state was never fully persisted.

Related errors

• failed to reattach to executor: %v
• failed to decode task state from handle: %v
• failed to create executor: %v
• plugin not found
• plugin not executable

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/c4ff2b0a8212ccdb. Report an issue: GitHub.