hashicorp/nomad · error

failed to decode task state from handle: %v

Error message

failed to decode task state from handle: %v

What it means

When reattaching to a task after agent restart, RecoverTask calls handle.GetDriverState(&taskState) to decode the previously persisted TaskState (including the executor's ReattachConfig). If decoding fails (missing, corrupt, or version-incompatible state blob), the driver logs and returns this error, abandoning recovery.

Source

Thrown at drivers/exec/driver.go:421

func (d *Driver) RecoverTask(handle *drivers.TaskHandle) error {
	if handle == nil {
		return fmt.Errorf("handle cannot be nil")
	}

	// If already attached to handle there's nothing to recover.
	if _, ok := d.tasks.Get(handle.Config.ID); ok {
		d.logger.Trace("nothing to recover; task already exists",
			"task_id", handle.Config.ID,
			"task_name", handle.Config.Name,
		)
		return nil
	}

	// Handle doesn't already exist, try to reattach
	var taskState TaskState
	if err := handle.GetDriverState(&taskState); err != nil {
		d.logger.Error("failed to decode task state from handle", "error", err, "task_id", handle.Config.ID)
		return fmt.Errorf("failed to decode task state from handle: %v", err)
	}

	// Create client for reattached executor
	plugRC, err := pstructs.ReattachConfigToGoPlugin(taskState.ReattachConfig)
	if err != nil {
		d.logger.Error("failed to build ReattachConfig from task state", "error", err, "task_id", handle.Config.ID)
		return fmt.Errorf("failed to build ReattachConfig from task state: %v", err)
	}

	exec, pluginClient, err := executor.ReattachToExecutor(
		plugRC,
		d.logger.With("task_name", handle.Config.Name, "alloc_id", handle.Config.AllocID),
		d.compute,
	)
	if err != nil {
		d.logger.Error("failed to reattach to executor", "error", err, "task_id", handle.Config.ID)
		return fmt.Errorf("failed to reattach to executor: %v", err)
	}

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Let Nomad GC/stop the orphaned task and re-run it rather than recovering the stale handle
2. Check Nomad version compatibility between the writer and reader of the state; align client versions
3. Inspect the wrapped error for JSON/gob decode details and clear corrupted client state

var probe drivers.TaskState
if err := handle.GetDriverState(&probe); err != nil {
  return fmt.Errorf("handle state unusable, skip recovery: %w", err)
}

func decodableHandle(h *drivers.TaskHandle) bool {
  var st TaskState
  return h != nil && h.GetDriverState(&st) == nil
}

if err := driver.RecoverTask(handle); err != nil {
  if strings.Contains(err.Error(), "failed to decode task state") {
    // destroy orphaned task and reschedule instead of retrying recovery
    return rescheduleTask(handle.Config.ID)
  }
  return err
}

When it happens

Trigger: Agent restart recovery where the handle's stored driver state was written by a different Nomad version, is truncated/corrupt, or contains data that no longer unmarshals into TaskState.

Common situations: Nomad client upgrades/rollbacks changing TaskState schema; corrupted plugin state files; manually copied data dirs; interrupted writes during crashes.

Understand the failure class

• Parsing and encoding errors: unexpected token, malformed input — why parsers reject input and how to find the real culprit.

Related errors

• failed to build ReattachConfig from task state: %v
• failed to reattach to executor: %v
• default_pid_mode must be %q or %q, got %q
• default_ipc_mode must be %q or %q, got %q
• allow_caps configured with capabilities not supported by sys

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/94b4d1cfde00da24. Report an issue: GitHub.