hashicorp/nomad · critical

failed to decode task state from handle: %v

Error message

failed to decode task state from handle: %v

What it means

Nomad's rawexec driver throws this in RecoverTask when the driver state persisted in the task handle cannot be JSON-decoded back into a TaskState struct. RecoverTask runs on client restart to re-adopt tasks, so this means the handle blob in the client state store is corrupt, empty, or was written by an incompatible version. The task cannot be recovered and the client will treat it as failed.

Source

Thrown at drivers/rawexec/driver.go:332

func (d *Driver) RecoverTask(handle *drivers.TaskHandle) error {
	if handle == nil {
		return fmt.Errorf("handle cannot be nil")
	}

	// If already attached to handle there's nothing to recover.
	if _, ok := d.tasks.Get(handle.Config.ID); ok {
		d.logger.Trace("nothing to recover; task already exists",
			"task_id", handle.Config.ID,
			"task_name", handle.Config.Name,
		)
		return nil
	}

	// Handle doesn't already exist, try to reattach
	var taskState TaskState
	if err := handle.GetDriverState(&taskState); err != nil {
		d.logger.Error("failed to decode task state from handle", "error", err, "task_id", handle.Config.ID)
		return fmt.Errorf("failed to decode task state from handle: %v", err)
	}

	plugRC, err := pstructs.ReattachConfigToGoPlugin(taskState.ReattachConfig)
	if err != nil {
		d.logger.Error("failed to build ReattachConfig from task state", "error", err, "task_id", handle.Config.ID)
		return fmt.Errorf("failed to build ReattachConfig from task state: %v", err)
	}

	// Create client for reattached executor
	exec, pluginClient, err := executor.ReattachToExecutor(
		plugRC,
		d.logger.With("task_name", handle.Config.Name, "alloc_id", handle.Config.AllocID),
		d.compute,
	)
	if err != nil {
		d.logger.Error("failed to reattach to executor", "error", err, "task_id", handle.Config.ID)
		return fmt.Errorf("failed to reattach to executor: %v", err)
	}

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Verify disk integrity and free space on the Nomad client data_dir; a truncated state file is the usual cause
2. Stop nomad, remove the corrupt task handle from <data_dir>/client (or the alloc dir) so the task is re-run instead of recovered, then restart the client
3. Check for a Nomad version mismatch (downgrade/restore from another version) and align versions before restarting
4. If recovery keeps failing, stop the affected allocation (nomad alloc stop) to let the scheduler place a fresh one

// before restart, sanity-check client state
cd <data_dir>/client && find . -name '*.json' | xargs -I{} sh -c 'jq empty {} || echo CORRUPT: {}'

// after GetDriverState, verify decoded state is usable
if taskState.ReattachConfig == nil || taskState.Pid <= 0 {
  // treat as unrecoverable; reschedule allocation
}

// Go: caller of RecoverTask (client hook)
if err := d.RecoverTask(handle); err != nil {
  if strings.Contains(err.Error(), "failed to decode task state") {
    d.logger.Warn("handle state corrupt; falling back to fresh start", "task", handle.Config.ID)
    // drop handle so task is re-run
  }
}

When it happens

Trigger: Client daemon restarts and calls Driver.RecoverTask for a task handle whose GetDriverState payload fails to unmarshal into drivers/rawexec TaskState (corrupt state store, truncated file, schema drift between Nomad versions, or a handle not created by this driver).

Common situations: Upgrading Nomad across versions where TaskState fields changed; disk corruption or full-disk truncation of the client data_dir; manually editing/copying state files; restoring a data_dir from a mismatched snapshot.

Understand the failure class

• Parsing and encoding errors: unexpected token, malformed input — why parsers reject input and how to find the real culprit.

Related errors

• failed to decode task state from handle: %v
• failed to build ReattachConfig from task state: %v
• only one of cgroups_v1_override and cgroups_v2_override may
• work_dir must be an absolute path
• unable to process ACLToken: %w

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/8e1c7f1bb6b60438. Report an issue: GitHub.