hashicorp/nomad · error

failed to decode taskConfig state from handle: %v

Error message

failed to decode taskConfig state from handle: %v

What it means

This error is returned by RecoverTask when handle.GetDriverState(&taskState) fails to decode the driver state that was previously stored in the task handle. The state contains the executor's reattach config and PID; without it the driver cannot reattach to the running task. Usually indicates corrupted, truncated, or version-incompatible state data.

Source

Thrown at drivers/java/driver.go:395

func (d *Driver) RecoverTask(handle *drivers.TaskHandle) error {
	if handle == nil {
		return fmt.Errorf("handle cannot be nil")
	}

	// If already attached to handle there's nothing to recover.
	if _, ok := d.tasks.Get(handle.Config.ID); ok {
		d.logger.Debug("nothing to recover; task already exists",
			"task_id", handle.Config.ID,
			"task_name", handle.Config.Name,
		)
		return nil
	}

	var taskState TaskState
	if err := handle.GetDriverState(&taskState); err != nil {
		d.logger.Error("failed to decode taskConfig state from handle", "error", err, "task_id", handle.Config.ID)
		return fmt.Errorf("failed to decode taskConfig state from handle: %v", err)
	}

	plugRC, err := pstructs.ReattachConfigToGoPlugin(taskState.ReattachConfig)
	if err != nil {
		d.logger.Error("failed to build ReattachConfig from taskConfig state", "error", err, "task_id", handle.Config.ID)
		return fmt.Errorf("failed to build ReattachConfig from taskConfig state: %v", err)
	}

	execImpl, pluginClient, err := executor.ReattachToExecutor(
		plugRC,
		d.logger.With("task_name", handle.Config.Name, "alloc_id", handle.Config.AllocID),
		d.nomadConfig.Topology.Compute(),
	)

	if err != nil {
		d.logger.Error("failed to reattach to executor", "error", err, "task_id", handle.Config.ID)
		return fmt.Errorf("failed to reattach to executor: %v", err)
	}

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Restore the client state from a good backup or let the task be stopped/restarted (nomad stop/alloc stop) to clear the bad state
2. Check Nomad upgrade notes; the task state may not be portable across versions - restart the affected allocations
3. Inspect/repair the client's data_dir state files
4. If reproducible across upgrades, report the incompatible TaskState change to Nomad maintainers

Example fix

// remediation: restart the affected allocation to rebuild clean state
// nomad alloc stop <alloc_id>

if err := driver.RecoverTask(handle); err != nil {
    if strings.Contains(err.Error(), "failed to decode taskConfig state from handle") {
        // state is unrecoverable: stop and restart the allocation
        _ = client.Allocations().Stop(ctx, alloc, nil)
    }
}

When it happens

Trigger: Nomad client restart recovery where the blob in the client state store cannot be gob/JSON-decoded into TaskState, e.g. after a Nomad version upgrade changed the struct, or the state file is corrupted/truncated.

Common situations: Upgrading Nomad across incompatible versions with running Java tasks; disk corruption or partial writes to the client data dir; state written by a different Nomad build.

Understand the failure class

• Parsing and encoding errors: unexpected token, malformed input — why parsers reject input and how to find the real culprit.

Related errors

• handle cannot be nil
• failed to build ReattachConfig from taskConfig state: %v
• failed to reattach to executor: %v
• unable to process ACLToken: %w
• default_pid_mode must be %q or %q, got %q

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/3a95c1c7b18d943c. Report an issue: GitHub.