hashicorp/nomad · error

failed to decode driver task state: %v

Error message

failed to decode driver task state: %v

What it means

RecoverTask reads the driver-specific state stored on the TaskHandle via handle.GetDriverState into a taskHandleState struct. If the stored JSON cannot be decoded into that struct (corrupt blob, empty state, or schema drift), the recovery is aborted with this wrapped error.

Source

Thrown at drivers/docker/driver.go:250

		TLSKey:      d.config.TLS.Key,
		TLSCA:       d.config.TLS.CA,
		StartTime:   startTime.Unix(),
	}); err != nil {
		pluginClient.Kill()
		return nil, nil, fmt.Errorf("failed to launch docker logger process %s: %v", container.Container.ID, err)
	}

	return dlogger, pluginClient, nil
}

func (d *Driver) RecoverTask(handle *drivers.TaskHandle) error {
	if _, ok := d.tasks.Get(handle.Config.ID); ok {
		return nil
	}

	var handleState taskHandleState
	if err := handle.GetDriverState(&handleState); err != nil {
		return fmt.Errorf("failed to decode driver task state: %v", err)
	}

	dockerClient, err := d.getDockerClient()
	if err != nil {
		return fmt.Errorf("failed to get docker client: %w", err)
	}

	dockerInfo, err := dockerClient.Info(d.ctx, mclient.InfoOptions{})
	if err != nil {
		return fmt.Errorf("failed to fetch docker daemon info: %v", err)
	}

	infinityClient, err := d.getInfinityClient()
	if err != nil {
		return fmt.Errorf("failed to get docker long operations client: %w", err)
	}

	container, err := dockerClient.ContainerInspect(d.ctx, handleState.ContainerID, mclient.ContainerInspectOptions{})

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Stop and reschedule the allocation so a fresh task handle state is written by StartTask.
2. Confirm both clients run the same Nomad version if the state was migrated; align versions before recovery.
3. Inspect the stored driver state blob in the client data_dir for truncation or corruption.
4. Check driver upgrade notes for taskHandleState schema changes that make old state unrecoverable.

Example fix

// before: attempting recovery across a version upgrade
$ nomad upgrade client 0.12 -> 1.x, then restart in-place allocation
// after: reschedule so new state is written
$ nomad alloc stop <alloc-id>  # job reschedules and StartTask recreates state

raw, err := handle.State()
if err != nil || len(raw) == 0 {
    return fmt.Errorf("handle state empty/corrupt; reschedule required")
}

func isStateDecodeErr(err error) bool { return err != nil && strings.Contains(err.Error(), "failed to decode driver task state") }

if err := driver.RecoverTask(handle); err != nil {
    if isStateDecodeErr(err) {
        logger.Warn("stored driver state undecodable; rescheduling allocation", "err", err)
        // nomad alloc stop -> job reschedules with fresh state
    } else {
        return err
    }
}

When it happens

Trigger: RecoverTask is invoked for a handle whose driver state was written by an incompatible driver version (fields renamed/retyped), whose state blob is truncated/corrupted on disk, or whose GetDriverState returns an empty/nil payload.

Common situations: Nomad upgrade/downgrade causing taskHandleState schema mismatch; interrupted client shutdown leaving partial state on disk; manually copying allocations between clients; state directory disk corruption.

Understand the failure class

• Parsing and encoding errors: unexpected token, malformed input — why parsers reject input and how to find the real culprit.

Related errors

• QEMU Guest Agent socket is unsupported on the Windows platfo
• monitorPath not set
• unable to unmarshal ACLToken: %w
• failed to reattach to docker logger process: %v
• failed to launch docker logger plugin: %v

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/6b85a8a69e73d733. Report an issue: GitHub.