hashicorp/nomad · error
failed to decode driver task state
Error message
failed to decode driver task state: %v
What it means
RecoverTask reads the driver-specific state stored on the TaskHandle via handle.GetDriverState into a taskHandleState struct. If the stored JSON cannot be decoded into that struct (corrupt blob, empty state, or schema drift), the recovery is aborted with this wrapped error.
Solutions
- Stop and reschedule the allocation so a fresh task handle state is written by StartTask.
- Confirm both clients run the same Nomad version if the state was migrated; align versions before recovery.
- Inspect the stored driver state blob in the client data_dir for truncation or corruption.
- Check driver upgrade notes for taskHandleState schema changes that make old state unrecoverable.
Example fix
// before: attempting recovery across a version upgrade $ nomad upgrade client 0.12 -> 1.x, then restart in-place allocation // after: reschedule so new state is written $ nomad alloc stop <alloc-id> # job reschedules and StartTask recreates state
Defensive patterns
Strategy: try-catch
Validate before calling
raw, err := handle.State()
if err != nil || len(raw) == 0 {
return fmt.Errorf("handle state empty/corrupt; reschedule required")
} Type guard
func isStateDecodeErr(err error) bool { return err != nil && strings.Contains(err.Error(), "failed to decode driver task state") } Try / catch
if err := driver.RecoverTask(handle); err != nil {
if isStateDecodeErr(err) {
logger.Warn("stored driver state undecodable; rescheduling allocation", "err", err)
// nomad alloc stop -> job reschedules with fresh state
} else {
return err
}
} Prevention
- Perform rolling upgrades so client and stored state versions stay compatible.
- Do not copy client data_dir/allocation state between hosts or versions.
- Use clean shutdown of nomad clients to avoid truncated state files.
- Back up and validate the client data_dir filesystem health.
When it happens
Trigger: RecoverTask is invoked for a handle whose driver state was written by an incompatible driver version (fields renamed/retyped), whose state blob is truncated/corrupted on disk, or whose GetDriverState returns an empty/nil payload.
Common situations: Nomad upgrade/downgrade causing taskHandleState schema mismatch; interrupted client shutdown leaving partial state on disk; manually copying allocations between clients; state directory disk corruption.
Understand the failure class
- Parsing and encoding errors: unexpected token, malformed input — why parsers reject input and how to find the real culprit.
Related errors
- cannot destroy running task
- Docker privileged mode is disabled on this Nomad agent
- executor: error waiting on process
- executor Shutdown failed
- failed to build mount for /etc/hosts
AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04).
Data as JSON: /api/errors/6b85a8a69e73d733.
Report an issue: GitHub.
Appendix: source
Thrown at drivers/docker/driver.go:250
TLSKey: d.config.TLS.Key,
TLSCA: d.config.TLS.CA,
StartTime: startTime.Unix(),
}); err != nil {
pluginClient.Kill()
return nil, nil, fmt.Errorf("failed to launch docker logger process %s: %v", container.Container.ID, err)
}
return dlogger, pluginClient, nil
}
func (d *Driver) RecoverTask(handle *drivers.TaskHandle) error {
if _, ok := d.tasks.Get(handle.Config.ID); ok {
return nil
}
var handleState taskHandleState
if err := handle.GetDriverState(&handleState); err != nil {
return fmt.Errorf("failed to decode driver task state: %v", err)
}
dockerClient, err := d.getDockerClient()
if err != nil {
return fmt.Errorf("failed to get docker client: %w", err)
}
dockerInfo, err := dockerClient.Info(d.ctx, mclient.InfoOptions{})
if err != nil {
return fmt.Errorf("failed to fetch docker daemon info: %v", err)
}
infinityClient, err := d.getInfinityClient()
if err != nil {
return fmt.Errorf("failed to get docker long operations client: %w", err)
}
container, err := dockerClient.ContainerInspect(d.ctx, handleState.ContainerID, mclient.ContainerInspectOptions{})View on GitHub (pinned to 482b49bf1a)