{"record":{"id":"6b85a8a69e73d733","repo":"hashicorp/nomad","slug":"failed-to-decode-driver-task-state-v","errorCode":null,"errorMessage":"failed to decode driver task state: %v","messagePattern":"failed to decode driver task state: (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"drivers/docker/driver.go","lineNumber":250,"sourceCode":"\t\tTLSKey:      d.config.TLS.Key,\n\t\tTLSCA:       d.config.TLS.CA,\n\t\tStartTime:   startTime.Unix(),\n\t}); err != nil {\n\t\tpluginClient.Kill()\n\t\treturn nil, nil, fmt.Errorf(\"failed to launch docker logger process %s: %v\", container.Container.ID, err)\n\t}\n\n\treturn dlogger, pluginClient, nil\n}\n\nfunc (d *Driver) RecoverTask(handle *drivers.TaskHandle) error {\n\tif _, ok := d.tasks.Get(handle.Config.ID); ok {\n\t\treturn nil\n\t}\n\n\tvar handleState taskHandleState\n\tif err := handle.GetDriverState(&handleState); err != nil {\n\t\treturn fmt.Errorf(\"failed to decode driver task state: %v\", err)\n\t}\n\n\tdockerClient, err := d.getDockerClient()\n\tif err != nil {\n\t\treturn fmt.Errorf(\"failed to get docker client: %w\", err)\n\t}\n\n\tdockerInfo, err := dockerClient.Info(d.ctx, mclient.InfoOptions{})\n\tif err != nil {\n\t\treturn fmt.Errorf(\"failed to fetch docker daemon info: %v\", err)\n\t}\n\n\tinfinityClient, err := d.getInfinityClient()\n\tif err != nil {\n\t\treturn fmt.Errorf(\"failed to get docker long operations client: %w\", err)\n\t}\n\n\tcontainer, err := dockerClient.ContainerInspect(d.ctx, handleState.ContainerID, mclient.ContainerInspectOptions{})","sourceCodeStart":232,"sourceCodeEnd":268,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/drivers/docker/driver.go#L232-L268","documentation":"RecoverTask reads the driver-specific state stored on the TaskHandle via handle.GetDriverState into a taskHandleState struct. If the stored JSON cannot be decoded into that struct (corrupt blob, empty state, or schema drift), the recovery is aborted with this wrapped error.","triggerScenarios":"RecoverTask is invoked for a handle whose driver state was written by an incompatible driver version (fields renamed/retyped), whose state blob is truncated/corrupted on disk, or whose GetDriverState returns an empty/nil payload.","commonSituations":"Nomad upgrade/downgrade causing taskHandleState schema mismatch; interrupted client shutdown leaving partial state on disk; manually copying allocations between clients; state directory disk corruption.","solutions":["Stop and reschedule the allocation so a fresh task handle state is written by StartTask.","Confirm both clients run the same Nomad version if the state was migrated; align versions before recovery.","Inspect the stored driver state blob in the client data_dir for truncation or corruption.","Check driver upgrade notes for taskHandleState schema changes that make old state unrecoverable."],"exampleFix":"// before: attempting recovery across a version upgrade\n$ nomad upgrade client 0.12 -> 1.x, then restart in-place allocation\n// after: reschedule so new state is written\n$ nomad alloc stop <alloc-id>  # job reschedules and StartTask recreates state","handlingStrategy":"try-catch","validationCode":"raw, err := handle.State()\nif err != nil || len(raw) == 0 {\n    return fmt.Errorf(\"handle state empty/corrupt; reschedule required\")\n}","typeGuard":"func isStateDecodeErr(err error) bool { return err != nil && strings.Contains(err.Error(), \"failed to decode driver task state\") }","tryCatchPattern":"if err := driver.RecoverTask(handle); err != nil {\n    if isStateDecodeErr(err) {\n        logger.Warn(\"stored driver state undecodable; rescheduling allocation\", \"err\", err)\n        // nomad alloc stop -> job reschedules with fresh state\n    } else {\n        return err\n    }\n}","preventionTips":["Perform rolling upgrades so client and stored state versions stay compatible.","Do not copy client data_dir/allocation state between hosts or versions.","Use clean shutdown of nomad clients to avoid truncated state files.","Back up and validate the client data_dir filesystem health."],"tags":["state-decode","task-recovery","json-unmarshal","nomad-driver"],"backgroundTag":"state-decode-failed","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}