{"record":{"id":"8e1c7f1bb6b60438","repo":"hashicorp/nomad","slug":"failed-to-decode-task-state-from-handle-v-8e1c7f","errorCode":null,"errorMessage":"failed to decode task state from handle: %v","messagePattern":"failed to decode task state from handle: (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"critical","filePath":"drivers/rawexec/driver.go","lineNumber":332,"sourceCode":"func (d *Driver) RecoverTask(handle *drivers.TaskHandle) error {\n\tif handle == nil {\n\t\treturn fmt.Errorf(\"handle cannot be nil\")\n\t}\n\n\t// If already attached to handle there's nothing to recover.\n\tif _, ok := d.tasks.Get(handle.Config.ID); ok {\n\t\td.logger.Trace(\"nothing to recover; task already exists\",\n\t\t\t\"task_id\", handle.Config.ID,\n\t\t\t\"task_name\", handle.Config.Name,\n\t\t)\n\t\treturn nil\n\t}\n\n\t// Handle doesn't already exist, try to reattach\n\tvar taskState TaskState\n\tif err := handle.GetDriverState(&taskState); err != nil {\n\t\td.logger.Error(\"failed to decode task state from handle\", \"error\", err, \"task_id\", handle.Config.ID)\n\t\treturn fmt.Errorf(\"failed to decode task state from handle: %v\", err)\n\t}\n\n\tplugRC, err := pstructs.ReattachConfigToGoPlugin(taskState.ReattachConfig)\n\tif err != nil {\n\t\td.logger.Error(\"failed to build ReattachConfig from task state\", \"error\", err, \"task_id\", handle.Config.ID)\n\t\treturn fmt.Errorf(\"failed to build ReattachConfig from task state: %v\", err)\n\t}\n\n\t// Create client for reattached executor\n\texec, pluginClient, err := executor.ReattachToExecutor(\n\t\tplugRC,\n\t\td.logger.With(\"task_name\", handle.Config.Name, \"alloc_id\", handle.Config.AllocID),\n\t\td.compute,\n\t)\n\tif err != nil {\n\t\td.logger.Error(\"failed to reattach to executor\", \"error\", err, \"task_id\", handle.Config.ID)\n\t\treturn fmt.Errorf(\"failed to reattach to executor: %v\", err)\n\t}","sourceCodeStart":314,"sourceCodeEnd":350,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/drivers/rawexec/driver.go#L314-L350","documentation":"Nomad's rawexec driver throws this in RecoverTask when the driver state persisted in the task handle cannot be JSON-decoded back into a TaskState struct. RecoverTask runs on client restart to re-adopt tasks, so this means the handle blob in the client state store is corrupt, empty, or was written by an incompatible version. The task cannot be recovered and the client will treat it as failed.","triggerScenarios":"Client daemon restarts and calls Driver.RecoverTask for a task handle whose GetDriverState payload fails to unmarshal into drivers/rawexec TaskState (corrupt state store, truncated file, schema drift between Nomad versions, or a handle not created by this driver).","commonSituations":"Upgrading Nomad across versions where TaskState fields changed; disk corruption or full-disk truncation of the client data_dir; manually editing/copying state files; restoring a data_dir from a mismatched snapshot.","solutions":["Verify disk integrity and free space on the Nomad client data_dir; a truncated state file is the usual cause","Stop nomad, remove the corrupt task handle from <data_dir>/client (or the alloc dir) so the task is re-run instead of recovered, then restart the client","Check for a Nomad version mismatch (downgrade/restore from another version) and align versions before restarting","If recovery keeps failing, stop the affected allocation (nomad alloc stop) to let the scheduler place a fresh one"],"exampleFix":null,"handlingStrategy":"fallback","validationCode":"// before restart, sanity-check client state\ncd <data_dir>/client && find . -name '*.json' | xargs -I{} sh -c 'jq empty {} || echo CORRUPT: {}'","typeGuard":"// after GetDriverState, verify decoded state is usable\nif taskState.ReattachConfig == nil || taskState.Pid <= 0 {\n  // treat as unrecoverable; reschedule allocation\n}","tryCatchPattern":"// Go: caller of RecoverTask (client hook)\nif err := d.RecoverTask(handle); err != nil {\n  if strings.Contains(err.Error(), \"failed to decode task state\") {\n    d.logger.Warn(\"handle state corrupt; falling back to fresh start\", \"task\", handle.Config.ID)\n    // drop handle so task is re-run\n  }\n}","preventionTips":["Do not hand-edit or partially copy the Nomad data_dir","Keep Nomad versions consistent across upgrades (drain clients first)","Monitor disk health and free space on clients","Run nomad operator raft / client state checks after unclean shutdowns"],"tags":["nomad","rawexec","state-corruption","task-recovery"],"backgroundTag":"driver-state-decode-failed","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}