{"record":{"id":"94b4d1cfde00da24","repo":"hashicorp/nomad","slug":"failed-to-decode-task-state-from-handle-v","errorCode":null,"errorMessage":"failed to decode task state from handle: %v","messagePattern":"failed to decode task state from handle: (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"drivers/exec/driver.go","lineNumber":421,"sourceCode":"func (d *Driver) RecoverTask(handle *drivers.TaskHandle) error {\n\tif handle == nil {\n\t\treturn fmt.Errorf(\"handle cannot be nil\")\n\t}\n\n\t// If already attached to handle there's nothing to recover.\n\tif _, ok := d.tasks.Get(handle.Config.ID); ok {\n\t\td.logger.Trace(\"nothing to recover; task already exists\",\n\t\t\t\"task_id\", handle.Config.ID,\n\t\t\t\"task_name\", handle.Config.Name,\n\t\t)\n\t\treturn nil\n\t}\n\n\t// Handle doesn't already exist, try to reattach\n\tvar taskState TaskState\n\tif err := handle.GetDriverState(&taskState); err != nil {\n\t\td.logger.Error(\"failed to decode task state from handle\", \"error\", err, \"task_id\", handle.Config.ID)\n\t\treturn fmt.Errorf(\"failed to decode task state from handle: %v\", err)\n\t}\n\n\t// Create client for reattached executor\n\tplugRC, err := pstructs.ReattachConfigToGoPlugin(taskState.ReattachConfig)\n\tif err != nil {\n\t\td.logger.Error(\"failed to build ReattachConfig from task state\", \"error\", err, \"task_id\", handle.Config.ID)\n\t\treturn fmt.Errorf(\"failed to build ReattachConfig from task state: %v\", err)\n\t}\n\n\texec, pluginClient, err := executor.ReattachToExecutor(\n\t\tplugRC,\n\t\td.logger.With(\"task_name\", handle.Config.Name, \"alloc_id\", handle.Config.AllocID),\n\t\td.compute,\n\t)\n\tif err != nil {\n\t\td.logger.Error(\"failed to reattach to executor\", \"error\", err, \"task_id\", handle.Config.ID)\n\t\treturn fmt.Errorf(\"failed to reattach to executor: %v\", err)\n\t}","sourceCodeStart":403,"sourceCodeEnd":439,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/drivers/exec/driver.go#L403-L439","documentation":"When reattaching to a task after agent restart, RecoverTask calls handle.GetDriverState(&taskState) to decode the previously persisted TaskState (including the executor's ReattachConfig). If decoding fails (missing, corrupt, or version-incompatible state blob), the driver logs and returns this error, abandoning recovery.","triggerScenarios":"Agent restart recovery where the handle's stored driver state was written by a different Nomad version, is truncated/corrupt, or contains data that no longer unmarshals into TaskState.","commonSituations":"Nomad client upgrades/rollbacks changing TaskState schema; corrupted plugin state files; manually copied data dirs; interrupted writes during crashes.","solutions":["Let Nomad GC/stop the orphaned task and re-run it rather than recovering the stale handle","Check Nomad version compatibility between the writer and reader of the state; align client versions","Inspect the wrapped error for JSON/gob decode details and clear corrupted client state"],"exampleFix":null,"handlingStrategy":"try-catch","validationCode":"var probe drivers.TaskState\nif err := handle.GetDriverState(&probe); err != nil {\n  return fmt.Errorf(\"handle state unusable, skip recovery: %w\", err)\n}","typeGuard":"func decodableHandle(h *drivers.TaskHandle) bool {\n  var st TaskState\n  return h != nil && h.GetDriverState(&st) == nil\n}","tryCatchPattern":"if err := driver.RecoverTask(handle); err != nil {\n  if strings.Contains(err.Error(), \"failed to decode task state\") {\n    // destroy orphaned task and reschedule instead of retrying recovery\n    return rescheduleTask(handle.Config.ID)\n  }\n  return err\n}","preventionTips":["Avoid rolling Nomad clients back across TaskState schema changes; stop tasks first","Ensure client data dirs aren't manually copied or truncated","Monitor agent crashes mid-write and GC orphaned allocs after incidents"],"tags":["nomad","exec-driver","task-recovery","state-deserialization"],"backgroundTag":"task-state-decode-failed","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}