{"record":{"id":"97a71c6771f9bb44","repo":"hashicorp/nomad","slug":"error-deleting-invalid-task-state-for-task-q-v","errorCode":null,"errorMessage":"error deleting invalid task state for task %q: %v","messagePattern":"error deleting invalid task state for task %q: (.+?)","errorType":"console","errorClass":null,"httpStatus":null,"severity":"error","filePath":"client/state/upgrade.go","lineNumber":216,"sourceCode":"\t\ttaskName := string(taskBucket)\n\t\ttaskLogger := logger.With(\"task_name\", taskName)\n\n\t\ttaskBkt := bkt.Bucket(taskBucket)\n\t\tif taskBkt == nil {\n\t\t\t// This should never happen as we just read the bucket.\n\t\t\treturn fmt.Errorf(\"unexpected bucket missing %q\", taskName)\n\t\t}\n\n\t\toldState, err := upgradeTaskBucket(taskLogger, taskBkt)\n\t\tif err != nil {\n\t\t\ttaskLogger.Warn(\"dropping invalid task due to error while upgrading state\",\n\t\t\t\t\"error\", err,\n\t\t\t)\n\n\t\t\t// Delete the invalid task bucket and treat failures\n\t\t\t// here as unrecoverable errors.\n\t\t\tif err := bkt.DeleteBucket(taskBucket); err != nil {\n\t\t\t\treturn fmt.Errorf(\"error deleting invalid task state for task %q: %v\",\n\t\t\t\t\ttaskName, err,\n\t\t\t\t)\n\t\t\t}\n\t\t\tcontinue\n\t\t}\n\n\t\t// Convert 0.8 task state to 0.9 task state\n\t\tlocalTaskState, err := oldState.Upgrade(allocID, taskName)\n\t\tif err != nil {\n\t\t\ttaskLogger.Warn(\"dropping invalid task due to error while upgrading state\",\n\t\t\t\t\"error\", err,\n\t\t\t)\n\n\t\t\t// Delete the invalid task bucket and treat failures\n\t\t\t// here as unrecoverable errors.\n\t\t\tif err := bkt.DeleteBucket(taskBucket); err != nil {\n\t\t\t\treturn fmt.Errorf(\"error deleting invalid task state for task %q: %v\",\n\t\t\t\t\ttaskName, err,","sourceCodeStart":198,"sourceCodeEnd":234,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/client/state/upgrade.go#L198-L234","documentation":"During Nomad client state migration from 0.8 to 0.9+ (boltdb layout), upgradeAllocBucket attempts to delete a task bucket whose state could not be upgraded, treating it as droppable invalid state. If boltdd's DeleteBucket itself fails, the migration aborts with this unrecoverable error so the transaction is not left half-applied.","triggerScenarios":"UpgradeAllocs iterates an alloc bucket and finds a task sub-bucket whose 'simple-all' entry is corrupt/undecodable (upgradeTaskBucket error) or whose state.Upgrade() fails; the subsequent bkt.DeleteBucket(taskBucket) then returns an error (e.g. key no longer resolves to a bucket, bucket already deleted, or boltdd/internal bbolt error).","commonSituations":"Upgrading a Nomad client from 0.8.x whose data dir contains corrupted or partially-written task state (crash during prior write); running the migration twice against a modified/rolled-back boltdb file; disk-level corruption of client state.","solutions":["Back up the client data_dir, then stop nomad and remove or restore the corrupted alloc state so migration can proceed","Inspect the wrapped %v error from boltdd to determine whether the bucket is missing/corrupt and repair or delete that alloc's state directory","Re-run nomad; invalid task state is intentionally dropped, so a clean data dir lets the agent re-sync from servers","Report persistent failures upstream with the wrapped error and alloc/task names"],"exampleFix":"// before (migration aborts on delete failure)\nif err := bkt.DeleteBucket(taskBucket); err != nil {\n    return fmt.Errorf(\"error deleting invalid task state for task %q: %v\", taskName, err)\n}\n// after (operational recovery, not a code change: restore/clean client state dir)\n// nomad agent stop && mv <data_dir>/client <data_dir>/client.bak && nomad agent start","handlingStrategy":"try-catch","validationCode":"// Go: before upgrading, verify state file integrity and backup\nif _, err := os.Stat(filepath.Join(dataDir, \"client\", \"state.db\")); err != nil {\n    return fmt.Errorf(\"client state missing, nothing to migrate: %w\", err)\n}\nif err := copyFile(stateDB, stateDB+\".pre-upgrade.bak\"); err != nil {\n    return err\n}","typeGuard":"// Go: ensure the key actually resolves to a bucket before deleting\nif bkt.Bucket(taskBucket) == nil {\n    return fmt.Errorf(\"task bucket %q is not a bucket; skipping delete\", string(taskBucket))\n}","tryCatchPattern":"// Go: wrap and surface the wrapped cause for operators\nerr := upgradeAllocs()\nif err != nil && strings.Contains(err.Error(), \"error deleting invalid task state\") {\n    logger.Error(\"state migration aborted; restore client state backup or wipe data_dir\", \"err\", err)\n}","preventionTips":["Always back up data_dir/client before upgrading Nomad versions","Upgrade through the documented version path instead of hand-editing state","Check disk health (fsck/SMART) if state corruption recurs","Run nomad with the same data_dir and never share it between concurrent agents"],"tags":["nomad","boltdb","state-migration","client"],"backgroundTag":"boltdb-bucket-delete-failed","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}