{"record":{"id":"0633d3c0edcd355a","repo":"hashicorp/nomad","slug":"error-delting-unexpected-task-key-q-v","errorCode":null,"errorMessage":"error delting unexpected task key %q: %v","messagePattern":"error delting unexpected task key %q: (.+?)","errorType":"console","errorClass":null,"httpStatus":null,"severity":"error","filePath":"client/state/upgrade.go","lineNumber":283,"sourceCode":"\t\t\t// value is nil: delete unexpected bucket\n\t\t\tlogger.Warn(\"deleting unexpected task state bucket\",\n\t\t\t\t\"bucket\", string(k),\n\t\t\t)\n\n\t\t\tif err := bkt.DeleteBucket(k); err != nil {\n\t\t\t\treturn nil, fmt.Errorf(\"error deleting unexpected task bucket %q: %v\", string(k), err)\n\t\t\t}\n\t\t\tcontinue\n\t\t}\n\n\t\tif !bytes.Equal(k, []byte(\"simple-all\")) {\n\t\t\t// value is non-nil: delete unexpected entry\n\t\t\tlogger.Warn(\"deleting unexpected task state entry\",\n\t\t\t\t\"key\", string(k), \"value_bytes\", len(v),\n\t\t\t)\n\n\t\t\tif err := cur.Delete(); err != nil {\n\t\t\t\treturn nil, fmt.Errorf(\"error delting unexpected task key %q: %v\", string(k), err)\n\t\t\t}\n\t\t\tcontinue\n\t\t}\n\n\t\t// Decode simple-all\n\t\tsimpleFound = true\n\t\tif err := codec.NewDecoderBytes(v, structs.MsgpackHandle).Decode(&trState); err != nil {\n\t\t\treturn nil, fmt.Errorf(\"failed to decode task state from 'simple-all' entry: %v\", err)\n\t\t}\n\t}\n\n\tif !simpleFound {\n\t\treturn nil, fmt.Errorf(\"task state entry not found\")\n\t}\n\n\treturn &trState, nil\n}\n","sourceCodeStart":265,"sourceCodeEnd":301,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/client/state/upgrade.go#L265-L301","documentation":"During task bucket upgrade, any key with a non-nil value that is not 'simple-all' is unexpected legacy state and is removed via the cursor's Delete(). If that delete fails, migration of the task aborts with this (typo'd 'delting') wrapped error.","triggerScenarios":"UpgradeAllocs -> upgradeAllocBucket -> upgradeTaskBucket encounters a stray key/value entry in a 0.8 task bucket and bbolt cursor Delete() returns an error (e.g. bucket opened read-only or internal bolt error).","commonSituations":"Migrating clients whose 0.8 state contains extra keys from experimental or third-party builds; corrupted boltdb pages during upgrade.","solutions":["Back up and clean the client data_dir before re-running nomad so the stray keys are removed","Check the wrapped error to identify the underlying bbolt failure and repair the state file","Allow the client to rebuild task state from the servers after state cleanup","Escalate upstream with key name and wrapped error if it persists"],"exampleFix":null,"handlingStrategy":"try-catch","validationCode":null,"typeGuard":null,"tryCatchPattern":"// Go: log and recover operationally\nif err := upgradeState(); err != nil && strings.Contains(err.Error(), \"unexpected task key\") {\n    logger.Error(\"stray legacy state key blocked migration; restore backup or clean data_dir\", \"err\", err)\n}","preventionTips":["Do not run experimental/third-party Nomad binaries against production state","Keep pre-upgrade backups of client state","Use clean shutdown procedures for the agent","Verify boltdb integrity with a bolt inspection tool if corruption is suspected"],"tags":["nomad","boltdb","state-migration"],"backgroundTag":"boltdb-delete-failed","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}