{"record":{"id":"d08c506c9d5c6c68","repo":"hashicorp/nomad","slug":"error-deleting-unexpected-key-q-v","errorCode":null,"errorMessage":"error deleting unexpected key %q: %v","messagePattern":"error deleting unexpected key %q: (.+?)","errorType":"error_code","errorClass":null,"httpStatus":null,"severity":"error","filePath":"client/state/upgrade.go","lineNumber":114,"sourceCode":"func UpgradeAllocs(logger hclog.Logger, tx *boltdd.Tx) error {\n\tbtx := tx.BoltTx()\n\tallocationsBucket := btx.Bucket(allocationsBucketName)\n\tif allocationsBucket == nil {\n\t\t// No state!\n\t\treturn nil\n\t}\n\n\t// Gather alloc buckets and remove unexpected key/value pairs\n\tallocBuckets := [][]byte{}\n\tcur := allocationsBucket.Cursor()\n\tfor k, v := cur.First(); k != nil; k, v = cur.Next() {\n\t\tif v != nil {\n\t\t\tlogger.Warn(\"deleting unexpected key in state db\",\n\t\t\t\t\"key\", string(k), \"value_bytes\", len(v),\n\t\t\t)\n\n\t\t\tif err := cur.Delete(); err != nil {\n\t\t\t\treturn fmt.Errorf(\"error deleting unexpected key %q: %v\", string(k), err)\n\t\t\t}\n\t\t\tcontinue\n\t\t}\n\n\t\tallocBuckets = append(allocBuckets, k)\n\t}\n\n\tfor _, allocBucket := range allocBuckets {\n\t\tallocID := string(allocBucket)\n\n\t\tbkt := allocationsBucket.Bucket(allocBucket)\n\t\tif bkt == nil {\n\t\t\t// This should never happen as we just read the bucket.\n\t\t\treturn fmt.Errorf(\"unexpected bucket missing %q\", allocID)\n\t\t}\n\n\t\tallocLogger := logger.With(\"alloc_id\", allocID)\n\t\tif err := upgradeAllocBucket(allocLogger, tx, bkt, allocID); err != nil {","sourceCodeStart":96,"sourceCodeEnd":132,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/client/state/upgrade.go#L96-L132","documentation":"While iterating the allocations bucket to upgrade alloc state, any key that is not itself an alloc bucket (unexpected value key) is logged and deleted; this error is returned if that delete fails, aborting the upgrade transaction. It exists so the upgrade never silently leaves garbage keys in the allocations bucket.","triggerScenarios":"cur.Delete() fails on an unexpected key inside the allocations bucket — typically a bolt write error (read-only tx, disk full, page corruption) during UpgradeAllocs.","commonSituations":"Disk full during a Nomad client upgrade, corrupted state.db pages, or bolt transaction constraints after earlier IO errors.","solutions":["Free disk space on the state_dir volume and retry the upgrade.","Check for I/O errors (dmesg) and filesystem corruption; run fsck if needed.","Restore state.db from state.db.backup and re-run the upgrade on healthy storage.","If corruption persists, move state.db aside and let the client re-register allocations."],"exampleFix":"// before\nupgrade aborts: error deleting unexpected key \"stale\": errno 28\n// after\ndf -h /var/lib/nomad   # free space first\nsystemctl restart nomad # upgrade re-runs\n","handlingStrategy":"retry","validationCode":"// preflight: ensure writable db file and free space\nif err := syscall.Access(stateDBPath, unix.W_OK); err != nil { return err }\nif !hasFreeSpace(stateDir, minFreeBytes) { return errors.New(\"insufficient disk space for upgrade\") }","typeGuard":"func isBoltWriteErr(err error) bool {\n    return errors.Is(err, unix.ENOSPC) || errors.Is(err, unix.EROFS) || errors.Is(err, unix.EIO)\n}","tryCatchPattern":"if err := UpgradeAllocs(tx); err != nil {\n    if isBoltWriteErr(err) {\n        freeDisk(); fixMount();\n        return retryUpgradeWithBackup(stateDir)\n    }\n    return err\n}","preventionTips":["Keep ample free disk on the state volume before upgrades","Ensure the nomad process has write access to state.db","Fix RO/EIO mount issues before retrying upgrades","Only one agent process may own the state_dir"],"tags":["bolt","upgrade","state-store","disk"],"backgroundTag":"boltdb-write-failed","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}