{"record":{"id":"4fd4ae90f9d24654","repo":"hashicorp/nomad","slug":"error-deleting-invalid-allocation-state-v","errorCode":null,"errorMessage":"error deleting invalid allocation state: %v","messagePattern":"error deleting invalid allocation state: (.+?)","errorType":"error_code","errorClass":null,"httpStatus":null,"severity":"critical","filePath":"client/state/upgrade.go","lineNumber":141,"sourceCode":"\t\tallocID := string(allocBucket)\n\n\t\tbkt := allocationsBucket.Bucket(allocBucket)\n\t\tif bkt == nil {\n\t\t\t// This should never happen as we just read the bucket.\n\t\t\treturn fmt.Errorf(\"unexpected bucket missing %q\", allocID)\n\t\t}\n\n\t\tallocLogger := logger.With(\"alloc_id\", allocID)\n\t\tif err := upgradeAllocBucket(allocLogger, tx, bkt, allocID); err != nil {\n\t\t\t// Log and drop invalid allocs\n\t\t\tallocLogger.Error(\"dropping invalid allocation due to error while upgrading state\",\n\t\t\t\t\"error\", err,\n\t\t\t)\n\n\t\t\t// If we can't delete the bucket something is seriously\n\t\t\t// wrong, fail hard.\n\t\t\tif err := allocationsBucket.DeleteBucket(allocBucket); err != nil {\n\t\t\t\treturn fmt.Errorf(\"error deleting invalid allocation state: %v\", err)\n\t\t\t}\n\t\t}\n\t}\n\n\treturn nil\n}\n\n// upgradeAllocBucket upgrades an alloc bucket.\nfunc upgradeAllocBucket(logger hclog.Logger, tx *boltdd.Tx, bkt *bbolt.Bucket, allocID string) error {\n\tallocFound := false\n\ttaskBuckets := [][]byte{}\n\tcur := bkt.Cursor()\n\tfor k, v := cur.First(); k != nil; k, v = cur.Next() {\n\t\tswitch string(k) {\n\t\tcase \"alloc\":\n\t\t\t// Alloc has not changed; leave it be\n\t\t\tallocFound = true\n\t\tcase \"alloc-dir\":","sourceCodeStart":123,"sourceCodeEnd":159,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/client/state/upgrade.go#L123-L159","documentation":"When upgrading an individual alloc bucket fails, the upgrade logs and drops that invalid allocation by deleting its bucket. If even DeleteBucket fails, something is seriously wrong with the DB, so the error is returned and the whole upgrade fails hard rather than continuing with a corrupt DB.","triggerScenarios":"allocationsBucket.DeleteBucket(allocBucket) errors after upgradeAllocBucket already failed — bolt-level write failure (disk full, read-only media, page corruption) during the upgrade transaction.","commonSituations":"Full disk during client upgrade, hardware/IO errors, or a state.db whose pages are corrupted such that even structural deletes fail.","solutions":["Free disk space / fix underlying I-O errors, then restart the agent to retry the upgrade.","Restore from state.db.backup or an external backup and re-upgrade.","If the DB is truly corrupted, move state.db aside and let the client rebuild; re-sync allocs from the servers.","Check dmesg/filesystem health; treat recurring cases as hardware failure."],"exampleFix":"// before\nerror deleting invalid allocation state: errno 28 (no space)\n// after\ndf -h /var/lib/nomad && rm -rf /var/lib/nomad/tmp/*\nsystemctl restart nomad # upgrade retries","handlingStrategy":"fallback","validationCode":"// ensure headroom and writability before upgrade attempt\nif !hasFreeSpace(stateDir, dbSize+minFreeBytes) { return errors.New(\"free disk space before upgrade\") }\nif err := unix.Access(stateDBPath, unix.W_OK); err != nil { return err }","typeGuard":null,"tryCatchPattern":"if err := UpgradeAllocs(tx); err != nil {\n    if strings.Contains(err.Error(), \"error deleting invalid allocation state\") {\n        logger.Error(\"state db badly corrupted; failing hard and restoring backup\", \"error\", err)\n        restoreFromBackup(stateDir) // or reset state and re-sync from servers\n    }\n}","preventionTips":["Monitor disk usage on client state volumes","Use graceful agent shutdown to avoid mid-write corruption","Schedule upgrades during healthy-disk windows only","Keep server reconciliation in mind: client state is recoverable by wiping state"],"tags":["bolt","upgrade","corruption","disk"],"backgroundTag":"boltdb-write-failed","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}