{"record":{"id":"7b03b92da3183efd","repo":"hashicorp/nomad","slug":"unexpected-bucket-missing-q","errorCode":null,"errorMessage":"unexpected bucket missing %q","messagePattern":"unexpected bucket missing %q","errorType":"error_code","errorClass":null,"httpStatus":null,"severity":"critical","filePath":"client/state/upgrade.go","lineNumber":128,"sourceCode":"\t\t\t\t\"key\", string(k), \"value_bytes\", len(v),\n\t\t\t)\n\n\t\t\tif err := cur.Delete(); err != nil {\n\t\t\t\treturn fmt.Errorf(\"error deleting unexpected key %q: %v\", string(k), err)\n\t\t\t}\n\t\t\tcontinue\n\t\t}\n\n\t\tallocBuckets = append(allocBuckets, k)\n\t}\n\n\tfor _, allocBucket := range allocBuckets {\n\t\tallocID := string(allocBucket)\n\n\t\tbkt := allocationsBucket.Bucket(allocBucket)\n\t\tif bkt == nil {\n\t\t\t// This should never happen as we just read the bucket.\n\t\t\treturn fmt.Errorf(\"unexpected bucket missing %q\", allocID)\n\t\t}\n\n\t\tallocLogger := logger.With(\"alloc_id\", allocID)\n\t\tif err := upgradeAllocBucket(allocLogger, tx, bkt, allocID); err != nil {\n\t\t\t// Log and drop invalid allocs\n\t\t\tallocLogger.Error(\"dropping invalid allocation due to error while upgrading state\",\n\t\t\t\t\"error\", err,\n\t\t\t)\n\n\t\t\t// If we can't delete the bucket something is seriously\n\t\t\t// wrong, fail hard.\n\t\t\tif err := allocationsBucket.DeleteBucket(allocBucket); err != nil {\n\t\t\t\treturn fmt.Errorf(\"error deleting invalid allocation state: %v\", err)\n\t\t\t}\n\t\t}\n\t}\n\n\treturn nil","sourceCodeStart":110,"sourceCodeEnd":146,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/client/state/upgrade.go#L110-L146","documentation":"During UpgradeAllocs, after collecting alloc bucket names from the allocations bucket, one of those buckets cannot be re-opened (Bucket(k) returns nil). The code comments say this should never happen since the bucket was just read, so it is treated as a hard, unrecoverable inconsistency and fails the upgrade.","triggerScenarios":"allocationsBucket.Bucket(allocBucket) returns nil for a key that was just iterated — indicating severe bolt page corruption or a concurrent modification of the DB during the upgrade transaction.","commonSituations":"Corrupted state.db after a crash mid-write; the state file being modified by another process during upgrade; restores from partial backups.","solutions":["Stop the agent and verify no other nomad process is using the same state_dir (check for stale pid/lock).","Restore state.db from state.db.backup or a backup taken before the upgrade.","If no backup exists, move the corrupt state.db aside; the client re-registers and rebuilds alloc state.","Run storage diagnostics — this points at on-disk corruption, not a Nomad bug."],"exampleFix":"// before\nupgrade aborts: unexpected bucket missing \"a1b2c3...\"\n// after\nsystemctl stop nomad\nmv /var/lib/nomad/state.db /var/lib/nomad/state.db.corrupt\nsystemctl start nomad # client rebuilds state","handlingStrategy":"fallback","validationCode":"// detect external tampering/corruption before upgrade\nif err := bolt.Open(stateDBPath, 0o600, &bolt.Options{ReadOnly: true, Timeout: time.Second}); err != nil {\n    return fmt.Errorf(\"state db not accessible/consistent: %w\", err)\n}","typeGuard":"func bucketExists(tx *boltdd.Tx, name string) bool { return tx.Bucket([]byte(name)) != nil }","tryCatchPattern":"if err := UpgradeAllocs(tx); err != nil {\n    if strings.Contains(err.Error(), \"unexpected bucket missing\") {\n        logger.Error(\"state db inconsistent; restoring from backup\", \"error\", err)\n        restoreFromBackup(stateDir)\n    }\n}","preventionTips":["Never run two nomad agents on one state_dir","Take a backup before running schema upgrades","Investigate crash-caused corruption with fsck/storage logs","Restore from backup rather than continuing past a nil bucket"],"tags":["bolt","upgrade","corruption","state-store"],"backgroundTag":"boltdb-bucket-missing","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}