hashicorp/nomad · critical

unexpected bucket missing %q

Error message

unexpected bucket missing %q

What it means

During UpgradeAllocs, after collecting alloc bucket names from the allocations bucket, one of those buckets cannot be re-opened (Bucket(k) returns nil). The code comments say this should never happen since the bucket was just read, so it is treated as a hard, unrecoverable inconsistency and fails the upgrade.

Source

Thrown at client/state/upgrade.go:128

				"key", string(k), "value_bytes", len(v),
			)

			if err := cur.Delete(); err != nil {
				return fmt.Errorf("error deleting unexpected key %q: %v", string(k), err)
			}
			continue
		}

		allocBuckets = append(allocBuckets, k)
	}

	for _, allocBucket := range allocBuckets {
		allocID := string(allocBucket)

		bkt := allocationsBucket.Bucket(allocBucket)
		if bkt == nil {
			// This should never happen as we just read the bucket.
			return fmt.Errorf("unexpected bucket missing %q", allocID)
		}

		allocLogger := logger.With("alloc_id", allocID)
		if err := upgradeAllocBucket(allocLogger, tx, bkt, allocID); err != nil {
			// Log and drop invalid allocs
			allocLogger.Error("dropping invalid allocation due to error while upgrading state",
				"error", err,
			)

			// If we can't delete the bucket something is seriously
			// wrong, fail hard.
			if err := allocationsBucket.DeleteBucket(allocBucket); err != nil {
				return fmt.Errorf("error deleting invalid allocation state: %v", err)
			}
		}
	}

	return nil

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Stop the agent and verify no other nomad process is using the same state_dir (check for stale pid/lock).
2. Restore state.db from state.db.backup or a backup taken before the upgrade.
3. If no backup exists, move the corrupt state.db aside; the client re-registers and rebuilds alloc state.
4. Run storage diagnostics — this points at on-disk corruption, not a Nomad bug.

Example fix

// before
upgrade aborts: unexpected bucket missing "a1b2c3..."
// after
systemctl stop nomad
mv /var/lib/nomad/state.db /var/lib/nomad/state.db.corrupt
systemctl start nomad # client rebuilds state

// detect external tampering/corruption before upgrade
if err := bolt.Open(stateDBPath, 0o600, &bolt.Options{ReadOnly: true, Timeout: time.Second}); err != nil {
    return fmt.Errorf("state db not accessible/consistent: %w", err)
}

func bucketExists(tx *boltdd.Tx, name string) bool { return tx.Bucket([]byte(name)) != nil }

if err := UpgradeAllocs(tx); err != nil {
    if strings.Contains(err.Error(), "unexpected bucket missing") {
        logger.Error("state db inconsistent; restoring from backup", "error", err)
        restoreFromBackup(stateDir)
    }
}

When it happens

Trigger: allocationsBucket.Bucket(allocBucket) returns nil for a key that was just iterated — indicating severe bolt page corruption or a concurrent modification of the DB during the upgrade transaction.

Common situations: Corrupted state.db after a crash mid-write; the state file being modified by another process during upgrade; restores from partial backups.

Related errors

• error deleting unexpected key %q: %v
• error deleting invalid allocation state: %v
• alloc entry not found
• detected corrupted token within the state store: missing rol
• failed to read device manager plugin state: %v

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/7b03b92da3183efd. Report an issue: GitHub.