hashicorp/nomad · error

error deleting unexpected key

Error message

error deleting unexpected key %q: %v

What it means

While iterating the allocations bucket to upgrade alloc state, any key that is not itself an alloc bucket (unexpected value key) is logged and deleted; this error is returned if that delete fails, aborting the upgrade transaction. It exists so the upgrade never silently leaves garbage keys in the allocations bucket.

Solutions

1. Free disk space on the state_dir volume and retry the upgrade.
2. Check for I/O errors (dmesg) and filesystem corruption; run fsck if needed.
3. Restore state.db from state.db.backup and re-run the upgrade on healthy storage.
4. If corruption persists, move state.db aside and let the client re-register allocations.

Example fix

// before
upgrade aborts: error deleting unexpected key "stale": errno 28
// after
df -h /var/lib/nomad   # free space first
systemctl restart nomad # upgrade re-runs

// preflight: ensure writable db file and free space
if err := syscall.Access(stateDBPath, unix.W_OK); err != nil { return err }
if !hasFreeSpace(stateDir, minFreeBytes) { return errors.New("insufficient disk space for upgrade") }

func isBoltWriteErr(err error) bool {
    return errors.Is(err, unix.ENOSPC) || errors.Is(err, unix.EROFS) || errors.Is(err, unix.EIO)
}

if err := UpgradeAllocs(tx); err != nil {
    if isBoltWriteErr(err) {
        freeDisk(); fixMount();
        return retryUpgradeWithBackup(stateDir)
    }
    return err
}

When it happens

Trigger: cur.Delete() fails on an unexpected key inside the allocations bucket — typically a bolt write error (read-only tx, disk full, page corruption) during UpgradeAllocs.

Common situations: Disk full during a Nomad client upgrade, corrupted state.db pages, or bolt transaction constraints after earlier IO errors.

Related errors

• error backing up state db
• error deleting invalid allocation state
• unexpected bucket missing
• alloc entry not found
• failed to read device manager plugin state

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/d08c506c9d5c6c68. Report an issue: GitHub.

Appendix: source

Thrown at client/state/upgrade.go:114

func UpgradeAllocs(logger hclog.Logger, tx *boltdd.Tx) error {
	btx := tx.BoltTx()
	allocationsBucket := btx.Bucket(allocationsBucketName)
	if allocationsBucket == nil {
		// No state!
		return nil
	}

	// Gather alloc buckets and remove unexpected key/value pairs
	allocBuckets := [][]byte{}
	cur := allocationsBucket.Cursor()
	for k, v := cur.First(); k != nil; k, v = cur.Next() {
		if v != nil {
			logger.Warn("deleting unexpected key in state db",
				"key", string(k), "value_bytes", len(v),
			)

			if err := cur.Delete(); err != nil {
				return fmt.Errorf("error deleting unexpected key %q: %v", string(k), err)
			}
			continue
		}

		allocBuckets = append(allocBuckets, k)
	}

	for _, allocBucket := range allocBuckets {
		allocID := string(allocBucket)

		bkt := allocationsBucket.Bucket(allocBucket)
		if bkt == nil {
			// This should never happen as we just read the bucket.
			return fmt.Errorf("unexpected bucket missing %q", allocID)
		}

		allocLogger := logger.With("alloc_id", allocID)
		if err := upgradeAllocBucket(allocLogger, tx, bkt, allocID); err != nil {

View on GitHub (pinned to 482b49bf1a)