hashicorp/nomad · error

error deleting unexpected key %q: %v

Error message

error deleting unexpected key %q: %v

What it means

While iterating the allocations bucket to upgrade alloc state, any key that is not itself an alloc bucket (unexpected value key) is logged and deleted; this error is returned if that delete fails, aborting the upgrade transaction. It exists so the upgrade never silently leaves garbage keys in the allocations bucket.

Source

Thrown at client/state/upgrade.go:114

func UpgradeAllocs(logger hclog.Logger, tx *boltdd.Tx) error {
	btx := tx.BoltTx()
	allocationsBucket := btx.Bucket(allocationsBucketName)
	if allocationsBucket == nil {
		// No state!
		return nil
	}

	// Gather alloc buckets and remove unexpected key/value pairs
	allocBuckets := [][]byte{}
	cur := allocationsBucket.Cursor()
	for k, v := cur.First(); k != nil; k, v = cur.Next() {
		if v != nil {
			logger.Warn("deleting unexpected key in state db",
				"key", string(k), "value_bytes", len(v),
			)

			if err := cur.Delete(); err != nil {
				return fmt.Errorf("error deleting unexpected key %q: %v", string(k), err)
			}
			continue
		}

		allocBuckets = append(allocBuckets, k)
	}

	for _, allocBucket := range allocBuckets {
		allocID := string(allocBucket)

		bkt := allocationsBucket.Bucket(allocBucket)
		if bkt == nil {
			// This should never happen as we just read the bucket.
			return fmt.Errorf("unexpected bucket missing %q", allocID)
		}

		allocLogger := logger.With("alloc_id", allocID)
		if err := upgradeAllocBucket(allocLogger, tx, bkt, allocID); err != nil {

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Free disk space on the state_dir volume and retry the upgrade.
2. Check for I/O errors (dmesg) and filesystem corruption; run fsck if needed.
3. Restore state.db from state.db.backup and re-run the upgrade on healthy storage.
4. If corruption persists, move state.db aside and let the client re-register allocations.

Example fix

// before
upgrade aborts: error deleting unexpected key "stale": errno 28
// after
df -h /var/lib/nomad   # free space first
systemctl restart nomad # upgrade re-runs

// preflight: ensure writable db file and free space
if err := syscall.Access(stateDBPath, unix.W_OK); err != nil { return err }
if !hasFreeSpace(stateDir, minFreeBytes) { return errors.New("insufficient disk space for upgrade") }

func isBoltWriteErr(err error) bool {
    return errors.Is(err, unix.ENOSPC) || errors.Is(err, unix.EROFS) || errors.Is(err, unix.EIO)
}

if err := UpgradeAllocs(tx); err != nil {
    if isBoltWriteErr(err) {
        freeDisk(); fixMount();
        return retryUpgradeWithBackup(stateDir)
    }
    return err
}

When it happens

Trigger: cur.Delete() fails on an unexpected key inside the allocations bucket — typically a bolt write error (read-only tx, disk full, page corruption) during UpgradeAllocs.

Common situations: Disk full during a Nomad client upgrade, corrupted state.db pages, or bolt transaction constraints after earlier IO errors.

Related errors

• error backing up state db: %v
• unexpected bucket missing %q
• error deleting invalid allocation state: %v
• failed to read device manager plugin state: %v
• failed to read driver manager plugin state: %v

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/d08c506c9d5c6c68. Report an issue: GitHub.