hashicorp/nomad · critical

error deleting invalid allocation state: %v

Error message

error deleting invalid allocation state: %v

What it means

When upgrading an individual alloc bucket fails, the upgrade logs and drops that invalid allocation by deleting its bucket. If even DeleteBucket fails, something is seriously wrong with the DB, so the error is returned and the whole upgrade fails hard rather than continuing with a corrupt DB.

Source

Thrown at client/state/upgrade.go:141

		allocID := string(allocBucket)

		bkt := allocationsBucket.Bucket(allocBucket)
		if bkt == nil {
			// This should never happen as we just read the bucket.
			return fmt.Errorf("unexpected bucket missing %q", allocID)
		}

		allocLogger := logger.With("alloc_id", allocID)
		if err := upgradeAllocBucket(allocLogger, tx, bkt, allocID); err != nil {
			// Log and drop invalid allocs
			allocLogger.Error("dropping invalid allocation due to error while upgrading state",
				"error", err,
			)

			// If we can't delete the bucket something is seriously
			// wrong, fail hard.
			if err := allocationsBucket.DeleteBucket(allocBucket); err != nil {
				return fmt.Errorf("error deleting invalid allocation state: %v", err)
			}
		}
	}

	return nil
}

// upgradeAllocBucket upgrades an alloc bucket.
func upgradeAllocBucket(logger hclog.Logger, tx *boltdd.Tx, bkt *bbolt.Bucket, allocID string) error {
	allocFound := false
	taskBuckets := [][]byte{}
	cur := bkt.Cursor()
	for k, v := cur.First(); k != nil; k, v = cur.Next() {
		switch string(k) {
		case "alloc":
			// Alloc has not changed; leave it be
			allocFound = true
		case "alloc-dir":

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Free disk space / fix underlying I-O errors, then restart the agent to retry the upgrade.
2. Restore from state.db.backup or an external backup and re-upgrade.
3. If the DB is truly corrupted, move state.db aside and let the client rebuild; re-sync allocs from the servers.
4. Check dmesg/filesystem health; treat recurring cases as hardware failure.

Example fix

// before
error deleting invalid allocation state: errno 28 (no space)
// after
df -h /var/lib/nomad && rm -rf /var/lib/nomad/tmp/*
systemctl restart nomad # upgrade retries

// ensure headroom and writability before upgrade attempt
if !hasFreeSpace(stateDir, dbSize+minFreeBytes) { return errors.New("free disk space before upgrade") }
if err := unix.Access(stateDBPath, unix.W_OK); err != nil { return err }

if err := UpgradeAllocs(tx); err != nil {
    if strings.Contains(err.Error(), "error deleting invalid allocation state") {
        logger.Error("state db badly corrupted; failing hard and restoring backup", "error", err)
        restoreFromBackup(stateDir) // or reset state and re-sync from servers
    }
}

When it happens

Trigger: allocationsBucket.DeleteBucket(allocBucket) errors after upgradeAllocBucket already failed — bolt-level write failure (disk full, read-only media, page corruption) during the upgrade transaction.

Common situations: Full disk during client upgrade, hardware/IO errors, or a state.db whose pages are corrupted such that even structural deletes fail.

Related errors

• error backing up state db: %v
• error deleting unexpected key %q: %v
• unexpected bucket missing %q
• alloc entry not found
• detected corrupted token within the state store: missing rol

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/4fd4ae90f9d24654. Report an issue: GitHub.