hashicorp/nomad · error

error delting unexpected task key %q: %v

Error message

error delting unexpected task key %q: %v

What it means

During task bucket upgrade, any key with a non-nil value that is not 'simple-all' is unexpected legacy state and is removed via the cursor's Delete(). If that delete fails, migration of the task aborts with this (typo'd 'delting') wrapped error.

Source

Thrown at client/state/upgrade.go:283

			// value is nil: delete unexpected bucket
			logger.Warn("deleting unexpected task state bucket",
				"bucket", string(k),
			)

			if err := bkt.DeleteBucket(k); err != nil {
				return nil, fmt.Errorf("error deleting unexpected task bucket %q: %v", string(k), err)
			}
			continue
		}

		if !bytes.Equal(k, []byte("simple-all")) {
			// value is non-nil: delete unexpected entry
			logger.Warn("deleting unexpected task state entry",
				"key", string(k), "value_bytes", len(v),
			)

			if err := cur.Delete(); err != nil {
				return nil, fmt.Errorf("error delting unexpected task key %q: %v", string(k), err)
			}
			continue
		}

		// Decode simple-all
		simpleFound = true
		if err := codec.NewDecoderBytes(v, structs.MsgpackHandle).Decode(&trState); err != nil {
			return nil, fmt.Errorf("failed to decode task state from 'simple-all' entry: %v", err)
		}
	}

	if !simpleFound {
		return nil, fmt.Errorf("task state entry not found")
	}

	return &trState, nil
}

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Back up and clean the client data_dir before re-running nomad so the stray keys are removed
2. Check the wrapped error to identify the underlying bbolt failure and repair the state file
3. Allow the client to rebuild task state from the servers after state cleanup
4. Escalate upstream with key name and wrapped error if it persists

// Go: log and recover operationally
if err := upgradeState(); err != nil && strings.Contains(err.Error(), "unexpected task key") {
    logger.Error("stray legacy state key blocked migration; restore backup or clean data_dir", "err", err)
}

When it happens

Trigger: UpgradeAllocs -> upgradeAllocBucket -> upgradeTaskBucket encounters a stray key/value entry in a 0.8 task bucket and bbolt cursor Delete() returns an error (e.g. bucket opened read-only or internal bolt error).

Common situations: Migrating clients whose 0.8 state contains extra keys from experimental or third-party builds; corrupted boltdb pages during upgrade.

Related errors

• error deleting invalid task state for task %q: %v
• error deleting unexpected task bucket %q: %v
• task state entry not found
• failed to read dynamic plugin registry state: %v
• timed out while opening database, is another Nomad process a

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/0633d3c0edcd355a. Report an issue: GitHub.