hashicorp/nomad · error

error deleting unexpected task bucket %q: %v

Error message

error deleting unexpected task bucket %q: %v

What it means

While upgrading a 0.8 task bucket, upgradeTaskBucket iterates keys; any nested bucket (nil value) other than expected entries is unexpected state and is deleted. If boltdd's DeleteBucket fails on that nested bucket, the whole task upgrade returns this wrapped error and the task's state is dropped.

Source

Thrown at client/state/upgrade.go:271

	return nil
}

// upgradeTaskBucket iterates over keys in a task bucket, deleting invalid keys
// and returning the 0.8 version of the state.
func upgradeTaskBucket(logger hclog.Logger, bkt *bbolt.Bucket) (*taskRunnerState08, error) {
	simpleFound := false
	var trState taskRunnerState08

	cur := bkt.Cursor()
	for k, v := cur.First(); k != nil; k, v = cur.Next() {
		if v == nil {
			// value is nil: delete unexpected bucket
			logger.Warn("deleting unexpected task state bucket",
				"bucket", string(k),
			)

			if err := bkt.DeleteBucket(k); err != nil {
				return nil, fmt.Errorf("error deleting unexpected task bucket %q: %v", string(k), err)
			}
			continue
		}

		if !bytes.Equal(k, []byte("simple-all")) {
			// value is non-nil: delete unexpected entry
			logger.Warn("deleting unexpected task state entry",
				"key", string(k), "value_bytes", len(v),
			)

			if err := cur.Delete(); err != nil {
				return nil, fmt.Errorf("error delting unexpected task key %q: %v", string(k), err)
			}
			continue
		}

		// Decode simple-all
		simpleFound = true

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Stop nomad, back up the data_dir, and remove/restore the affected client state file
2. Inspect the wrapped %v error to see the boltdd failure cause and repair or discard that alloc's state
3. Let the agent re-sync task state from the Nomad servers after cleaning state
4. Report upstream with the bucket key and wrapped error if reproducible

// Go: recover by cleaning client state on wrapped delete failure
if err := runMigration(); err != nil {
    var wrapped *fmt.Errorf
    if errors.As(err, &wrapped) && strings.Contains(err.Error(), "unexpected task bucket") {
        logger.Warn("corrupt nested bucket; restore backup or clear client state", "err", err)
    }
}

When it happens

Trigger: A task sub-bucket inside an alloc's task bucket contains a nested bucket (v == nil) during UpgradeAllocs, and DeleteBucket on it fails (corrupt page, key not a bucket, tx inconsistency).

Common situations: Corrupted or hand-modified 0.8 client boltdb state being migrated to 0.9+; leftover buckets from unofficial Nomad versions or interrupted writes.

Related errors

• error deleting invalid task state for task %q: %v
• error delting unexpected task key %q: %v
• task state entry not found
• failed to read dynamic plugin registry state: %v
• timed out while opening database, is another Nomad process a

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/feeb297cce9eed5d. Report an issue: GitHub.