hashicorp/nomad · error

error deleting invalid task state for task %q: %v

Error message

error deleting invalid task state for task %q: %v

What it means

During Nomad client state migration from 0.8 to 0.9+ (boltdb layout), upgradeAllocBucket attempts to delete a task bucket whose state could not be upgraded, treating it as droppable invalid state. If boltdd's DeleteBucket itself fails, the migration aborts with this unrecoverable error so the transaction is not left half-applied.

Source

Thrown at client/state/upgrade.go:216

		taskName := string(taskBucket)
		taskLogger := logger.With("task_name", taskName)

		taskBkt := bkt.Bucket(taskBucket)
		if taskBkt == nil {
			// This should never happen as we just read the bucket.
			return fmt.Errorf("unexpected bucket missing %q", taskName)
		}

		oldState, err := upgradeTaskBucket(taskLogger, taskBkt)
		if err != nil {
			taskLogger.Warn("dropping invalid task due to error while upgrading state",
				"error", err,
			)

			// Delete the invalid task bucket and treat failures
			// here as unrecoverable errors.
			if err := bkt.DeleteBucket(taskBucket); err != nil {
				return fmt.Errorf("error deleting invalid task state for task %q: %v",
					taskName, err,
				)
			}
			continue
		}

		// Convert 0.8 task state to 0.9 task state
		localTaskState, err := oldState.Upgrade(allocID, taskName)
		if err != nil {
			taskLogger.Warn("dropping invalid task due to error while upgrading state",
				"error", err,
			)

			// Delete the invalid task bucket and treat failures
			// here as unrecoverable errors.
			if err := bkt.DeleteBucket(taskBucket); err != nil {
				return fmt.Errorf("error deleting invalid task state for task %q: %v",
					taskName, err,

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Back up the client data_dir, then stop nomad and remove or restore the corrupted alloc state so migration can proceed
2. Inspect the wrapped %v error from boltdd to determine whether the bucket is missing/corrupt and repair or delete that alloc's state directory
3. Re-run nomad; invalid task state is intentionally dropped, so a clean data dir lets the agent re-sync from servers
4. Report persistent failures upstream with the wrapped error and alloc/task names

Example fix

// before (migration aborts on delete failure)
if err := bkt.DeleteBucket(taskBucket); err != nil {
    return fmt.Errorf("error deleting invalid task state for task %q: %v", taskName, err)
}
// after (operational recovery, not a code change: restore/clean client state dir)
// nomad agent stop && mv <data_dir>/client <data_dir>/client.bak && nomad agent start

// Go: before upgrading, verify state file integrity and backup
if _, err := os.Stat(filepath.Join(dataDir, "client", "state.db")); err != nil {
    return fmt.Errorf("client state missing, nothing to migrate: %w", err)
}
if err := copyFile(stateDB, stateDB+".pre-upgrade.bak"); err != nil {
    return err
}

// Go: ensure the key actually resolves to a bucket before deleting
if bkt.Bucket(taskBucket) == nil {
    return fmt.Errorf("task bucket %q is not a bucket; skipping delete", string(taskBucket))
}

// Go: wrap and surface the wrapped cause for operators
err := upgradeAllocs()
if err != nil && strings.Contains(err.Error(), "error deleting invalid task state") {
    logger.Error("state migration aborted; restore client state backup or wipe data_dir", "err", err)
}

When it happens

Trigger: UpgradeAllocs iterates an alloc bucket and finds a task sub-bucket whose 'simple-all' entry is corrupt/undecodable (upgradeTaskBucket error) or whose state.Upgrade() fails; the subsequent bkt.DeleteBucket(taskBucket) then returns an error (e.g. key no longer resolves to a bucket, bucket already deleted, or boltdd/internal bbolt error).

Common situations: Upgrading a Nomad client from 0.8.x whose data dir contains corrupted or partially-written task state (crash during prior write); running the migration twice against a modified/rolled-back boltdb file; disk-level corruption of client state.

Related errors

• error deleting unexpected task bucket %q: %v
• error delting unexpected task key %q: %v
• task state entry not found
• failed to read dynamic plugin registry state: %v
• no servers

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/97a71c6771f9bb44. Report an issue: GitHub.