hashicorp/nomad · error

alloc delete failed: %v

Error message

alloc delete failed: %v

What it means

In the batch eval/alloc delete transaction, this error wraps a failure from txn.Delete("allocs", raw) after the alloc was found. The alloc exists but the memdb write transaction cannot remove it, indicating store-level corruption or an invalid transaction state. The whole transaction aborts, so deletions do not commit and subsequent per-alloc service-registration cleanup does not run.

Source

Thrown at nomad/state/state_store.go:3791

		eval := existing.(*structs.Evaluation)

		tuple := structs.NamespacedID{
			ID:        eval.JobID,
			Namespace: eval.Namespace,
		}
		jobs[tuple] = ""
	}

	for _, alloc := range allocs {
		raw, err := txn.First("allocs", "id", alloc)
		if err != nil {
			return fmt.Errorf("alloc lookup failed: %v", err)
		}
		if raw == nil {
			continue
		}
		if err := txn.Delete("allocs", raw); err != nil {
			return fmt.Errorf("alloc delete failed: %v", err)
		}

		// Mark that we have made a successful modification to the allocs
		// table.
		allocsTableUpdated = true

		if err := s.deleteServiceRegistrationByAllocIDTxn(txn, index, alloc); err != nil {
			return fmt.Errorf("service registration delete for alloc failed: %v", err)
		}
	}

	// Update the indexes
	if evalsTableUpdated {
		if err := txn.Insert("index", &IndexEntry{"evals", index}); err != nil {
			return fmt.Errorf("index update failed: %v", err)
		}
	}
	if allocsTableUpdated {

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Inspect the wrapped cause for the actual delete failure
2. Verify/restore state store integrity (clean snapshot, healthy server restart)
3. Retry the operation once the server is healthy — the transactional design makes re-running safe
4. If a specific alloc is persistently undeletable, rebuild the server (rejoin as fresh member) to rebuild its tables

// Verify the alloc is still present and server healthy before deleting
alloc, _, err := client.Allocs().GetAlloc(allocID, nil)
if err != nil || alloc == nil { return nil }
h, _, _ := client.Agent().Health(); if !h.Server.Ok { return fmt.Errorf("server unhealthy") }

func isAllocDeleteStoreError(err error) bool {
	return err != nil && strings.Contains(err.Error(), "alloc delete failed")
}

if err != nil {
	if isAllocDeleteStoreError(err) {
		return fmt.Errorf("alloc delete aborted atomically; fix server, then retry: %w", err)
	}
	return err
}

When it happens

Trigger: Batch alloc deletion where a specific alloc record cannot be deleted — schema mismatch on the alloc object, corrupted allocs table, or an aborted memdb transaction.

Common situations: Snapshot restore corruption; alloc records written under an older Nomad schema being deleted by a newer server; disk corruption during GC.

Related errors

• alloc lookup failed: %v
• index update failed: %v
• csi_plugin lookup error: %s %v
• csi_plugins insert error: %v
• csi_plugins lookup failed: %v

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/c4b9caab44539cc7. Report an issue: GitHub.