hashicorp/nomad · error

service registration delete for alloc failed: %v

Error message

service registration delete for alloc failed: %v

What it means

After successfully deleting each alloc in the batch delete transaction, Nomad also deletes service registrations anchored to that alloc via deleteServiceRegistrationByAllocIDTxn. This error wraps any failure from that helper, aborting the transaction so the alloc deletions do not commit. It keeps the allocs table and the service registrations table consistent within one transaction.

Source

Thrown at nomad/state/state_store.go:3799

	for _, alloc := range allocs {
		raw, err := txn.First("allocs", "id", alloc)
		if err != nil {
			return fmt.Errorf("alloc lookup failed: %v", err)
		}
		if raw == nil {
			continue
		}
		if err := txn.Delete("allocs", raw); err != nil {
			return fmt.Errorf("alloc delete failed: %v", err)
		}

		// Mark that we have made a successful modification to the allocs
		// table.
		allocsTableUpdated = true

		if err := s.deleteServiceRegistrationByAllocIDTxn(txn, index, alloc); err != nil {
			return fmt.Errorf("service registration delete for alloc failed: %v", err)
		}
	}

	// Update the indexes
	if evalsTableUpdated {
		if err := txn.Insert("index", &IndexEntry{"evals", index}); err != nil {
			return fmt.Errorf("index update failed: %v", err)
		}
	}
	if allocsTableUpdated {
		if err := txn.Insert("index", &IndexEntry{"allocs", index}); err != nil {
			return fmt.Errorf("index update failed: %v", err)
		}
	}

	// TODO: should we really be doing this here?  The only time this will affect the
	// status of a job is if it's the last eval and alloc for a client, at which point
	// the status of the job will already be "dead" from handling the alloc update.

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Inspect the wrapped cause from deleteServiceRegistrationByAllocIDTxn for the root failure
2. Verify the service_registrations table/index integrity against peers or a clean snapshot
3. Restore a known-good Raft snapshot or replace the degraded server
4. Re-run the batch delete after the store is healthy

// Before deleting allocs, check for service registrations anchored to them
svcs, _, err := client.Services().Get("*", nil) // or list registrations per alloc
if err != nil { return err }
h, _, _ := client.Agent().Health(); if !h.Server.Ok { return fmt.Errorf("server unhealthy") }

func isServiceRegDeleteError(err error) bool {
	return err != nil && strings.Contains(err.Error(), "service registration delete for alloc failed")
}

if err != nil {
	if isServiceRegDeleteError(err) {
		return fmt.Errorf("service registration cleanup failed; verify service_registrations table then retry: %w", err)
	}
	return err
}

When it happens

Trigger: Batch alloc deletion when the service-registrations-by-alloc-id index is corrupted, the service_registrations table write fails, or the registration lookup/delete errors inside the transaction.

Common situations: Corrupted service registration index after snapshot restore; servers upgraded across the service-registration schema introduction with partially built indexes; deleting allocs that carry stale registrations on a degraded server.

Related errors

• service registration not found
• alloc lookup failed: %v
• failed to lookup eval allocs: %v
• alloc delete failed: %v
• error parsing: root should be an object

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/9dfc514e3e089dcd. Report an issue: GitHub.