What causes "alloc %q lookup failed: %v" in hashicorp/nomad?

Retry the RPC — transient memdb errors may resolve; the Raft transaction is atomic so no partial writes persist

alloc %q lookup failed: %v

Error message

alloc %q lookup failed: %v

What it means

In UpsertDeploymentAllocHealth, the inner setAllocHealth closure looks up each allocation ID from HealthyAllocationIDs/UnhealthyAllocationIDs in the allocs table via txn.First; this error wraps a low-level memdb/txn lookup failure. Unlike 'unknown alloc', the ID may exist — the state store itself failed to perform the query. The whole Raft transaction aborts so no partial health is written.

Source

Thrown at nomad/state/state_store.go:5159

	defer txn.Abort()

	// Retrieve deployment and ensure it is not terminal and is active
	ws := memdb.NewWatchSet()
	deployment, err := s.deploymentByIDImpl(ws, req.DeploymentID, txn)
	if err != nil {
		return err
	} else if deployment == nil {
		return fmt.Errorf("Deployment ID %q couldn't be updated as it does not exist", req.DeploymentID)
	} else if !deployment.Active() {
		return fmt.Errorf("Deployment %q has terminal status %q:", deployment.ID, deployment.Status)
	}

	// Update the health status of each allocation
	if total := len(req.HealthyAllocationIDs) + len(req.UnhealthyAllocationIDs); total != 0 {
		setAllocHealth := func(id string, healthy bool, ts time.Time) error {
			existing, err := txn.First("allocs", "id", id)
			if err != nil {
				return fmt.Errorf("alloc %q lookup failed: %v", id, err)
			}
			if existing == nil {
				return fmt.Errorf("unknown alloc %q", id)
			}

			old := existing.(*structs.Allocation)
			if old.DeploymentID != req.DeploymentID {
				return fmt.Errorf("alloc %q is not part of deployment %q", id, req.DeploymentID)
			}

			// Set the health
			copy := old.Copy()
			if copy.DeploymentStatus == nil {
				copy.DeploymentStatus = &structs.AllocDeploymentStatus{}
			}
			copy.DeploymentStatus.Healthy = new(healthy)
			copy.DeploymentStatus.Timestamp = ts
			copy.DeploymentStatus.ModifyIndex = index

View on GitHub (pinned to 482b49bf1a)

Solutions

Retry the RPC — transient memdb errors may resolve; the Raft transaction is atomic so no partial writes persist
Check Nomad server logs around the same timestamp for memdb/panic messages indicating store corruption
If corruption persists, restore the server from a raft snapshot/backup or resync state from clients
Report persistent occurrences to Nomad maintainers with the wrapped inner error (%v) from logs

Defensive patterns

Strategy: retry

Try / catch

err := store.UpsertDeploymentAllocHealth(idx, req)
if err != nil && strings.Contains(err.Error(), "lookup failed") {
	// transient memdb failure: retry after backoff; Raft txn aborted atomically
	return retryWithBackoff(3, time.Second, func() error {
		return store.UpsertDeploymentAllocHealth(idx, req)
	})
}

Prevention

Retry with backoff — the aborted transaction leaves no partial state
Alert on repeated lookup failures; they indicate memdb corruption, not bad input
Keep raft snapshots/automated backups for recovery
Log the wrapped inner error verbatim for upstream triage

When it happens

Trigger: memdb transaction corruption or an internal error from txn.First("allocs", "id", id) while applying a DeploymentAllocHealthRequest containing allocation IDs.

Common situations: Underlying memdb/store corruption after an unclean shutdown; bugs in table index configuration; resource exhaustion on the server preventing memdb operations.

Related errors

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/5162e4ce0b7aa368. Report an issue: GitHub.