hashicorp/nomad · error

unknown alloc %q

Error message

unknown alloc %q

What it means

Thrown when an allocation ID listed in a DeploymentAllocHealthRequest does not exist in the allocs table at all (txn.First returned nil, no error). The request references an allocation the cluster has never written, so its health cannot be recorded. The Raft transaction aborts atomically, rejecting the entire health update.

Source

Thrown at nomad/state/state_store.go:5162

	ws := memdb.NewWatchSet()
	deployment, err := s.deploymentByIDImpl(ws, req.DeploymentID, txn)
	if err != nil {
		return err
	} else if deployment == nil {
		return fmt.Errorf("Deployment ID %q couldn't be updated as it does not exist", req.DeploymentID)
	} else if !deployment.Active() {
		return fmt.Errorf("Deployment %q has terminal status %q:", deployment.ID, deployment.Status)
	}

	// Update the health status of each allocation
	if total := len(req.HealthyAllocationIDs) + len(req.UnhealthyAllocationIDs); total != 0 {
		setAllocHealth := func(id string, healthy bool, ts time.Time) error {
			existing, err := txn.First("allocs", "id", id)
			if err != nil {
				return fmt.Errorf("alloc %q lookup failed: %v", id, err)
			}
			if existing == nil {
				return fmt.Errorf("unknown alloc %q", id)
			}

			old := existing.(*structs.Allocation)
			if old.DeploymentID != req.DeploymentID {
				return fmt.Errorf("alloc %q is not part of deployment %q", id, req.DeploymentID)
			}

			// Set the health
			copy := old.Copy()
			if copy.DeploymentStatus == nil {
				copy.DeploymentStatus = &structs.AllocDeploymentStatus{}
			}
			copy.DeploymentStatus.Healthy = new(healthy)
			copy.DeploymentStatus.Timestamp = ts
			copy.DeploymentStatus.ModifyIndex = index
			copy.ModifyTime = req.Timestamp.UnixNano()
			copy.ModifyIndex = index

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Re-list the deployment's allocations (`nomad deployment status <id>` / allocs by DeploymentID) and submit only currently existing alloc IDs
2. Shrink the request window: fetch allocs and submit health immediately to avoid GC races
3. Verify the request targets the same cluster/region the allocs were scheduled in
4. If GC is too aggressive, raise the eval/alloc GC thresholds (node_gc_grace, job_gc_threshold)

Example fix

// before
ids := computedFromOldEval()
store.UpsertDeploymentAllocHealth(idx, &structs.DeploymentAllocHealthRequest{DeploymentID: depID, HealthyAllocationIDs: ids})
// after
var healthy []string
for _, a := range deploymentAllocs(store, ws, depID) { // only allocs that still exist
	healthy = append(healthy, a.ID)
}
store.UpsertDeploymentAllocHealth(idx, &structs.DeploymentAllocHealthRequest{DeploymentID: depID, HealthyAllocationIDs: healthy})

func filterExistingAllocs(store *state.StateStore, ws memdb.WatchSet, ids []string) (healthy []string, unknown []string) {
	for _, id := range ids {
		if a, _ := store.AllocByID(ws, id); a != nil {
			healthy = append(healthy, id)
		} else {
			unknown = append(unknown, id)
		}
	}
	return
}

func allocKnown(a *structs.Allocation) bool { return a != nil && a.ID != "" }

if err := store.UpsertDeploymentAllocHealth(idx, req); err != nil {
	if strings.Contains(err.Error(), "unknown alloc") {
		// refresh alloc list from state and resubmit only existing allocs
	}
	return err
}

When it happens

Trigger: Client passes alloc IDs in HealthyAllocationIDs/UnhealthyAllocationIDs that were never scheduled, were GC'd before the health update, or belong to another cluster/region.

Common situations: Custom autoscaler/orchestrator scripts computing alloc IDs from stale plan/eval output; allocations reaped between plan and health submission; multi-region IDs pasted into the wrong cluster; SDK/tooling bugs sending placeholder IDs.

Understand the failure class

Background: 'Could not be found', 'does not exist', 'not found in database': the resource-not-found family when an ID, slug, key, or URI lookup comes back empty — this error's family across 20 libraries.

Related errors

• error querying plugin %q: %v
• index update failed: %v
• JobByID lookup failed: %w
• UpsertJob failed: %w
• index update failed: %v

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/82daa40daec709cf. Report an issue: GitHub.