{"record":{"id":"82daa40daec709cf","repo":"hashicorp/nomad","slug":"unknown-alloc-q","errorCode":null,"errorMessage":"unknown alloc %q","messagePattern":"unknown alloc %q","errorType":"validation","errorClass":null,"httpStatus":null,"severity":"error","filePath":"nomad/state/state_store.go","lineNumber":5162,"sourceCode":"\tws := memdb.NewWatchSet()\n\tdeployment, err := s.deploymentByIDImpl(ws, req.DeploymentID, txn)\n\tif err != nil {\n\t\treturn err\n\t} else if deployment == nil {\n\t\treturn fmt.Errorf(\"Deployment ID %q couldn't be updated as it does not exist\", req.DeploymentID)\n\t} else if !deployment.Active() {\n\t\treturn fmt.Errorf(\"Deployment %q has terminal status %q:\", deployment.ID, deployment.Status)\n\t}\n\n\t// Update the health status of each allocation\n\tif total := len(req.HealthyAllocationIDs) + len(req.UnhealthyAllocationIDs); total != 0 {\n\t\tsetAllocHealth := func(id string, healthy bool, ts time.Time) error {\n\t\t\texisting, err := txn.First(\"allocs\", \"id\", id)\n\t\t\tif err != nil {\n\t\t\t\treturn fmt.Errorf(\"alloc %q lookup failed: %v\", id, err)\n\t\t\t}\n\t\t\tif existing == nil {\n\t\t\t\treturn fmt.Errorf(\"unknown alloc %q\", id)\n\t\t\t}\n\n\t\t\told := existing.(*structs.Allocation)\n\t\t\tif old.DeploymentID != req.DeploymentID {\n\t\t\t\treturn fmt.Errorf(\"alloc %q is not part of deployment %q\", id, req.DeploymentID)\n\t\t\t}\n\n\t\t\t// Set the health\n\t\t\tcopy := old.Copy()\n\t\t\tif copy.DeploymentStatus == nil {\n\t\t\t\tcopy.DeploymentStatus = &structs.AllocDeploymentStatus{}\n\t\t\t}\n\t\t\tcopy.DeploymentStatus.Healthy = new(healthy)\n\t\t\tcopy.DeploymentStatus.Timestamp = ts\n\t\t\tcopy.DeploymentStatus.ModifyIndex = index\n\t\t\tcopy.ModifyTime = req.Timestamp.UnixNano()\n\t\t\tcopy.ModifyIndex = index\n","sourceCodeStart":5144,"sourceCodeEnd":5180,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/nomad/state/state_store.go#L5144-L5180","documentation":"Thrown when an allocation ID listed in a DeploymentAllocHealthRequest does not exist in the allocs table at all (txn.First returned nil, no error). The request references an allocation the cluster has never written, so its health cannot be recorded. The Raft transaction aborts atomically, rejecting the entire health update.","triggerScenarios":"Client passes alloc IDs in HealthyAllocationIDs/UnhealthyAllocationIDs that were never scheduled, were GC'd before the health update, or belong to another cluster/region.","commonSituations":"Custom autoscaler/orchestrator scripts computing alloc IDs from stale plan/eval output; allocations reaped between plan and health submission; multi-region IDs pasted into the wrong cluster; SDK/tooling bugs sending placeholder IDs.","solutions":["Re-list the deployment's allocations (`nomad deployment status <id>` / allocs by DeploymentID) and submit only currently existing alloc IDs","Shrink the request window: fetch allocs and submit health immediately to avoid GC races","Verify the request targets the same cluster/region the allocs were scheduled in","If GC is too aggressive, raise the eval/alloc GC thresholds (node_gc_grace, job_gc_threshold)"],"exampleFix":"// before\nids := computedFromOldEval()\nstore.UpsertDeploymentAllocHealth(idx, &structs.DeploymentAllocHealthRequest{DeploymentID: depID, HealthyAllocationIDs: ids})\n// after\nvar healthy []string\nfor _, a := range deploymentAllocs(store, ws, depID) { // only allocs that still exist\n\thealthy = append(healthy, a.ID)\n}\nstore.UpsertDeploymentAllocHealth(idx, &structs.DeploymentAllocHealthRequest{DeploymentID: depID, HealthyAllocationIDs: healthy})","handlingStrategy":"validation","validationCode":"func filterExistingAllocs(store *state.StateStore, ws memdb.WatchSet, ids []string) (healthy []string, unknown []string) {\n\tfor _, id := range ids {\n\t\tif a, _ := store.AllocByID(ws, id); a != nil {\n\t\t\thealthy = append(healthy, id)\n\t\t} else {\n\t\t\tunknown = append(unknown, id)\n\t\t}\n\t}\n\treturn\n}","typeGuard":"func allocKnown(a *structs.Allocation) bool { return a != nil && a.ID != \"\" }","tryCatchPattern":"if err := store.UpsertDeploymentAllocHealth(idx, req); err != nil {\n\tif strings.Contains(err.Error(), \"unknown alloc\") {\n\t\t// refresh alloc list from state and resubmit only existing allocs\n\t}\n\treturn err\n}","preventionTips":["List allocs from live state right before submitting health — never reuse stale ID lists","Filter out alloc IDs that were GC'd or belong to other clusters","Keep the fetch-to-submit window short to dodge GC races","Log dropped unknown IDs for debugging rather than failing the whole batch silently"],"tags":["nomad","state-store","allocation","raft"],"backgroundTag":"resource-not-found","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}