{"record":{"id":"5162e4ce0b7aa368","repo":"hashicorp/nomad","slug":"alloc-q-lookup-failed-v","errorCode":null,"errorMessage":"alloc %q lookup failed: %v","messagePattern":"alloc %q lookup failed: (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"nomad/state/state_store.go","lineNumber":5159,"sourceCode":"\tdefer txn.Abort()\n\n\t// Retrieve deployment and ensure it is not terminal and is active\n\tws := memdb.NewWatchSet()\n\tdeployment, err := s.deploymentByIDImpl(ws, req.DeploymentID, txn)\n\tif err != nil {\n\t\treturn err\n\t} else if deployment == nil {\n\t\treturn fmt.Errorf(\"Deployment ID %q couldn't be updated as it does not exist\", req.DeploymentID)\n\t} else if !deployment.Active() {\n\t\treturn fmt.Errorf(\"Deployment %q has terminal status %q:\", deployment.ID, deployment.Status)\n\t}\n\n\t// Update the health status of each allocation\n\tif total := len(req.HealthyAllocationIDs) + len(req.UnhealthyAllocationIDs); total != 0 {\n\t\tsetAllocHealth := func(id string, healthy bool, ts time.Time) error {\n\t\t\texisting, err := txn.First(\"allocs\", \"id\", id)\n\t\t\tif err != nil {\n\t\t\t\treturn fmt.Errorf(\"alloc %q lookup failed: %v\", id, err)\n\t\t\t}\n\t\t\tif existing == nil {\n\t\t\t\treturn fmt.Errorf(\"unknown alloc %q\", id)\n\t\t\t}\n\n\t\t\told := existing.(*structs.Allocation)\n\t\t\tif old.DeploymentID != req.DeploymentID {\n\t\t\t\treturn fmt.Errorf(\"alloc %q is not part of deployment %q\", id, req.DeploymentID)\n\t\t\t}\n\n\t\t\t// Set the health\n\t\t\tcopy := old.Copy()\n\t\t\tif copy.DeploymentStatus == nil {\n\t\t\t\tcopy.DeploymentStatus = &structs.AllocDeploymentStatus{}\n\t\t\t}\n\t\t\tcopy.DeploymentStatus.Healthy = new(healthy)\n\t\t\tcopy.DeploymentStatus.Timestamp = ts\n\t\t\tcopy.DeploymentStatus.ModifyIndex = index","sourceCodeStart":5141,"sourceCodeEnd":5177,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/nomad/state/state_store.go#L5141-L5177","documentation":"In UpsertDeploymentAllocHealth, the inner setAllocHealth closure looks up each allocation ID from HealthyAllocationIDs/UnhealthyAllocationIDs in the allocs table via txn.First; this error wraps a low-level memdb/txn lookup failure. Unlike 'unknown alloc', the ID may exist — the state store itself failed to perform the query. The whole Raft transaction aborts so no partial health is written.","triggerScenarios":"memdb transaction corruption or an internal error from txn.First(\"allocs\", \"id\", id) while applying a DeploymentAllocHealthRequest containing allocation IDs.","commonSituations":"Underlying memdb/store corruption after an unclean shutdown; bugs in table index configuration; resource exhaustion on the server preventing memdb operations.","solutions":["Retry the RPC — transient memdb errors may resolve; the Raft transaction is atomic so no partial writes persist","Check Nomad server logs around the same timestamp for memdb/panic messages indicating store corruption","If corruption persists, restore the server from a raft snapshot/backup or resync state from clients","Report persistent occurrences to Nomad maintainers with the wrapped inner error (%v) from logs"],"exampleFix":null,"handlingStrategy":"retry","validationCode":null,"typeGuard":null,"tryCatchPattern":"err := store.UpsertDeploymentAllocHealth(idx, req)\nif err != nil && strings.Contains(err.Error(), \"lookup failed\") {\n\t// transient memdb failure: retry after backoff; Raft txn aborted atomically\n\treturn retryWithBackoff(3, time.Second, func() error {\n\t\treturn store.UpsertDeploymentAllocHealth(idx, req)\n\t})\n}","preventionTips":["Retry with backoff — the aborted transaction leaves no partial state","Alert on repeated lookup failures; they indicate memdb corruption, not bad input","Keep raft snapshots/automated backups for recovery","Log the wrapped inner error verbatim for upstream triage"],"tags":["nomad","state-store","memdb","raft"],"backgroundTag":"state-store-lookup-failed","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}