{"record":{"id":"c11cc158392008fc","repo":"hashicorp/nomad","slug":"deployment-lookup-for-job-s-failed-v","errorCode":null,"errorMessage":"deployment lookup for job %s failed: %v","messagePattern":"deployment lookup for job (.+?) failed: (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"nomad/state/state_store.go","lineNumber":2034,"sourceCode":"\t}\n\n\t// Delete the job\n\tif err := txn.Delete(\"jobs\", existing); err != nil {\n\t\treturn fmt.Errorf(\"job delete failed: %v\", err)\n\t}\n\tif err := txn.Insert(\"index\", &IndexEntry{\"jobs\", index}); err != nil {\n\t\treturn fmt.Errorf(\"index update failed: %v\", err)\n\t}\n\n\t// Delete the job versions\n\tif err := s.deleteJobVersions(index, job, txn); err != nil {\n\t\treturn err\n\t}\n\n\t// Delete job deployments\n\tdeployments, err := s.DeploymentsByJobID(nil, namespace, job.ID, true)\n\tif err != nil {\n\t\treturn fmt.Errorf(\"deployment lookup for job %s failed: %v\", job.ID, err)\n\t}\n\n\tdeploymentIDs := []string{}\n\tfor _, d := range deployments {\n\t\tdeploymentIDs = append(deploymentIDs, d.ID)\n\t}\n\n\tif err := s.DeleteDeploymentTxn(index, deploymentIDs, txn); err != nil {\n\t\treturn err\n\t}\n\n\t// Mark all \"pending\" evals for this job as \"complete\"\n\tevals, err := s.EvalsByJob(nil, namespace, job.ID)\n\tif err != nil {\n\t\treturn fmt.Errorf(\"eval lookup for job %s failed: %v\", job.ID, err)\n\t}\n\n\tfor _, eval := range evals {","sourceCodeStart":2016,"sourceCodeEnd":2052,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/nomad/state/state_store.go#L2016-L2052","documentation":"DeleteJobTxn (nomad/state/state_store.go:2034) wraps an error from s.DeploymentsByJobID(nil, namespace, job.ID, true) while cleaning up deployments of the job being deleted. The deployment listing failed (a read using a separate read-only txn), so the deregistration aborts with 'deployment lookup for job <id> failed'. The wrapped %v carries the lookup root cause.","triggerScenarios":"Deleting a job that had deployments, when the DeploymentsByJobID iterator errors — MemDB read failure, wrong-type rows in the deployments table, or an iterator fault.","commonSituations":"Corrupted deployment entries restored from a snapshot; servers under memory pressure during deregistration storms; version mismatches in deployment structs after upgrades.","solutions":["Retry the deregistration; the txn rolled back","Inspect the wrapped error from DeploymentsByJobID in server logs","Restart the server agent to rebuild state from Raft; verify snapshot integrity if a restore preceded the errors","Ensure all servers run the same Nomad version to avoid struct decoding mismatches"],"exampleFix":"// before\nreturn s.DeleteJobTxn(idx, ns, jobID, txn)\n// after\nerr := s.DeleteJobTxn(idx, ns, jobID, txn)\nif err != nil && strings.Contains(err.Error(), \"deployment lookup for job\") {\n    return retryWithBackoff(func() error { return s.DeleteJobTxn(idx, ns, jobID, txn) })\n}\nreturn err","handlingStrategy":"retry","validationCode":"// pre-flight the same read the delete path uses\n_, err := stateStore.DeploymentsByJobID(nil, ns, jobID, true)\nif err != nil {\n    return fmt.Errorf(\"deployment reads unhealthy; fix before deregistering: %w\", err)\n}","typeGuard":"func hasDeployments(list []*structs.Deployment) bool { return len(list) > 0 }","tryCatchPattern":"if err := s.DeleteJobTxn(idx, ns, jobID, txn); err != nil {\n    if strings.Contains(err.Error(), \"deployment lookup for job\") {\n        return retry(3, backoff, func() error { return s.DeleteJobTxn(idx, ns, jobID, txn) })\n    }\n    return err\n}","preventionTips":["Test deployment reads (GET /v1/job/<id>/deployments) when deregistrations fail","Run uniform Nomad versions across servers to avoid struct decode mismatches","Verify snapshot integrity after restores","Retry deregistrations — failed txns roll back cleanly"],"tags":["state-store","deployments","lookup"],"backgroundTag":"memdb-lookup-failed","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}