{"record":{"id":"b307fbce5d80371a","repo":"hashicorp/nomad","slug":"attempting-to-upsert-allocation-q-without-a-job","errorCode":null,"errorMessage":"attempting to upsert allocation %q without a job","messagePattern":"attempting to upsert allocation %q without a job","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"nomad/state/state_store.go","lineNumber":4293,"sourceCode":"\t\t\talloc.ModifyIndex = index\n\t\t\talloc.AllocModifyIndex = index\n\t\t\tif alloc.DeploymentStatus != nil {\n\t\t\t\talloc.DeploymentStatus.ModifyIndex = index\n\t\t\t}\n\n\t\t\t// Issue https://github.com/hashicorp/nomad/issues/2583 uncovered\n\t\t\t// a race between a forced garbage collection and the scheduler\n\t\t\t// marking an allocation as terminal. The issue is that the\n\t\t\t// allocation from the scheduler has its job normalized and the FSM\n\t\t\t// will only denormalize if the allocation is not terminal. However\n\t\t\t// if the allocation is garbage collected, that will result in an\n\t\t\t// allocation being upserted for the first time without a job\n\t\t\t// attached. By returning an error here, it will cause the FSM to\n\t\t\t// error, causing the plan_apply to error and thus causing the\n\t\t\t// evaluation to be failed. This will force an index refresh that\n\t\t\t// should solve this issue.\n\t\t\tif alloc.Job == nil {\n\t\t\t\treturn fmt.Errorf(\"attempting to upsert allocation %q without a job\", alloc.ID)\n\t\t\t}\n\n\t\t\t// Read the job directly from state. This ensures we do not\n\t\t\t// encounter an order of operations issue where the job was stopped\n\t\t\t// after the worker started processing the evaluation but before the\n\t\t\t// allocation was upserted.\n\t\t\texistingJob, err := txn.First(\"jobs\", indexID, alloc.Namespace, alloc.JobID)\n\t\t\tif err != nil {\n\t\t\t\treturn fmt.Errorf(\"job lookup failed: %v\", err)\n\t\t\t}\n\n\t\t\texistingJobReal, _ := existingJob.(*structs.Job)\n\n\t\t\t// Do not return this check as an error. If we did, the scheduler\n\t\t\t// would retry the scheduling process using the same state snapshot\n\t\t\t// that showed the job as running. This would lead to a retry loop\n\t\t\t// that would waste CPU time and scheduling worker time.\n\t\t\tif existingJobReal == nil || existingJobReal.Stopped() {","sourceCodeStart":4275,"sourceCodeEnd":4311,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/nomad/state/state_store.go#L4275-L4311","documentation":"A hard validation error raised when upserting a NEW allocation whose Job field is nil. Per the surrounding comment, this deliberately fails the FSM/plan_apply so the evaluation is marked failed, forcing an index refresh that lets the scheduler re-run with fresh state.","triggerScenarios":"Scheduler or plan applier submits an alloc via UpsertAllocs with alloc.Job == nil (new alloc, no embedded job).","commonSituations":"Custom tooling / SDK calls that build structs.Allocation by hand and forget to set Job; scheduler bugs after partial state snapshots; order-of-operations race where a job stopped mid-evaluation.","solutions":["Ensure alloc.Job is populated before calling UpsertAllocs (scheduler normally embeds the job)","Trigger a new evaluation (nomad eval list / job run) so the scheduler retries with fresh state","If from custom code, set the Job field on the allocation before submitting the plan","Report to Nomad if triggered by the built-in scheduler with full eval/plan logs"],"exampleFix":null,"handlingStrategy":"validation","validationCode":"// Before submitting allocs to the plan applier (custom tooling):\nfor _, a := range allocs {\n    if a.Job == nil {\n        return fmt.Errorf(\"alloc %s has nil Job; embed the job before upsert\", a.ID)\n    }\n}","typeGuard":"func hasJob(a *structs.Allocation) bool { return a != nil && a.Job != nil }","tryCatchPattern":"if err := stateStore.UpsertAllocs(index, allocs); err != nil {\n    if strings.Contains(err.Error(), \"without a job\") {\n        // force index refresh / new evaluation so scheduler retries with fresh state\n    }\n}","preventionTips":["Always populate alloc.Job when constructing allocations","Use nomad job run / scheduler paths rather than manual state writes","Trigger a new evaluation when this error appears"],"tags":["nomad","alloc","validation"],"backgroundTag":"alloc-without-job","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}