What causes "attempting to upsert allocation %q without a job" in hashicorp/nomad?

Ensure alloc.Job is populated before calling UpsertAllocs (scheduler normally embeds the job)

attempting to upsert allocation %q without a job

Error message

attempting to upsert allocation %q without a job

What it means

A hard validation error raised when upserting a NEW allocation whose Job field is nil. Per the surrounding comment, this deliberately fails the FSM/plan_apply so the evaluation is marked failed, forcing an index refresh that lets the scheduler re-run with fresh state.

Source

Thrown at nomad/state/state_store.go:4293

			alloc.ModifyIndex = index
			alloc.AllocModifyIndex = index
			if alloc.DeploymentStatus != nil {
				alloc.DeploymentStatus.ModifyIndex = index
			}

			// Issue https://github.com/hashicorp/nomad/issues/2583 uncovered
			// a race between a forced garbage collection and the scheduler
			// marking an allocation as terminal. The issue is that the
			// allocation from the scheduler has its job normalized and the FSM
			// will only denormalize if the allocation is not terminal. However
			// if the allocation is garbage collected, that will result in an
			// allocation being upserted for the first time without a job
			// attached. By returning an error here, it will cause the FSM to
			// error, causing the plan_apply to error and thus causing the
			// evaluation to be failed. This will force an index refresh that
			// should solve this issue.
			if alloc.Job == nil {
				return fmt.Errorf("attempting to upsert allocation %q without a job", alloc.ID)
			}

			// Read the job directly from state. This ensures we do not
			// encounter an order of operations issue where the job was stopped
			// after the worker started processing the evaluation but before the
			// allocation was upserted.
			existingJob, err := txn.First("jobs", indexID, alloc.Namespace, alloc.JobID)
			if err != nil {
				return fmt.Errorf("job lookup failed: %v", err)
			}

			existingJobReal, _ := existingJob.(*structs.Job)

			// Do not return this check as an error. If we did, the scheduler
			// would retry the scheduling process using the same state snapshot
			// that showed the job as running. This would lead to a retry loop
			// that would waste CPU time and scheduling worker time.
			if existingJobReal == nil || existingJobReal.Stopped() {

View on GitHub (pinned to 482b49bf1a)

Solutions

Ensure alloc.Job is populated before calling UpsertAllocs (scheduler normally embeds the job)
Trigger a new evaluation (nomad eval list / job run) so the scheduler retries with fresh state
If from custom code, set the Job field on the allocation before submitting the plan
Report to Nomad if triggered by the built-in scheduler with full eval/plan logs

Defensive patterns

Strategy: validation

Validate before calling

// Before submitting allocs to the plan applier (custom tooling):
for _, a := range allocs {
    if a.Job == nil {
        return fmt.Errorf("alloc %s has nil Job; embed the job before upsert", a.ID)
    }
}

Type guard

func hasJob(a *structs.Allocation) bool { return a != nil && a.Job != nil }

Try / catch

if err := stateStore.UpsertAllocs(index, allocs); err != nil {
    if strings.Contains(err.Error(), "without a job") {
        // force index refresh / new evaluation so scheduler retries with fresh state
    }
}

Prevention

Always populate alloc.Job when constructing allocations
Use nomad job run / scheduler paths rather than manual state writes
Trigger a new evaluation when this error appears

When it happens

Trigger: Scheduler or plan applier submits an alloc via UpsertAllocs with alloc.Job == nil (new alloc, no embedded job).

Common situations: Custom tooling / SDK calls that build structs.Allocation by hand and forget to set Job; scheduler bugs after partial state snapshots; order-of-operations race where a job stopped mid-evaluation.

Related errors

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/b307fbce5d80371a. Report an issue: GitHub.