hashicorp/nomad · error

failed to get job's allocations: %v

Error message

failed to get job's allocations: %v

What it means

propertySet.populateExisting() fetches all previously placed allocations for the job from the Nomad state store via StateStore.AllocsByJob. If that state store lookup returns an error, the scheduler wraps it in this message and marks errorBuilding, aborting feasibility evaluation for the property set. It signals an internal state-store failure, not a scheduling constraint problem.

Source

Thrown at scheduler/feasible/propertyset.go:149

	// Populate the proposed when setting the constraint. We do this because
	// when detecting if we can inplace update an allocation we stage an
	// eviction and then select. This means the plan has an eviction before a
	// single select has finished.
	p.PopulateProposed()
}

func (p *propertySet) SetTargetValues(values []string) {
	p.targetValues = set.From(values)
}

// populateExisting is a helper shared when setting the constraint to populate
// the existing values.
func (p *propertySet) populateExisting() {
	// Retrieve all previously placed allocations
	ws := memdb.NewWatchSet()
	allocs, err := p.ctx.State().AllocsByJob(ws, p.namespace, p.jobID, false)
	if err != nil {
		p.errorBuilding = fmt.Errorf("failed to get job's allocations: %v", err)
		p.logger.Error("failed to get job's allocations", "job", p.jobID, "namespace", p.namespace, "error", err)
		return
	}

	// Filter to the correct set of allocs
	allocs = p.filterAllocs(allocs, true)

	// Get all the nodes that have been used by the allocs
	nodes, err := p.buildNodeMap(allocs)
	if err != nil {
		p.errorBuilding = err
		p.logger.Error("failed to build node map", "error", err)
		return
	}

	// Build existing properties map
	p.populateProperties(allocs, nodes, p.existingValues)
}

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Inspect the wrapped inner error (%v) in the server logs to identify the underlying state store failure
2. Restart the Nomad server leader election (or the agent) to rebuild the state store in memory
3. Check server health: memory, disk, and raft state; run 'nomad operator raft list-peers' and verify cluster stability
4. If corruption is suspected, restore from a raft snapshot or resync from other servers

// Verify cluster/state health before submitting work
status := exec.Command("nomad", "status").Run()
if status != nil { log.Fatal("nomad cluster unreachable or unhealthy") }

// Transient state store failures: rely on Nomad's eval retry
// Operators: monitor evals
nomad eval list | grep -i failed
nomad server members
# If persistent, restart leader or restore raft snapshot

When it happens

Trigger: StateStore.AllocsByJob returns an error while building the property set during placement evaluation — e.g. memdb/state store corruption, an aborted or closed state store snapshot, or an index iteration failure. Called via setTargetAttributeWithCount during stack feasibility checks.

Common situations: Server state store pressure/corruption, Raft restore in progress, state store snapshot closed mid-eval, or resource exhaustion on the Nomad server (OOM, file descriptor limits).

Related errors

• eval broker is enabled; eval broker must be paused to delete
• failed to lookup node ID %q: %v
• failed to get job %q: %v
• failed to get job deployment %q: %v
• failed to get allocs for job '%s': %v

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/7dcd4dea9364c8de. Report an issue: GitHub.