hashicorp/nomad · critical

failed to process eval: %v

Error message

failed to process eval: %v

What it means

GenericScheduler.Process recovers from any panic raised while processing an evaluation and converts it into this error, asking the user to report it as a bug. It is not an expected operational error — it means the scheduler code panicked (nil map/deref, index out of range, etc.) while handling an eval.

Source

Thrown at scheduler/generic_sched.go:109

// NewBatchScheduler is a factory function to instantiate a new batch scheduler
func NewBatchScheduler(logger log.Logger, eventsCh chan<- any, state sstructs.State, planner sstructs.Planner) sstructs.Scheduler {
	s := &GenericScheduler{
		logger:   logger.Named("batch_sched"),
		eventsCh: eventsCh,
		state:    state,
		planner:  planner,
		batch:    true,
	}
	return s
}

// Process is used to handle a single evaluation
func (s *GenericScheduler) Process(eval *structs.Evaluation) (err error) {

	defer func() {
		if r := recover(); r != nil {
			s.logger.Error("processing eval panicked scheduler - please report this as a bug!", "eval_id", eval.ID, "error", r, "stack_trace", string(debug.Stack()))
			err = fmt.Errorf("failed to process eval: %v", r)
		}
	}()

	// Store the evaluation
	s.eval = eval

	// Update our logger with the eval's information
	s.logger = s.logger.With("eval_id", eval.ID, "job_id", eval.JobID, "namespace", eval.Namespace)

	// Verify the evaluation trigger reason is understood
	switch eval.TriggeredBy {
	case structs.EvalTriggerJobRegister, structs.EvalTriggerJobDeregister,
		structs.EvalTriggerNodeDrain, structs.EvalTriggerNodeUpdate,
		structs.EvalTriggerAllocStop, structs.EvalTriggerAllocReschedule,
		structs.EvalTriggerRollingUpdate, structs.EvalTriggerQueuedAllocs,
		structs.EvalTriggerPeriodicJob, structs.EvalTriggerMaxPlans,
		structs.EvalTriggerDeploymentWatcher, structs.EvalTriggerRetryFailedAlloc,
		structs.EvalTriggerFailedFollowUp, structs.EvalTriggerPreemption,

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Capture the stack_trace field from the log line and check the Nomad GitHub issues for a matching report
2. Upgrade Nomad to the latest patch release — many panics are fixed per version
3. Identify the eval's job and resubmit it after validating its spec with 'nomad job validate'
4. Report the bug with the eval ID, stack trace, and Nomad version if it persists

Example fix

// before
job with nil TaskGroups entry submitted, panicking the scheduler
// after
nomad job validate job.nomad.hcl && nomad job run job.nomad.hcl

nomad job validate job.nomad.hcl

func jobSpecSane(j *api.Job) bool {
	return j != nil && len(j.TaskGroups) > 0
}

// Panics are converted to errors by Process(); operators should:
// 1. grep server logs for 'processing eval panicked' and capture stack_trace
// 2. check the job with 'nomad job validate'
// 3. upgrade Nomad if the panic recurs

When it happens

Trigger: Any panic during eval processing: nil job or task group fields in a malformed job spec, unexpected nil allocation/node from the state store, or a genuine scheduler bug triggered by unusual eval content.

Common situations: Malformed job submissions that evade API validation (e.g. via older clients or direct RPC), races with state changes mid-eval, or scheduler bugs on specific Nomad versions.

Related errors

• failed to process eval: %v
• failed to process eval: %v
• eval broker is enabled; eval broker must be paused to delete
• eval broker is enabled; eval broker must be paused to delete
• Job registration, dispatch, and scale are disabled by the sc

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/2885932e6f7fc7a5. Report an issue: GitHub.