hashicorp/nomad · error

failed to process eval: %v

Error message

failed to process eval: %v

What it means

SysBatchScheduler.Process wraps any panic raised while handling a single evaluation, logs the stack trace, and converts it into this error. It signals an internal scheduler bug (nil dereference, index-out-of-range, etc.) rather than a user-caused condition. Nomad explicitly asks users to report it as a bug.

Source

Thrown at scheduler/scheduler_sysbatch.go:68

	planAnnotations *structs.PlanAnnotations
}

func NewSysBatchScheduler(logger log.Logger, eventsCh chan<- any, state sstructs.State, planner sstructs.Planner) sstructs.Scheduler {
	return &SysBatchScheduler{
		logger:   logger.Named("sysbatch_sched"),
		eventsCh: eventsCh,
		state:    state,
		planner:  planner,
	}
}

// Process is used to handle a single evaluation.
func (s *SysBatchScheduler) Process(eval *structs.Evaluation) (err error) {

	defer func() {
		if r := recover(); r != nil {
			s.logger.Error("processing eval panicked scheduler - please report this as a bug!", "eval_id", eval.ID, "error", r, "stack_trace", string(debug.Stack()))
			err = fmt.Errorf("failed to process eval: %v", r)
		}
	}()

	// Store the evaluation
	s.eval = eval

	// Update our logger with the eval's information
	s.logger = s.logger.With("eval_id", eval.ID, "job_id", eval.JobID, "namespace", eval.Namespace)

	// Verify the evaluation trigger reason is understood
	if !s.canHandle(eval.TriggeredBy) {
		desc := fmt.Sprintf("scheduler cannot handle '%s' evaluation reason", eval.TriggeredBy)
		return setStatus(s.logger, s.planner, s.eval, nil,
			s.failedTGAllocs, s.planAnnotations, structs.EvalStatusFailed, desc,
			s.queuedAllocs, "")
	}

	limit := maxSysBatchScheduleAttempts

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Capture the 'stack_trace' from the server log and file a bug report with Nomad including eval ID and Nomad version.
2. Inspect the evaluation that triggered the panic (nomad eval status / state store) for malformed data.
3. Retry the evaluation; transient panics on one eval may not recur.
4. Upgrade to the latest Nomad version where the panic may already be fixed.

Example fix

// caller-side handling
// before
err := sysBatch.Process(eval)
// after
if err := recoverPanic(func() error { return sysBatch.Process(eval) }); err != nil {
    logger.Error("scheduler panicked", "err", err) // send stack trace upstream
}

if eval == nil || eval.ID == "" || eval.JobID == "" {
    return fmt.Errorf("malformed evaluation: missing id/job fields")
}

func isValidEval(e *structs.Evaluation) bool {
    return e != nil && e.ID != "" && e.Namespace != "" && e.JobID != ""
}

func safeProcess(s *scheduler.SysBatchScheduler, eval *structs.Evaluation) (err error) {
    defer func() {
        if r := recover(); r != nil {
            err = fmt.Errorf("failed to process eval: %v", r) // report stack trace upstream
        }
    }()
    return s.Process(eval)
}

When it happens

Trigger: Any panic inside the sysbatch scheduler during Process — e.g. nil job/plan fields, malformed evaluation passed to Process, or a bug in a downstream scheduler step that panics.

Common situations: A nil or partially initialized *structs.Evaluation passed by custom tooling; Nomad bug triggered by unusual job/eval data; regression after a Nomad upgrade.

Related errors

• failed to process eval: %v
• failed to get job '%s': %v
• failed to get ready nodes: %v
• failed to get job node pool %q: %v
• failed to get scheduler configuration: %v

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/9812f3ec2e38de82. Report an issue: GitHub.