hashicorp/nomad · error

failed to process eval: %v

Error message

failed to process eval: %v

What it means

SystemScheduler.Process recovers any panic raised while handling a single evaluation, logs the stack trace, and converts it into this error. Like the sysbatch variant, it indicates an internal scheduler bug rather than a user configuration problem, and Nomad requests a bug report.

Source

Thrown at scheduler/scheduler_system.go:77

// NewSystemScheduler is a factory function to instantiate a new system
// scheduler.
func NewSystemScheduler(logger log.Logger, eventsCh chan<- any, state sstructs.State, planner sstructs.Planner) sstructs.Scheduler {
	return &SystemScheduler{
		logger:   logger.Named("system_sched"),
		eventsCh: eventsCh,
		state:    state,
		planner:  planner,
	}
}

// Process is used to handle a single evaluation.
func (s *SystemScheduler) Process(eval *structs.Evaluation) (err error) {

	defer func() {
		if r := recover(); r != nil {
			s.logger.Error("processing eval panicked scheduler - please report this as a bug!", "eval_id", eval.ID, "error", r, "stack_trace", string(debug.Stack()))
			err = fmt.Errorf("failed to process eval: %v", r)
		}
	}()

	// Store the evaluation
	s.eval = eval

	// Update our logger with the eval's information
	s.logger = s.logger.With("eval_id", eval.ID, "job_id", eval.JobID, "namespace", eval.Namespace)

	// Verify the evaluation trigger reason is understood
	if !s.canHandle(eval.TriggeredBy) {
		desc := fmt.Sprintf("scheduler cannot handle '%s' evaluation reason", eval.TriggeredBy)
		return setStatus(s.logger, s.planner, s.eval, nil,
			s.failedTGAllocs, s.planAnnotations, structs.EvalStatusFailed, desc,
			s.queuedAllocs, s.deployment.GetID())
	}

	limit := maxSystemScheduleAttempts

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Collect the 'stack_trace' from the server log and report the bug to Nomad with eval ID and version.
2. Inspect the triggering evaluation and system job spec for malformed or unexpected data.
3. Retry the evaluation; panics may be data-specific rather than persistent.
4. Upgrade to the latest Nomad release where the panic may already be patched.

if eval == nil || eval.ID == "" || eval.JobID == "" {
    return fmt.Errorf("malformed evaluation for system scheduler")
}

func isValidEval(e *structs.Evaluation) bool {
    return e != nil && e.ID != "" && e.Namespace != "" && e.JobID != ""
}

func safeProcessSystem(s *scheduler.SystemScheduler, eval *structs.Evaluation) (err error) {
    defer func() {
        if r := recover(); r != nil {
            err = fmt.Errorf("failed to process eval: %v", r) // log and report stack trace
        }
    }()
    return s.Process(eval)
}

When it happens

Trigger: Any panic inside the system scheduler during Process — e.g. a nil dereference in the scheduling stack, malformed evaluation input, or a downstream step (placement, plan) panicking.

Common situations: Nomad bug triggered by unusual system-job or eval data; partially initialized evaluation passed via custom tooling; regression introduced in a Nomad release.

Related errors

• failed to process eval: %v
• failed to process eval: %v
• eval broker is enabled; eval broker must be paused to delete
• eval broker is enabled; eval broker must be paused to delete
• Job registration, dispatch, and scale are disabled by the sc

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/84f859f1fed01c81. Report an issue: GitHub.