hashicorp/nomad · error

core scheduler cannot handle job '%s'

Error message

core scheduler cannot handle job '%s'

What it means

The core scheduler only implements a fixed set of core jobs (GC variants, root key rotation, variables rekey, etc.). Process returns this error when an evaluation carries a CoreJob JobID not in that switch. It is a programming/state-integrity guard: an unknown core job type reached the core scheduler, typically indicating an internal inconsistency or a newer core job type handled by a different version.

Source

Thrown at nomad/core_sched.go:89

		return c.deploymentGC(customThreshold)
	case structs.CoreJobCSIVolumeClaimGC:
		return c.csiVolumeClaimGC(eval, customThreshold)
	case structs.CoreJobCSIPluginGC:
		return c.csiPluginGC(eval, customThreshold)
	case structs.CoreJobOneTimeTokenGC:
		return c.expiredOneTimeTokenGC(eval)
	case structs.CoreJobLocalTokenExpiredGC:
		return c.expiredACLTokenGC(eval, false, customThreshold)
	case structs.CoreJobGlobalTokenExpiredGC:
		return c.expiredACLTokenGC(eval, true, customThreshold)
	case structs.CoreJobRootKeyRotateOrGC:
		return c.rootKeyRotateOrGC(eval)
	case structs.CoreJobVariablesRekey:
		return c.variablesRekey(eval)
	case structs.CoreJobForceGC:
		return c.forceGC(eval)
	default:
		return fmt.Errorf("core scheduler cannot handle job '%s'", eval.JobID)
	}
}

// forceGC is used to garbage collect all eligible objects.
func (c *CoreScheduler) forceGC(eval *structs.Evaluation) error {
	// set a minimal threshold for all objects to make force GC possible
	force := new(time.Millisecond)

	if err := c.jobGC(eval, force); err != nil {
		return err
	}
	if err := c.evalGC(force); err != nil {
		return err
	}
	if err := c.deploymentGC(force); err != nil {
		return err
	}
	if err := c.csiPluginGC(eval, force); err != nil {

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Complete the rolling upgrade of all servers so the binary matches the evaluation's core job type.
2. Check `nomad server members` versions are uniform; restart laggard servers.
3. Inspect the evaluation: `nomad eval status <eval-id>` to see which job ID triggered it.
4. If caused by custom tooling, stop submitting evaluations with reserved core job ID prefixes.

Example fix

// before: mixed versions
nomad server members  # 1.6.1, 1.6.1, 1.7.0
// after: uniform upgrade
# upgrade remaining servers to 1.7.0, then restart the core scheduler workload

// ensure version uniformity before scheduling sensitive operations
members, _ := client.Agent().Members().ServerMembers
versions := map[string]int{}
for _, m := range members { versions[m.Tags["build"]]++ }
if len(versions) > 1 { return errors.New("servers on mixed versions; upgrade first") }

When it happens

Trigger: An evaluation whose JobID encodes a core job type (structs.CoreJob* prefix) that the running server binary does not recognize — e.g. a core job created by a newer Nomad version, or corrupt/forged evaluation data with an unexpected JobID.

Common situations: Rolling cluster upgrade where an evaluation created by the new binary is processed by the old binary before restart; manually inserted or corrupted evaluation rows; custom tooling submitting evaluations that mimic core job naming.

Related errors

• eval broker is enabled; eval broker must be paused to delete
• eval broker is enabled; eval broker must be paused to delete
• Job registration, dispatch, and scale are disabled by the sc
• Error querying job evaluations: %s
• All servers should be running version %v or later to use one

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/0f6cddf7105566b9. Report an issue: GitHub.