hashicorp/nomad · error

JobByID lookup failed: %w

Error message

JobByID lookup failed: %w

What it means

After removing periodic tracking, handleJobDeregister fetches the current job record with state.JobByIDTxn to mark it stopped and re-insert it. An error from that lookup (distinct from a nil result) is wrapped as "JobByID lookup failed". This is a state-store read failure, not a missing job.

Source

Thrown at nomad/fsm.go:913

	}

	if purge {
		if err := n.state.DeleteJobTxn(index, namespace, jobID, tx); err != nil {
			return fmt.Errorf("DeleteJob failed: %w", err)
		}

		// We always delete from the periodic launch table because it is possible that
		// the job was updated to be non-periodic, thus checking if it is periodic
		// doesn't ensure we clean it up properly.
		n.state.DeletePeriodicLaunchTxn(index, namespace, jobID, tx)
		return nil
	}

	// Get the current job and mark it as stopped and re-insert it.
	ws := memdb.NewWatchSet()
	current, err := n.state.JobByIDTxn(ws, namespace, jobID, tx)
	if err != nil {
		return fmt.Errorf("JobByID lookup failed: %w", err)
	}

	if current == nil {
		return fmt.Errorf("job %q in namespace %q doesn't exist to be deregistered", jobID, namespace)
	}

	stopped := current.Copy()
	stopped.Stop = true
	if submitTime != 0 {
		stopped.SubmitTime = submitTime
	}

	// Disable scaling policies to avoid monitoring stopped jobs
	scalingPolicies := stopped.GetScalingPolicies()
	for _, policy := range scalingPolicies {
		policy.Enabled = false
	}

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Read the wrapped underlying error to identify the memdb/state-store cause.
2. Restart the server agent to rebuild state from Raft.
3. Check server resource limits (RAM) and host logs for OOM events.
4. If reproducible, capture a goroutine/profile dump and report to Nomad.

err := client.Jobs().Deregister(jobID, false, nil)
if err != nil && strings.Contains(err.Error(), "JobByID lookup failed") {
    // transient server-side error: check server health, retry
}

When it happens

Trigger: JobByIDTxn returns a non-nil error during FSM apply of a deregistration — internal memdb iterator error, transaction abort, or memory failure.

Common situations: Server under memory pressure, corrupted in-memory state store, or txn issues after a failed prior operation in the same apply.

Related errors

• index update failed: %v
• UpsertJob failed: %w
• error querying plugin %q: %v
• failed adding job to periodic dispatcher: %v
• failed to retrieve latest deployment: %v

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/851ac3ef9718b5be. Report an issue: GitHub.