hashicorp/nomad · error

job %q in namespace %q not found

Error message

job %q in namespace %q not found

What it means

When a plan arrives with only JobInfo (no embedded Job), the leader looks the job up in the state store by namespace and ID. If the job no longer exists — it was dropped while the plan was in flight — the handler returns this error rather than processing a plan for a missing job.

Source

Thrown at nomad/plan_endpoint.go:60

		return fmt.Errorf("cannot submit nil plan")
	}

	plan := args.Plan
	if plan.Job == nil {
		if plan.JobInfo == nil {
			return fmt.Errorf("cannot submit plan without job info")
		}

		// we lookup the job immediately after the plan submission is requested,
		// in order to save time not having to look it up whenever needed and
		// more importantly, to avoid nil jobs in the plan in situations when
		// job gets dropped from the state store while plan is still in flight.
		job, err := p.srv.State().JobByID(nil, plan.JobInfo.Namespace, plan.JobInfo.ID)
		if err != nil {
			return err
		}
		if job == nil {
			return fmt.Errorf("job %q in namespace %q not found", plan.JobInfo.ID, plan.JobInfo.Namespace)
		}
		plan.Job = job
	}

	// Pause the Nack timer for the eval as it is making progress as long as it
	// is in the plan queue. We resume immediately after we get a result to
	// handle the case that the receiving worker dies.
	id := plan.EvalID
	token := plan.EvalToken
	if err := p.srv.evalBroker.PauseNackTimeout(id, token); err != nil {
		return err
	}
	defer p.srv.evalBroker.ResumeNackTimeout(id, token)

	// Submit the plan to the queue
	future, err := p.srv.planQueue.Enqueue(plan)
	if err != nil {
		return err

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Re-check job existence before relying on in-flight plans; re-submit an evaluation after re-registering the job
2. Avoid deleting jobs while their evaluations are pending; deregister gracefully and let evals drain
3. Retry plan submission after re-registering the job with the same ID/namespace
4. Verify namespace and ID in JobInfo match the actual registered job (typos yield the same error)

Example fix

// before
job, _ := state.JobByID(nil, ns, id)
submitPlan(plan) // plan for job already deleted
// after
job, _ := state.JobByID(nil, ns, id)
if job == nil {
    _, _ = jobs.Register(registerReq) // re-register before planning
}
submitPlan(plan)

job, err := state.JobByID(nil, ns, id)
if err != nil || job == nil {
    // re-register job or abort plan submission
}

if err := submitPlan(plan); err != nil && strings.Contains(err.Error(), "not found") {
    // re-register the job then resubmit the plan
    if _, rerr := client.Jobs().Register(regReq, nil); rerr == nil {
        err = submitPlan(plan)
    }
}

When it happens

Trigger: Plan.Submit with plan.Job==nil whose plan.JobInfo.Namespace/ID do not match any job in the state store at submit time — e.g. the job was deregistered between eval creation and plan application.

Common situations: Job deleted (nomad job stop) while its evaluations/plans were still being processed; aggressive cleanup scripts deleting jobs during deployments; race after `nomad job purge`.

Related errors

• could not resolve node secret: %w
• could not place volume %q: %w
• cannot submit nil plan
• cannot submit plan without job info
• error parsing: root should be an object

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/c7978dab2a474a61. Report an issue: GitHub.