What causes "heap doesn't contain job %q (%s)" in hashicorp/nomad?

Push the job into the heap before updating it

heap doesn't contain job %q (%s)

Error message

heap doesn't contain job %q (%s)

What it means

periodicHeap.Update could not find the job's namespace/ID tuple in the heap index, so there is nothing to re-heapify; the job was never added (or already removed) before the update was attempted.

Source

Thrown at nomad/periodic.go:546

	}
	_, ok := p.index[tuple]
	return ok
}

func (p *periodicHeap) Update(job *structs.Job, next time.Time) error {
	tuple := structs.NamespacedID{
		ID:        job.ID,
		Namespace: job.Namespace,
	}
	if pJob, ok := p.index[tuple]; ok {
		// Need to update the job as well because its spec can change.
		pJob.job = job
		pJob.next = next
		heap.Fix(&p.heap, pJob.index)
		return nil
	}

	return fmt.Errorf("heap doesn't contain job %q (%s)", job.ID, job.Namespace)
}

func (p *periodicHeap) Remove(job *structs.Job) error {
	tuple := structs.NamespacedID{
		ID:        job.ID,
		Namespace: job.Namespace,
	}
	if pJob, ok := p.index[tuple]; ok {
		heap.Remove(&p.heap, pJob.index)
		delete(p.index, tuple)
		return nil
	}

	return fmt.Errorf("heap doesn't contain job %q (%s)", job.ID, job.Namespace)
}

func (p *periodicHeap) Length() int {
	return len(p.heap)

View on GitHub (pinned to 482b49bf1a)

Solutions

Push the job into the heap before updating it
Check for a concurrent removal that raced the update
Fix caller ordering so Add precedes Update

Example fix

// before
err := p.heap.Update(job, next) // fails if absent
// after
if err := p.heap.Update(job, next); err != nil {
    if err2 := p.heap.Push(job, next); err2 != nil { return err2 }
}

Defensive patterns

Strategy: validation

Validate before calling

// Go: confirm presence before Update
if _, ok := p.index[tuple]; !ok {
    return p.Push(job, next)
}
return p.Update(job, next)

Try / catch

// Go
if err := p.Update(job, next); err != nil {
    if strings.Contains(err.Error(), "heap doesn't contain") {
        return p.Push(job, next)
    }
    return err
}

Prevention

Perform check-and-mutate under the dispatcher lock
Avoid concurrent Remove/Update of the same job tuple
Rebuild heap from p.tracked when diverging

When it happens

Trigger: PeriodicDispatch.Add taking the update branch (job considered tracked) or dispatch (job not yet dispatched) calling Update for a tuple missing from p.index — the job was removed concurrently or never pushed.

Common situations: Updating a periodic job that was deregistered between check and update; dispatching a job that was just removed; race between Remove and Add under a lost lock; restore gaps.

Understand the failure class

Background: "Not found" and "does not exist" errors: why "Task not found", "No such folder", and "Can't find" fire when a lookup comes back empty — this error's family across 14 libraries.

Related errors

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/565537ed98e6f52b. Report an issue: GitHub.