What causes "heap doesn't contain object with ID %q (%s)" in hashicorp/nomad?

Verify the node is still in the heap (Push succeeded and Remove/Pop hasn't run) before calling Update

heap doesn't contain object with ID %q (%s)

Error message

heap doesn't contain object with ID %q (%s)

What it means

DelayHeap.Update mutates an existing node's payload and WaitUntil time, but only if the node's NamespacedID is present in the index. If the tuple (ID, namespace) is absent, it returns this error instead of silently creating or missing an entry.

Source

Thrown at lib/delayheap/delay_heap.go:150

	}
	_, ok := p.index[tuple]
	return ok
}

func (p *DelayHeap) Update(heapNode HeapNode, waitUntil time.Time) error {
	tuple := structs.NamespacedID{
		ID:        heapNode.ID(),
		Namespace: heapNode.Namespace(),
	}
	if existingHeapNode, ok := p.index[tuple]; ok {
		// Need to update the job as well because its spec can change.
		existingHeapNode.Node = heapNode
		existingHeapNode.WaitUntil = waitUntil
		heap.Fix(&p.heap, existingHeapNode.index)
		return nil
	}

	return fmt.Errorf("heap doesn't contain object with ID %q (%s)", heapNode.ID(), heapNode.Namespace())
}

func (p *DelayHeap) Remove(heapNode HeapNode) error {
	tuple := structs.NamespacedID{
		ID:        heapNode.ID(),
		Namespace: heapNode.Namespace(),
	}
	if node, ok := p.index[tuple]; ok {
		heap.Remove(&p.heap, node.index)
		delete(p.index, tuple)
		return nil
	}

	return fmt.Errorf("heap doesn't contain object with ID %q (%s)", heapNode.ID(), heapNode.Namespace())
}

func (p *DelayHeap) Length() int {
	return len(p.heap)

View on GitHub (pinned to 482b49bf1a)

Solutions

Verify the node is still in the heap (Push succeeded and Remove/Pop hasn't run) before calling Update
Treat this error as benign when it results from the timer already firing; remove instead of update in that path
If the node may not exist, check containment first (or add a Get/Lookup helper on DelayHeap) and choose Push vs Update accordingly
Check for out-of-order processing between deletion events and update events for the same ID

Example fix

// before
if err := heap.Update(node, when); err != nil {
	return err
}
// after
if err := heap.Update(node, when); err != nil {
	// node may have already fired/was removed; enqueue fresh
	return heap.Push(node, when)
}

Defensive patterns

Strategy: try-catch

Validate before calling

// Ensure the node was pushed and not yet removed before updating
if !nodePushed[node.ID()] || nodeRemoved[node.ID()] {
	return fmt.Errorf("cannot update node %q: not in heap", node.ID())
}

Try / catch

if err := delayHeap.Update(node, when); err != nil {
	if strings.Contains(err.Error(), "heap doesn't contain object") {
		// node already fired or was removed; push it fresh instead
		return delayHeap.Push(node, when)
	}
	return err
}

Prevention

Serialize Update/Remove/Pop operations per node ID to avoid firing-vs-update races
Drop stale update events for nodes whose timers already fired
Re-check heap membership after any Remove/Pop before issuing an Update
In tests, always Push nodes before calling Update

When it happens

Trigger: Update is called with a HeapNode that was never Pushed, or was already removed via Remove/Pop — e.g. updating an evaluation whose timer already fired or was cancelled before the update arrives.

Common situations: Race between a node's deadline firing (Pop) and a late Update from the scheduler; calling Update after Remove during reconciliation; stale references to nodes across heap rebuilds/restarts; test code constructing nodes without pushing first.

Related errors

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/f69759bdec72358b. Report an issue: GitHub.