What causes "node %q (%s) already exists" in hashicorp/nomad?

Before pushing, call DelayHeap.Update if the node may already exist, or Remove then Push

node %q (%s) already exists

Error message

node %q (%s) already exists

What it means

DelayHeap.Push registers a node keyed by its NamespacedID (ID + namespace). If an entry with that key already exists in the index, Push refuses to insert a duplicate and returns this error, protecting the heap/in-index consistency invariant.

Source

Thrown at lib/delayheap/delay_heap.go:97

	node.index = -1 // for safety
	*h = old[0 : n-1]
	return node
}

func NewDelayHeap() *DelayHeap {
	return &DelayHeap{
		index: make(map[structs.NamespacedID]*delayHeapNode),
		heap:  make(delayedHeapImp, 0),
	}
}

func (p *DelayHeap) Push(dataNode HeapNode, next time.Time) error {
	tuple := structs.NamespacedID{
		ID:        dataNode.ID(),
		Namespace: dataNode.Namespace(),
	}
	if _, ok := p.index[tuple]; ok {
		return fmt.Errorf("node %q (%s) already exists", dataNode.ID(), dataNode.Namespace())
	}

	delayHeapNode := &delayHeapNode{dataNode, next, 0}
	p.index[tuple] = delayHeapNode
	heap.Push(&p.heap, delayHeapNode)
	return nil
}

func (p *DelayHeap) Pop() *delayHeapNode {
	if len(p.heap) == 0 {
		return nil
	}

	delayHeapNode := heap.Pop(&p.heap).(*delayHeapNode)
	tuple := structs.NamespacedID{
		ID:        delayHeapNode.Node.ID(),
		Namespace: delayHeapNode.Node.Namespace(),
	}

View on GitHub (pinned to 482b49bf1a)

Solutions

Before pushing, call DelayHeap.Update if the node may already exist, or Remove then Push
Check whether the same enqueue event is being applied twice (duplicate raft/scheduler invocation) and deduplicate
If rebuilding after restart, clear/rebuild the index rather than pushing nodes that persist
Log the ID and namespace in the error and audit why that node was still indexed

Example fix

// before
if err := heap.Push(node, when); err != nil {
	return err
}
// after
if err := heap.Push(node, when); err != nil {
	if err := heap.Update(node, when); err != nil {
		return err
	}
}

Defensive patterns

Strategy: try-catch

Validate before calling

// If DelayHeap exposes a lookup; otherwise track locally
if _, exists := trackedNodes[structs.NamespacedID{ID: node.ID(), Namespace: node.Namespace()}]; exists {
	return heap.Update(node, when) // update instead of push
}

Try / catch

if err := delayHeap.Push(node, when); err != nil {
	if strings.Contains(err.Error(), "already exists") {
		// fall back to update
		return delayHeap.Update(node, when)
	}
	return err
}

Prevention

Never call Push for a node that may already be indexed; prefer Update semantics
Deduplicate enqueue events before applying (guard against raft replay / double-apply)
Ensure every Push has a matching Remove on completion/cancel
After restart, rebuild the heap from scratch rather than pushing persisted nodes

When it happens

Trigger: processEnqueue (or tests) calls Push with a HeapNode whose (ID, Namespace) tuple is already tracked — e.g. re-enqueueing an evaluation/deployment that is already in the delayed heap without removing or updating it first.

Common situations: Raft apply replays or duplicated enqueue scheduling logic calling Push twice for the same node; caller intended Update but used Push; a prior Remove failed silently leaving stale index entries; a restore/reconcile loop re-adding existing nodes.

Related errors

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/0934cb25a7f83854. Report an issue: GitHub.