{"record":{"id":"0934cb25a7f83854","repo":"hashicorp/nomad","slug":"node-q-s-already-exists","errorCode":null,"errorMessage":"node %q (%s) already exists","messagePattern":"node %q \\((.+?)\\) already exists","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"lib/delayheap/delay_heap.go","lineNumber":97,"sourceCode":"\tnode.index = -1 // for safety\n\t*h = old[0 : n-1]\n\treturn node\n}\n\nfunc NewDelayHeap() *DelayHeap {\n\treturn &DelayHeap{\n\t\tindex: make(map[structs.NamespacedID]*delayHeapNode),\n\t\theap:  make(delayedHeapImp, 0),\n\t}\n}\n\nfunc (p *DelayHeap) Push(dataNode HeapNode, next time.Time) error {\n\ttuple := structs.NamespacedID{\n\t\tID:        dataNode.ID(),\n\t\tNamespace: dataNode.Namespace(),\n\t}\n\tif _, ok := p.index[tuple]; ok {\n\t\treturn fmt.Errorf(\"node %q (%s) already exists\", dataNode.ID(), dataNode.Namespace())\n\t}\n\n\tdelayHeapNode := &delayHeapNode{dataNode, next, 0}\n\tp.index[tuple] = delayHeapNode\n\theap.Push(&p.heap, delayHeapNode)\n\treturn nil\n}\n\nfunc (p *DelayHeap) Pop() *delayHeapNode {\n\tif len(p.heap) == 0 {\n\t\treturn nil\n\t}\n\n\tdelayHeapNode := heap.Pop(&p.heap).(*delayHeapNode)\n\ttuple := structs.NamespacedID{\n\t\tID:        delayHeapNode.Node.ID(),\n\t\tNamespace: delayHeapNode.Node.Namespace(),\n\t}","sourceCodeStart":79,"sourceCodeEnd":115,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/lib/delayheap/delay_heap.go#L79-L115","documentation":"DelayHeap.Push registers a node keyed by its NamespacedID (ID + namespace). If an entry with that key already exists in the index, Push refuses to insert a duplicate and returns this error, protecting the heap/in-index consistency invariant.","triggerScenarios":"processEnqueue (or tests) calls Push with a HeapNode whose (ID, Namespace) tuple is already tracked — e.g. re-enqueueing an evaluation/deployment that is already in the delayed heap without removing or updating it first.","commonSituations":"Raft apply replays or duplicated enqueue scheduling logic calling Push twice for the same node; caller intended Update but used Push; a prior Remove failed silently leaving stale index entries; a restore/reconcile loop re-adding existing nodes.","solutions":["Before pushing, call DelayHeap.Update if the node may already exist, or Remove then Push","Check whether the same enqueue event is being applied twice (duplicate raft/scheduler invocation) and deduplicate","If rebuilding after restart, clear/rebuild the index rather than pushing nodes that persist","Log the ID and namespace in the error and audit why that node was still indexed"],"exampleFix":"// before\nif err := heap.Push(node, when); err != nil {\n\treturn err\n}\n// after\nif err := heap.Push(node, when); err != nil {\n\tif err := heap.Update(node, when); err != nil {\n\t\treturn err\n\t}\n}","handlingStrategy":"try-catch","validationCode":"// If DelayHeap exposes a lookup; otherwise track locally\nif _, exists := trackedNodes[structs.NamespacedID{ID: node.ID(), Namespace: node.Namespace()}]; exists {\n\treturn heap.Update(node, when) // update instead of push\n}","typeGuard":null,"tryCatchPattern":"if err := delayHeap.Push(node, when); err != nil {\n\tif strings.Contains(err.Error(), \"already exists\") {\n\t\t// fall back to update\n\t\treturn delayHeap.Update(node, when)\n\t}\n\treturn err\n}","preventionTips":["Never call Push for a node that may already be indexed; prefer Update semantics","Deduplicate enqueue events before applying (guard against raft replay / double-apply)","Ensure every Push has a matching Remove on completion/cancel","After restart, rebuild the heap from scratch rather than pushing persisted nodes"],"tags":["heap","duplicate","scheduler","internal-state"],"backgroundTag":"duplicate-node-in-delay-heap","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}