{"record":{"id":"ee68abb42b2bef6f","repo":"hashicorp/nomad","slug":"evaluation-id-ack-d-after-nack-timer-expiration","errorCode":null,"errorMessage":"Evaluation ID Ack'd after Nack timer expiration","messagePattern":"Evaluation ID Ack'd after Nack timer expiration","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"nomad/eval_broker.go","lineNumber":621,"sourceCode":"\t// Always delete the requeued evaluation. Either the Ack is successful and\n\t// we requeue it or it isn't and we want to remove it.\n\tdefer delete(b.requeue, token)\n\n\t// Lookup the unack'd eval\n\tunack, ok := b.unack[evalID]\n\tif !ok {\n\t\treturn fmt.Errorf(\"Evaluation ID not found\")\n\t}\n\tif unack.Token != token {\n\t\treturn fmt.Errorf(\"Token does not match for Evaluation ID\")\n\t}\n\tjobID := unack.Eval.JobID\n\n\tdefer b.handleAckNackLocked(unack.Eval)\n\n\t// Ensure we were able to stop the timer\n\tif !unack.NackTimer.Stop() {\n\t\treturn fmt.Errorf(\"Evaluation ID Ack'd after Nack timer expiration\")\n\t}\n\n\t// Update the stats\n\tb.stats.TotalUnacked -= 1\n\tqueue := unack.Eval.Type\n\tif b.evals[evalID] > b.deliveryLimit {\n\t\tqueue = failedQueue\n\t}\n\tbySched := b.stats.ByScheduler[queue]\n\tbySched.Unacked -= 1\n\n\t// Cleanup\n\tdelete(b.unack, evalID)\n\tdelete(b.evals, evalID)\n\n\tnamespacedID := structs.NamespacedID{\n\t\tID:        jobID,\n\t\tNamespace: unack.Eval.Namespace,","sourceCodeStart":603,"sourceCodeEnd":639,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/nomad/eval_broker.go#L603-L639","documentation":"Ack located the unacked eval and the token matched, but NackTimer.Stop() returned false, meaning the nack timer had already fired. The broker has already treated the eval as nacked (it was requeued/re-delivered), so this ack arrives too late and is rejected. This prevents the same evaluation from being acknowledged after it was given to another worker.","triggerScenarios":"A worker finishes processing and calls Ack strictly after the nack timeout elapsed — the timer goroutine already ran handleNack, removed the unack entry from the active path, or is mid-expiry when Ack races it.","commonSituations":"Handler exceeds the nack timeout under load (slow API calls, blocked RPCs); nack timeout configured too aggressively short; long GC pauses or scheduling starvation delaying the ack past the deadline; serialized duplicate job evals waiting behind the delivery limit.","solutions":["Increase the eval broker nack timeout (NewEvalBroker timeout / server config) to exceed worst-case handler latency.","Make handlers faster or process asynchronously with the ack issued promptly after dequeue of the result.","Treat this error as benign: the eval was already requeued; discard the stale ack and any duplicated work should be idempotent.","Instrument handler duration vs nackTimeout to detect chronic overruns.","Avoid blocking the worker between Dequeue and Ack with synchronous long-running RPCs."],"exampleFix":"// before\nprocessLongJob(eval) // 5 min\nbroker.Ack(eval.ID, token) // nack timer (1 min) already fired\n// after\nackCh := make(chan struct{})\ngo func() { processLongJob(eval); close(ackCh) }()\nselect {\ncase <-ackCh:\n\tbroker.Ack(eval.ID, token)\ncase <-time.After(30 * time.Second):\n\tbroker.Nack(eval.ID, token)\n}","handlingStrategy":"try-catch","validationCode":"// pre-check: handler deadline shorter than nack timeout\ndeadline := nackTimeout - 5*time.Second\nctx, cancel := context.WithTimeout(ctx, deadline)\ndefer cancel()","typeGuard":null,"tryCatchPattern":"if err := broker.Ack(evalID, token); err != nil {\n\tif strings.Contains(err.Error(), \"Nack timer expiration\") {\n\t\t// timer already fired; eval requeued. Make work idempotent.\n\t\tlog.Printf(\"late ack for eval %s\", evalID)\n\t\treturn nil\n\t}\n\treturn err\n}","preventionTips":["Size nackTimeout above p99 handler latency","Issue the ack promptly; run post-processing async after ack","Make evaluation processing idempotent (eval may run twice)","Monitor handler-duration vs nackTimeout metrics","Avoid synchronous long RPCs between Dequeue and Ack"],"tags":["go","nomad","scheduler","broker","timeout","concurrency"],"backgroundTag":"ack-after-timeout","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}