{"record":{"id":"f124cf00aa3af88c","repo":"hashicorp/nomad","slug":"token-does-not-match-for-evaluation-id","errorCode":null,"errorMessage":"Token does not match for Evaluation ID","messagePattern":"Token does not match for Evaluation ID","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"nomad/eval_broker.go","lineNumber":613,"sourceCode":"\treturn nil\n}\n\n// Ack is used to positively acknowledge handling an evaluation\nfunc (b *EvalBroker) Ack(evalID, token string) error {\n\tb.l.Lock()\n\tdefer b.l.Unlock()\n\n\t// Always delete the requeued evaluation. Either the Ack is successful and\n\t// we requeue it or it isn't and we want to remove it.\n\tdefer delete(b.requeue, token)\n\n\t// Lookup the unack'd eval\n\tunack, ok := b.unack[evalID]\n\tif !ok {\n\t\treturn fmt.Errorf(\"Evaluation ID not found\")\n\t}\n\tif unack.Token != token {\n\t\treturn fmt.Errorf(\"Token does not match for Evaluation ID\")\n\t}\n\tjobID := unack.Eval.JobID\n\n\tdefer b.handleAckNackLocked(unack.Eval)\n\n\t// Ensure we were able to stop the timer\n\tif !unack.NackTimer.Stop() {\n\t\treturn fmt.Errorf(\"Evaluation ID Ack'd after Nack timer expiration\")\n\t}\n\n\t// Update the stats\n\tb.stats.TotalUnacked -= 1\n\tqueue := unack.Eval.Type\n\tif b.evals[evalID] > b.deliveryLimit {\n\t\tqueue = failedQueue\n\t}\n\tbySched := b.stats.ByScheduler[queue]\n\tbySched.Unacked -= 1","sourceCodeStart":595,"sourceCodeEnd":631,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/nomad/eval_broker.go#L595-L631","documentation":"Ack found the evaluation in the unacked map, but the supplied dequeue token does not match the token issued at Dequeue time. The token proves ownership of the in-flight evaluation; a mismatch means a different worker (or a stale worker) is trying to ack. This guards against double-processing across workers.","triggerScenarios":"Calling Ack(evalID, wrongToken): mixing up tokens between two dequeued evals, acking with a token from a previous dequeue of the same eval ID, or passing an empty/zero-value token.","commonSituations":"Worker crash and restart resends an old ack with a lost in-memory token; a shared handler closes over the wrong token variable; eval ID collisions in custom code that reuses eval IDs.","solutions":["Pass the exact token string returned by the matching Dequeue call.","Keep (evalID, token) paired in the worker's state; never cache tokens across restarts.","Fix variable shadowing/mix-ups where multiple dequeues are in flight.","If the token was lost, Nack cannot help either — let the nack timer expire and pick up the eval on a fresh Dequeue."],"exampleFix":"// before\nbroker.Ack(eval.ID, oldToken) // token from earlier dequeue\n// after\nbroker.Ack(eval.ID, dequeueToken) // token captured from same Dequeue result","handlingStrategy":"validation","validationCode":"func validateAckPair(evalID, token string, issued map[string]string) error {\n\tif want, ok := issued[evalID]; !ok || want != token {\n\t\treturn fmt.Errorf(\"token mismatch for eval %s\", evalID)\n\t}\n\treturn nil\n}","typeGuard":null,"tryCatchPattern":"if err := broker.Ack(evalID, token); err != nil {\n\tif strings.Contains(err.Error(), \"Token does not match\") {\n\t\tlog.Printf(\"lost ownership of eval %s; skipping ack\", evalID)\n\t\treturn nil\n\t}\n\treturn err\n}","preventionTips":["Bind the token tightly to the dequeue in worker state (struct field, not global)","Never persist/reuse tokens across worker restarts","Guard against concurrent handlers sharing token variables","Use one goroutine per dequeue to avoid token mix-ups"],"tags":["go","nomad","scheduler","broker","concurrency"],"backgroundTag":"token-mismatch","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T15:18:49.778Z"}