{"record":{"id":"82eaa93f0f03c994","repo":"hashicorp/nomad","slug":"task-group-q-has-d-d-healthy-allocations","errorCode":null,"errorMessage":"Task group %q has %d/%d healthy allocations","messagePattern":"Task group %q has (.+?)/(.+?) healthy allocations","errorType":"validation","errorClass":null,"httpStatus":null,"severity":"error","filePath":"nomad/state/state_store.go","lineNumber":5072,"sourceCode":"\n\t\thealthyCounts[alloc.TaskGroup]++\n\t\tpromotable = append(promotable, alloc)\n\t}\n\n\t// Determine if we have enough healthy allocations\n\tvar unhealthyErr multierror.Error\n\tfor tg, dstate := range deployment.TaskGroups {\n\t\tif _, ok := groupIndex[tg]; !req.All && !ok {\n\t\t\tcontinue\n\t\t}\n\n\t\tneed := dstate.DesiredCanaries\n\t\tif need == 0 {\n\t\t\tcontinue\n\t\t}\n\n\t\tif have := healthyCounts[tg]; have < need {\n\t\t\tmultierror.Append(&unhealthyErr, fmt.Errorf(\"Task group %q has %d/%d healthy allocations\", tg, have, need))\n\t\t}\n\t}\n\n\tif err := unhealthyErr.ErrorOrNil(); err != nil {\n\t\treturn err\n\t}\n\n\t// Update deployment\n\tcopy := deployment.Copy()\n\tcopy.ModifyIndex = index\n\tfor tg, status := range copy.TaskGroups {\n\t\t_, ok := groupIndex[tg]\n\t\tif !req.All && !ok {\n\t\t\tcontinue\n\t\t}\n\n\t\t// reset the progress deadline\n\t\tif status.ProgressDeadline > 0 && !status.RequireProgressBy.IsZero() {","sourceCodeStart":5054,"sourceCodeEnd":5090,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/nomad/state/state_store.go#L5054-L5090","documentation":"When promotion of a canary deployment is requested, every task group with canaries (DesiredCanaries > 0) must have at least that many healthy allocations. This error aggregates per-task-group shortfalls, reporting how many healthy allocs exist versus how many are required, and blocks the promotion.","triggerScenarios":"POST /v1/deployment/promotion (PromoteDeploymentLinearlyRequest / PromoteOneDeploymentGroupRequest) when healthyCounts[tg] < dstate.DesiredCanaries for one or more groups — canary allocs are still pending, unhealthy, or failing.","commonSituations":"Promoting before canaries finish health checks in fast CI pipelines; canary allocs crash-looping due to bad config or failing health checks; resource-starved nodes keeping canaries pending.","solutions":["Wait until all canary allocations are healthy (watch GET /v1/deployment/:id) before promoting","Inspect nomad alloc status for the canary allocs and fix the underlying failure (logs, health checks, resources)","If canaries are acceptable, use the force-promotion/allow-missing flag on the promotion request"],"exampleFix":"// before: promote immediately after job submit\nclient.Deployments().PromoteAll(deployID)\n// after\nfor {\n  dep, _ := client.Deployments().Info(deployID)\n  if allCanariesHealthy(dep) { break }\n  time.Sleep(5 * time.Second)\n}\nclient.Deployments().PromoteAll(deployID)","handlingStrategy":"validation","validationCode":"dep, _ := client.Deployments().Info(deployID, nil)\nfor tg, state := range dep.TaskGroups {\n  if state.DesiredCanaries > 0 && len(healthyAllocsFor(tg)) < state.DesiredCanaries {\n    return fmt.Errorf(\"group %s canaries not healthy yet\", tg)\n  }\n}","typeGuard":null,"tryCatchPattern":"err := client.Deployments().PromoteAll(deployID, nil)\nif err != nil && strings.Contains(err.Error(), \"healthy allocations\") {\n  // wait for canaries or use force promotion\n  time.Sleep(10 * time.Second)\n  retry()\n}","preventionTips":["Poll deployment status until canaries are healthy before promoting","Verify canary alloc health (nomad alloc status) when promotion fails","Use ForcePromotion/allow-missing only deliberately","Fix crash-looping canaries (config, health checks, resources) before promoting"],"tags":["nomad","deployment","canary","health-check"],"backgroundTag":"unhealthy-allocations","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}