hashicorp/nomad · error

Task group %q has %d/%d healthy allocations

Error message

Task group %q has %d/%d healthy allocations

What it means

When promotion of a canary deployment is requested, every task group with canaries (DesiredCanaries > 0) must have at least that many healthy allocations. This error aggregates per-task-group shortfalls, reporting how many healthy allocs exist versus how many are required, and blocks the promotion.

Source

Thrown at nomad/state/state_store.go:5072

		healthyCounts[alloc.TaskGroup]++
		promotable = append(promotable, alloc)
	}

	// Determine if we have enough healthy allocations
	var unhealthyErr multierror.Error
	for tg, dstate := range deployment.TaskGroups {
		if _, ok := groupIndex[tg]; !req.All && !ok {
			continue
		}

		need := dstate.DesiredCanaries
		if need == 0 {
			continue
		}

		if have := healthyCounts[tg]; have < need {
			multierror.Append(&unhealthyErr, fmt.Errorf("Task group %q has %d/%d healthy allocations", tg, have, need))
		}
	}

	if err := unhealthyErr.ErrorOrNil(); err != nil {
		return err
	}

	// Update deployment
	copy := deployment.Copy()
	copy.ModifyIndex = index
	for tg, status := range copy.TaskGroups {
		_, ok := groupIndex[tg]
		if !req.All && !ok {
			continue
		}

		// reset the progress deadline
		if status.ProgressDeadline > 0 && !status.RequireProgressBy.IsZero() {

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Wait until all canary allocations are healthy (watch GET /v1/deployment/:id) before promoting
2. Inspect nomad alloc status for the canary allocs and fix the underlying failure (logs, health checks, resources)
3. If canaries are acceptable, use the force-promotion/allow-missing flag on the promotion request

Example fix

// before: promote immediately after job submit
client.Deployments().PromoteAll(deployID)
// after
for {
  dep, _ := client.Deployments().Info(deployID)
  if allCanariesHealthy(dep) { break }
  time.Sleep(5 * time.Second)
}
client.Deployments().PromoteAll(deployID)

dep, _ := client.Deployments().Info(deployID, nil)
for tg, state := range dep.TaskGroups {
  if state.DesiredCanaries > 0 && len(healthyAllocsFor(tg)) < state.DesiredCanaries {
    return fmt.Errorf("group %s canaries not healthy yet", tg)
  }
}

err := client.Deployments().PromoteAll(deployID, nil)
if err != nil && strings.Contains(err.Error(), "healthy allocations") {
  // wait for canaries or use force promotion
  time.Sleep(10 * time.Second)
  retry()
}

When it happens

Trigger: POST /v1/deployment/promotion (PromoteDeploymentLinearlyRequest / PromoteOneDeploymentGroupRequest) when healthyCounts[tg] < dstate.DesiredCanaries for one or more groups — canary allocs are still pending, unhealthy, or failing.

Common situations: Promoting before canaries finish health checks in fast CI pipelines; canary allocs crash-looping due to bad config or failing health checks; resource-starved nodes keeping canaries pending.

Related errors

• can't promote terminal deployment
• auto_promote must be true for all groups to enable automatic
• deployment promotion cannot be undone
• can't cancel terminal deployment
• can't fail terminal deployment

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/82eaa93f0f03c994. Report an issue: GitHub.