hashicorp/nomad · warning

can't set health of allocations for a terminal deployment

Error message

can't set health of allocations for a terminal deployment

What it means

ErrDeploymentTerminalNoSetHealth is returned when a client tries to set allocation health on a deployment that is already terminal (completed, failed, cancelled, or paused). Terminal deployments no longer accept state updates, so the server rejects the SetAllocHealth RPC rather than mutating a finished deployment. It keeps deployment state transitions strictly one-way.

Source

Thrown at nomad/structs/errors.go:85

	ErrMalformedChooseParameter   = errors.New(errMalformedChooseParameter)

	// ErrResultPaginatorCreation is returned by list RPC handlers when the
	// result paginator cannot be built, for example when the server cannot
	// evaluate a requested filter expression. api.ResultPaginatorErrorContent
	// duplicates its message so the CLI can match it without importing structs.
	// Keep the two in sync.
	ErrResultPaginatorCreation = errors.New(errResultPaginatorCreation)

	ErrUnknownNode = errors.New(ErrUnknownNodePrefix)

	ErrDeploymentTerminalNoCancel    = errors.New(errDeploymentTerminalNoCancel)
	ErrDeploymentTerminalNoFail      = errors.New(errDeploymentTerminalNoFail)
	ErrDeploymentTerminalNoPause     = errors.New(errDeploymentTerminalNoPause)
	ErrDeploymentTerminalNoPromote   = errors.New(errDeploymentTerminalNoPromote)
	ErrDeploymentTerminalNoResume    = errors.New(errDeploymentTerminalNoResume)
	ErrDeploymentTerminalNoUnblock   = errors.New(errDeploymentTerminalNoUnblock)
	ErrDeploymentTerminalNoRun       = errors.New(errDeploymentTerminalNoRun)
	ErrDeploymentTerminalNoSetHealth = errors.New(errDeploymentTerminalNoSetHealth)
	ErrDeploymentRunningNoUnblock    = errors.New(errDeploymentRunningNoUnblock)

	ErrCSIClientRPCIgnorable  = errors.New("CSI client error (ignorable)")
	ErrCSIClientRPCRetryable  = errors.New("CSI client error (retryable)")
	ErrCSIVolumeMaxClaims     = errors.New("volume max claims reached")
	ErrCSIVolumeUnschedulable = errors.New("volume is currently unschedulable")
	ErrCSIPluginInUse         = errors.New("plugin in use")
)

// IsErrNoLeader returns whether the error is due to there being no leader.
func IsErrNoLeader(err error) bool {
	return err != nil && strings.Contains(err.Error(), errNoLeader)
}

// IsErrNoRegionPath returns whether the error is due to there being no path to
// the given region.
func IsErrNoRegionPath(err error) bool {
	return err != nil && strings.Contains(err.Error(), errNoRegionPath)

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Check deployment status before calling SetAllocHealth and skip if it is not 'running' (use the deployment's StatusIsTerminal or Active check)
2. Treat this error as a benign no-op: log it and stop sending health updates for that deployment ID
3. Refresh the deployment via the Deployment.Get RPC and reconcile local state; if terminal, update the alloc's deployment status locally without the RPC
4. Upgrade/patch client code to ignore ErrDeploymentTerminalNoSetHealth the same way other terminal-deployment errors are ignored

Example fix

// before
dep, _, _ := client.Deployments().Info(depID, nil)
client.Deployments().SetAllocHealth(depID, healthy, unhealthy, nil)
// after
dep, _, _ := client.Deployments().Info(depID, nil)
if dep != nil && dep.Active() {
    client.Deployments().SetAllocHealth(depID, healthy, unhealthy, nil)
}

dep, _, err := client.Deployments().Info(depID, nil)
if err != nil || dep == nil || !dep.Active() {
    return nil // skip health update, deployment is terminal
}

func deploymentActive(dep *api.Deployment) bool {
    return dep != nil && dep.Active()
}

err := client.Deployments().SetAllocHealth(depID, healthy, unhealthy, nil, nil)
if err != nil && strings.Contains(err.Error(), "terminal deployment") {
    log.Printf("deployment %s already terminal, skipping health update", depID)
    return nil
}
return err

When it happens

Trigger: Calling the Deployment.SetAllocHealth RPC with a deployment ID whose deployment is no longer Active(), e.g. reporting alloc health after the deployment was cancelled, promoted to completion, failed, or paused.

Common situations: A client (nomad agent alloc status / CSI or health hooks) races with an operator cancelling the deployment; retries of an old health update arrive after the deployment finished; automation polls a stale deployment ID.

Related errors

• can't unblock running deployment
• deployment promotion cannot be undone
• can't cancel terminal deployment
• can't fail terminal deployment
• can't pause terminal deployment

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/179f76903d20c31e. Report an issue: GitHub.