hashicorp/nomad · warning

can't set health of allocations for a terminal deployment

Error message

can't set health of allocations for a terminal deployment

What it means

ErrDeploymentTerminalNoSetHealth is returned when a client tries to set allocation health on a deployment that is already terminal (completed, failed, cancelled, or paused). Terminal deployments no longer accept state updates, so the server rejects the SetAllocHealth RPC rather than mutating a finished deployment. It keeps deployment state transitions strictly one-way.

Solutions

  1. Check deployment status before calling SetAllocHealth and skip if it is not 'running' (use the deployment's StatusIsTerminal or Active check)
  2. Treat this error as a benign no-op: log it and stop sending health updates for that deployment ID
  3. Refresh the deployment via the Deployment.Get RPC and reconcile local state; if terminal, update the alloc's deployment status locally without the RPC
  4. Upgrade/patch client code to ignore ErrDeploymentTerminalNoSetHealth the same way other terminal-deployment errors are ignored

Example fix

// before
dep, _, _ := client.Deployments().Info(depID, nil)
client.Deployments().SetAllocHealth(depID, healthy, unhealthy, nil)
// after
dep, _, _ := client.Deployments().Info(depID, nil)
if dep != nil && dep.Active() {
    client.Deployments().SetAllocHealth(depID, healthy, unhealthy, nil)
}
Defensive patterns

Strategy: try-catch

Validate before calling

dep, _, err := client.Deployments().Info(depID, nil)
if err != nil || dep == nil || !dep.Active() {
    return nil // skip health update, deployment is terminal
}

Type guard

func deploymentActive(dep *api.Deployment) bool {
    return dep != nil && dep.Active()
}

Try / catch

err := client.Deployments().SetAllocHealth(depID, healthy, unhealthy, nil, nil)
if err != nil && strings.Contains(err.Error(), "terminal deployment") {
    log.Printf("deployment %s already terminal, skipping health update", depID)
    return nil
}
return err

Prevention

When it happens

Trigger: Calling the Deployment.SetAllocHealth RPC with a deployment ID whose deployment is no longer Active(), e.g. reporting alloc health after the deployment was cancelled, promoted to completion, failed, or paused.

Common situations: A client (nomad agent alloc status / CSI or health hooks) races with an operator cancelling the deployment; retries of an old health update arrive after the deployment finished; automation polls a stale deployment ID.

Related errors


AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/179f76903d20c31e. Report an issue: GitHub.

Appendix: source

Thrown at nomad/structs/errors.go:85

	ErrMalformedChooseParameter   = errors.New(errMalformedChooseParameter)

	// ErrResultPaginatorCreation is returned by list RPC handlers when the
	// result paginator cannot be built, for example when the server cannot
	// evaluate a requested filter expression. api.ResultPaginatorErrorContent
	// duplicates its message so the CLI can match it without importing structs.
	// Keep the two in sync.
	ErrResultPaginatorCreation = errors.New(errResultPaginatorCreation)

	ErrUnknownNode = errors.New(ErrUnknownNodePrefix)

	ErrDeploymentTerminalNoCancel    = errors.New(errDeploymentTerminalNoCancel)
	ErrDeploymentTerminalNoFail      = errors.New(errDeploymentTerminalNoFail)
	ErrDeploymentTerminalNoPause     = errors.New(errDeploymentTerminalNoPause)
	ErrDeploymentTerminalNoPromote   = errors.New(errDeploymentTerminalNoPromote)
	ErrDeploymentTerminalNoResume    = errors.New(errDeploymentTerminalNoResume)
	ErrDeploymentTerminalNoUnblock   = errors.New(errDeploymentTerminalNoUnblock)
	ErrDeploymentTerminalNoRun       = errors.New(errDeploymentTerminalNoRun)
	ErrDeploymentTerminalNoSetHealth = errors.New(errDeploymentTerminalNoSetHealth)
	ErrDeploymentRunningNoUnblock    = errors.New(errDeploymentRunningNoUnblock)

	ErrCSIClientRPCIgnorable  = errors.New("CSI client error (ignorable)")
	ErrCSIClientRPCRetryable  = errors.New("CSI client error (retryable)")
	ErrCSIVolumeMaxClaims     = errors.New("volume max claims reached")
	ErrCSIVolumeUnschedulable = errors.New("volume is currently unschedulable")
	ErrCSIPluginInUse         = errors.New("plugin in use")
)

// IsErrNoLeader returns whether the error is due to there being no leader.
func IsErrNoLeader(err error) bool {
	return err != nil && strings.Contains(err.Error(), errNoLeader)
}

// IsErrNoRegionPath returns whether the error is due to there being no path to
// the given region.
func IsErrNoRegionPath(err error) bool {
	return err != nil && strings.Contains(err.Error(), errNoRegionPath)

View on GitHub (pinned to 482b49bf1a)