hashicorp/nomad · error

Deployment %q has terminal status %q:

Error message

Deployment %q has terminal status %q:

What it means

Returned by UpdateDeploymentStatus when the deployment exists but has a terminal (non-active) status, so its status can no longer be changed. structs.Deployment.Active() is false for statuses like cancelled, paused-terminal, failed, or successful.

Source

Thrown at nomad/state/state_store.go:4811

		if err := s.nestedUpsertEval(txn, index, req.Eval); err != nil {
			return err
		}
	}

	return txn.Commit()
}

// updateDeploymentStatusImpl is used to make deployment status updates
func (s *StateStore) updateDeploymentStatusImpl(index uint64, u *structs.DeploymentStatusUpdate, txn *txn) error {
	// Retrieve deployment
	ws := memdb.NewWatchSet()
	deployment, err := s.deploymentByIDImpl(ws, u.DeploymentID, txn)
	if err != nil {
		return err
	} else if deployment == nil {
		return fmt.Errorf("Deployment ID %q couldn't be updated as it does not exist", u.DeploymentID)
	} else if !deployment.Active() {
		return fmt.Errorf("Deployment %q has terminal status %q:", deployment.ID, deployment.Status)
	}

	// Apply the new status
	copy := deployment.Copy()
	copy.Status = u.Status
	copy.StatusDescription = u.StatusDescription
	copy.ModifyIndex = index
	copy.ModifyTime = u.UpdatedAt

	// check each TaskGroup for ProgressDeadline and reset RequireProgressBy
	// to ProgressDeadline if the deployment is running or paused. This is to
	// ensure that the RequireProgressBy is reset on a deployment that has been
	// paused and resumed.
	for _, dState := range copy.TaskGroups {
		if dState == nil || dState.ProgressDeadline == 0 {
			continue
		}

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Check deployment.Status before issuing an update; ignore updates to terminal deployments
2. Guard the caller: only send DeploymentStatusUpdate for active deployments
3. If an evaluation is stale, discard it instead of applying the status change
4. Reconcile with nomad deployment status to see the authoritative terminal state

Example fix

// before
req := &structs.DeploymentStatusUpdateRequest{Deployment: u}
srv.raft.Apply(req)
// after
if dep, _, _ := state.DeploymentByID(nil, u.DeploymentID); dep == nil || !dep.Active() {
  return nil // skip stale/terminal deployment update
}
srv.raft.Apply(req)

dep, _, err := state.DeploymentByID(nil, u.DeploymentID)
if err != nil { return err }
if dep != nil && !dep.Active() {
  return nil // terminal deployment: nothing to update
}

func deploymentActive(s *state.StateStore, id string) bool {
  d, _, err := s.DeploymentByID(nil, id)
  return err == nil && d != nil && d.Active()
}

if err := srv.raft.Apply(req).Error(); err != nil {
  if strings.Contains(err.Error(), "terminal status") {
    // stale update; discard rather than retry
  }
  return err
}

When it happens

Trigger: DeploymentStatusUpdate applied against a deployment already in a terminal state (e.g. a late evaluation update arrives after the deployment was cancelled or completed).

Common situations: Stale evaluations processed after deployment completion; duplicate status updates; users manually cancelling a deployment while an agent still reports to it.

Related errors

• deployment promotion cannot be undone
• deployment id not found: %q
• deployment %q references unknown job %q
• unknown deployment %q
• failed to retrieve latest deployment: %v

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/64cc98e7141528ff. Report an issue: GitHub.