hashicorp/nomad · error

deployment %q is terminal

Error message

deployment %q is terminal

What it means

addLocked refuses to start a watcher for a deployment whose status is terminal (not Active()). Nomad throws this because only active deployments need live watching; requesting a watcher for a finished/cancelled/paused-to-terminal deployment is a caller bug.

Source

Thrown at nomad/deploymentwatcher/deployments_watcher.go:280

// add adds a deployment to the watch list
func (w *Watcher) add(d *structs.Deployment) error {
	w.l.Lock()
	defer w.l.Unlock()
	_, err := w.addLocked(d)
	return err
}

// addLocked adds a deployment to the watch list and should only be called when
// locked. Creating the deploymentWatcher starts a go routine to .watch() it
func (w *Watcher) addLocked(d *structs.Deployment) (*deploymentWatcher, error) {
	// Not enabled so no-op
	if !w.enabled {
		return nil, nil
	}

	if !d.Active() {
		return nil, fmt.Errorf("deployment %q is terminal", d.ID)
	}

	// Already watched so just update the deployment
	if w, ok := w.watchers[d.ID]; ok {
		w.updateDeployment(d)
		return nil, nil
	}

	// Get the job the deployment is referencing
	snap, err := w.state.Snapshot()
	if err != nil {
		return nil, err
	}

	job, err := snap.JobByID(nil, d.Namespace, d.JobID)
	if err != nil {
		return nil, err
	}

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Check `nomad deployment status <id>` — if the deployment is terminal, no watcher is needed; ignore the error
2. Only call force-add/reevaluation on active deployments
3. Filter deployments by Active() before attempting to register a watcher (e.g. after leader election)
4. Create a new deployment by updating the job instead of trying to revive a terminal one

Example fix

// before
w.addLocked(dep)
// after
if dep.Active() {
    w.addLocked(dep)
}

// guard before re-adding a watcher
d, err := state.DeploymentByID(nil, id)
if err == nil && d != nil && d.Active() {
    watcher.Add(d) // safe
}

func canWatch(d *structs.Deployment) bool { return d != nil && d.Active() }

try {
    watcher.add(dep)
} catch (e) {
    if (e.message.includes('is terminal')) {
        // skip: deployment finished; no action needed
        return null
    }
    throw e
}

When it happens

Trigger: add() or forceAdd() call addLocked with a deployment d where d.Active() is false (status failed, cancelled, or successful).

Common situations: Re-adding watchers after leader failover for deployments that completed before the election; calling force-add/reevaluate APIs against an old deployment ID; replaying state snapshots containing terminal deployments.

Related errors

• users: release of unused uid/gid
• deployment promotion cannot be undone
• can't cancel terminal deployment
• can't fail terminal deployment
• can't pause terminal deployment

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/e2d2a09662de475f. Report an issue: GitHub.