hashicorp/nomad · error

deployment id not found: %q

Error message

deployment id not found: %q

What it means

deploymentwatcher fails the shouldFail check when the deployment being watched cannot be found in the state store. Nomad throws this because a watcher can outlive its deployment record — most commonly after a system garbage collection removes the deployment while the watcher is still active. The watcher cannot evaluate whether the deployment should fail, so it returns an error instead of a decision.

Source

Thrown at nomad/deploymentwatcher/deployment_watcher.go:667

	return res, nil
}

// shouldFail returns whether the job should be failed and whether it should
// rolled back to an earlier stable version by examining the allocations in the
// deployment.
func (w *deploymentWatcher) shouldFail() (fail, rollback bool, err error) {
	snap, err := w.state.Snapshot()
	if err != nil {
		return false, false, err
	}

	d, err := snap.DeploymentByID(nil, w.deploymentID)
	if err != nil {
		return false, false, err
	}
	if d == nil {
		// The deployment wasn't in the state store, possibly due to a system gc
		return false, false, fmt.Errorf("deployment id not found: %q", w.deploymentID)
	}

	fail = false
	if d.Status == structs.DeploymentStatusPaused {
		return false, false, nil
	}
	for tg, dstate := range d.TaskGroups {
		// If we are in a canary state we fail if there aren't enough healthy
		// allocs to satisfy DesiredCanaries
		if dstate.DesiredCanaries > 0 && !dstate.Promoted {
			if dstate.HealthyAllocs >= dstate.DesiredCanaries {
				continue
			}
		} else if dstate.HealthyAllocs >= dstate.DesiredTotal {
			continue
		}

		// We have failed this TG

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Verify the deployment still exists with `nomad deployment status <id>`; if it was GC'd, this error is expected and the watcher will be cleaned up
2. Re-register or re-run the job to create a fresh deployment and watcher
3. Check state store GC settings (default eval/deployment GC intervals) if this fires too eagerly
4. Restart the Nomad server leader if stale watchers persist after GC

Example fix

// no caller-side fix; confirm deployment exists before relying on its watcher
nomad deployment status <deployment-id>  # if 'not found', the deployment was GC'd

nomad deployment status <deployment-id>  # must return an ACTIVE deployment, not 'not found'

// Go-style guard before acting on a deployment
func deploymentUsable(d *structs.Deployment) bool { return d != nil && d.Active() }

try {
    watcher.shouldFail(depID)
} catch (e) {
    if (e.message.includes('deployment id not found')) {
        // deployment GC'd: treat as terminal, re-run job for a fresh deployment
        rerunJob(jobID)
    } else { throw e }
}

When it happens

Trigger: watch() calls shouldFail, which does snap.DeploymentByID(nil, w.deploymentID); if the state store returns nil for the deployment (e.g. it was GC'd out of state between watcher creation and this evaluation), the error is returned.

Common situations: A long-running deployment whose job was deregistered and state was garbage collected; compacted/GC'd state on busy clusters; races where the deployment finishes and is GC'd while its watcher is still ticking.

Related errors

• deployment promotion cannot be undone
• deployment %q references unknown job %q
• unknown deployment %q
• failed to retrieve latest deployment: %v
• csi_plugins delete error: %v

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/d2ca8ee80624cdef. Report an issue: GitHub.