hashicorp/nomad · error

deployment %q references unknown job %q

Error message

deployment %q references unknown job %q

What it means

addLocked validates that the deployment's job still exists before creating its watcher. Nomad throws this when the deployment record points to a job that is no longer in the state store (e.g. the job was purged/GC'd while the deployment row remains).

Source

Thrown at nomad/deploymentwatcher/deployments_watcher.go:300

	// Already watched so just update the deployment
	if w, ok := w.watchers[d.ID]; ok {
		w.updateDeployment(d)
		return nil, nil
	}

	// Get the job the deployment is referencing
	snap, err := w.state.Snapshot()
	if err != nil {
		return nil, err
	}

	job, err := snap.JobByID(nil, d.Namespace, d.JobID)
	if err != nil {
		return nil, err
	}
	if job == nil {
		return nil, fmt.Errorf("deployment %q references unknown job %q", d.ID, d.JobID)
	}

	watcher := newDeploymentWatcher(w.ctx, w.queryLimiter, w.logger, w.state, d, job,
		w, w.deploymentRPC, w.jobRPC)
	w.watchers[d.ID] = watcher
	return watcher, nil
}

// remove stops watching a deployment. This can be because the deployment is
// complete or being deleted.
func (w *Watcher) remove(d *structs.Deployment) {
	w.l.Lock()
	defer w.l.Unlock()
	w.removeByIDLocked(d.ID)
}

func (w *Watcher) removeByIDLocked(id string) {
	// Not enabled so no-op

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Re-register the job (nomad job run) so the deployment's JobID resolves again
2. Verify the job exists in the deployment's namespace, not the default one
3. Cancel the orphaned deployment (nomad deployment fail) since it can no longer progress
4. Check that job GC isn't purging jobs while deployments still reference them

Example fix

// before: assuming default namespace
job, _ := snap.JobByID(nil, d.Namespace, d.JobID)
// after: ensure the job is registered in the deployment's namespace before re-adding
if job == nil { cancel deployment or re-register: nomad job run -namespace <d.Namespace> <d.JobID> }

job, err := state.JobByID(nil, dep.Namespace, dep.JobID)
if err == nil && job != nil {
    watcher.Add(dep) // safe
}

try {
    watcher.add(dep)
} catch (e) {
    if (e.message.includes('references unknown job')) {
        // job was GC'd/purged; fail the deployment and re-register the job
        failDeployment(dep.ID)
        registerJob(dep.Namespace, dep.JobID)
    } else { throw e }
}

When it happens

Trigger: add()/forceAdd() -> addLocked: snap.JobByID(nil, d.Namespace, d.JobID) returns nil for the deployment's namespace/job ID.

Common situations: Job deregistration followed by a deployment re-add after leader failover; namespace-scoped lookups where the job exists in a different namespace than queried; GC removing jobs faster than deployments.

Related errors

• deployment promotion cannot be undone
• deployment id not found: %q
• unknown deployment %q
• failed to retrieve latest deployment: %v
• Deployment ID %q couldn't be updated as it does not exist

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/35038058d6b3b976. Report an issue: GitHub.