hashicorp/nomad · error

deployment lookup for job %s failed: %v

Error message

deployment lookup for job %s failed: %v

What it means

DeleteJobTxn (nomad/state/state_store.go:2034) wraps an error from s.DeploymentsByJobID(nil, namespace, job.ID, true) while cleaning up deployments of the job being deleted. The deployment listing failed (a read using a separate read-only txn), so the deregistration aborts with 'deployment lookup for job <id> failed'. The wrapped %v carries the lookup root cause.

Source

Thrown at nomad/state/state_store.go:2034

	}

	// Delete the job
	if err := txn.Delete("jobs", existing); err != nil {
		return fmt.Errorf("job delete failed: %v", err)
	}
	if err := txn.Insert("index", &IndexEntry{"jobs", index}); err != nil {
		return fmt.Errorf("index update failed: %v", err)
	}

	// Delete the job versions
	if err := s.deleteJobVersions(index, job, txn); err != nil {
		return err
	}

	// Delete job deployments
	deployments, err := s.DeploymentsByJobID(nil, namespace, job.ID, true)
	if err != nil {
		return fmt.Errorf("deployment lookup for job %s failed: %v", job.ID, err)
	}

	deploymentIDs := []string{}
	for _, d := range deployments {
		deploymentIDs = append(deploymentIDs, d.ID)
	}

	if err := s.DeleteDeploymentTxn(index, deploymentIDs, txn); err != nil {
		return err
	}

	// Mark all "pending" evals for this job as "complete"
	evals, err := s.EvalsByJob(nil, namespace, job.ID)
	if err != nil {
		return fmt.Errorf("eval lookup for job %s failed: %v", job.ID, err)
	}

	for _, eval := range evals {

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Retry the deregistration; the txn rolled back
2. Inspect the wrapped error from DeploymentsByJobID in server logs
3. Restart the server agent to rebuild state from Raft; verify snapshot integrity if a restore preceded the errors
4. Ensure all servers run the same Nomad version to avoid struct decoding mismatches

Example fix

// before
return s.DeleteJobTxn(idx, ns, jobID, txn)
// after
err := s.DeleteJobTxn(idx, ns, jobID, txn)
if err != nil && strings.Contains(err.Error(), "deployment lookup for job") {
    return retryWithBackoff(func() error { return s.DeleteJobTxn(idx, ns, jobID, txn) })
}
return err

// pre-flight the same read the delete path uses
_, err := stateStore.DeploymentsByJobID(nil, ns, jobID, true)
if err != nil {
    return fmt.Errorf("deployment reads unhealthy; fix before deregistering: %w", err)
}

func hasDeployments(list []*structs.Deployment) bool { return len(list) > 0 }

if err := s.DeleteJobTxn(idx, ns, jobID, txn); err != nil {
    if strings.Contains(err.Error(), "deployment lookup for job") {
        return retry(3, backoff, func() error { return s.DeleteJobTxn(idx, ns, jobID, txn) })
    }
    return err
}

When it happens

Trigger: Deleting a job that had deployments, when the DeploymentsByJobID iterator errors — MemDB read failure, wrong-type rows in the deployments table, or an iterator fault.

Common situations: Corrupted deployment entries restored from a snapshot; servers under memory pressure during deregistration storms; version mismatches in deployment structs after upgrades.

Related errors

• launch lookup failed: %v
• scaling_policy lookup failed: %v
• ACL binding rules lookup failed: %v
• ACL auth method lookup failed: %v
• error parsing: root should be an object

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/c11cc158392008fc. Report an issue: GitHub.