hashicorp/nomad · error

failed to update job stability: %v

Error message

failed to update job stability: %v

What it means

When a deployment is marked successful, the state store also flips the parent job to stable via updateJobStabilityImpl. This error wraps any failure from that stability update, aborting the deployment status update so the job's stable flag never diverges from the deployment's terminal status.

Source

Thrown at nomad/state/state_store.go:4848

		if copy.Status == structs.DeploymentStatusPaused || copy.Status == structs.DeploymentStatusRunning {
			dState.RequireProgressBy = time.Unix(0, u.UpdatedAt).Add(dState.ProgressDeadline)
		}
	}

	// Insert the deployment
	if err := txn.Insert("deployment", copy); err != nil {
		return err
	}

	// Update the index
	if err := txn.Insert("index", &IndexEntry{"deployment", index}); err != nil {
		return fmt.Errorf("index update failed: %v", err)
	}

	// If the deployment is being marked as complete, set the job to stable.
	if copy.Status == structs.DeploymentStatusSuccessful {
		if err := s.updateJobStabilityImpl(index, copy.Namespace, copy.JobID, copy.JobVersion, true, txn); err != nil {
			return fmt.Errorf("failed to update job stability: %v", err)
		}
	}

	return nil
}

// UpdateJobStability updates the stability of the given job and version to the
// desired status.
func (s *StateStore) UpdateJobStability(index uint64, namespace, jobID string, jobVersion uint64, stable bool) error {
	txn := s.db.WriteTxn(index)
	defer txn.Abort()

	if err := s.updateJobStabilityImpl(index, namespace, jobID, jobVersion, stable, txn); err != nil {
		return err
	}

	return txn.Commit()
}

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Verify the job still exists at copy.Namespace/copy.JobID with nomad job status before/after promoting the deployment
2. Check the wrapped error to distinguish 'job not found' from an index insert failure
3. Re-register the job or re-run the deployment if the job version was overwritten by a newer submit

job, err := client.Jobs().Info(deployment.JobID)
if err != nil || job == nil {
  return fmt.Errorf("job %s missing; cannot mark deployment stable", deployment.JobID)
}

defer func() {
  if r := recover(); r != nil { log.Printf("stability update skipped: %v", r) }
}()

When it happens

Trigger: UpsertDeploymentStatusUpdate called with DeploymentStatusSuccessful for a deployment whose job no longer exists, whose JobVersion/JobID/Namespace no longer match, or whose job-stability txn.Insert fails.

Common situations: Job was purged or garbage-collected between deployment creation and completion; namespace mismatch after job re-registration; concurrent job deregistration racing the deployment promotion.

Related errors

• deployment promotion cannot be undone
• deployment id not found: %q
• deployment %q references unknown job %q
• unknown deployment %q
• failed to retrieve latest deployment: %v

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/da8f120e9526a883. Report an issue: GitHub.