hashicorp/nomad · error

deployment promotion cannot be undone

Error message

deployment promotion cannot be undone

What it means

When UpsertDeployment updates an existing deployment, upsertDeploymentImpl detects write skew: the stored deployment has Promoted=true for some task group while the incoming deployment flips it back to false. Since promotions are irreversible by design, the state store rejects the update rather than silently regressing deployment state.

Source

Thrown at nomad/state/state_store.go:595

	return txn.Commit()
}

func (s *StateStore) upsertDeploymentImpl(index uint64, deployment *structs.Deployment, txn *txn) error {
	// Check if the deployment already exists
	raw, err := txn.First("deployment", "id", deployment.ID)
	if err != nil {
		return fmt.Errorf("deployment lookup failed: %v", err)
	}

	// Setup the indexes and timestamps correctly
	if raw != nil {
		existing := raw.(*structs.Deployment)
		deployment.CreateIndex = existing.CreateIndex
		deployment.ModifyIndex = index
		for tg, dstate := range existing.TaskGroups {
			newDstate := deployment.TaskGroups[tg]
			if dstate != nil && newDstate != nil && dstate.Promoted && !newDstate.Promoted {
				return errors.New("deployment promotion cannot be undone") // write skew
			}
		}
	} else {
		deployment.CreateIndex = index
		deployment.ModifyIndex = index
	}
	// Insert the deployment
	if err := txn.Insert("deployment", deployment); err != nil {
		return err
	}

	// Update the indexes table for deployment
	if err := txn.Insert("index", &IndexEntry{"deployment", index}); err != nil {
		return fmt.Errorf("index update failed: %v", err)
	}

	// If the deployment is being marked as complete, set the job to stable.
	if deployment.Status == structs.DeploymentStatusSuccessful {

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Never write a deployment object with Promoted downgraded from the stored value; fetch the current deployment first and merge
2. Use a fresh deployment ID for new deployment lifecycles instead of reusing an existing one
3. Refetch the deployment inside the same transaction immediately before the update to avoid operating on stale data

Example fix

// before
deployment.TaskGroups["web"].Promoted = false
stateStore.UpsertDeployment(1000, deployment)
// after
existing, _ := stateStore.DeploymentByID(nil, deployment.ID)
for tg, ds := range deployment.TaskGroups {
    ds.Promoted = existing.TaskGroups[tg].Promoted || ds.Promoted
}
stateStore.UpsertDeployment(index, deployment)

func promotable(next, existing *structs.Deployment) bool {
    for tg, ds := range existing.TaskGroups {
        if ds != nil && ds.Promoted && next.TaskGroups[tg] != nil && !next.TaskGroups[tg].Promoted {
            return false
        }
    }
    return true
}

func promotionNotUndone(prev, next *structs.Deployment) bool {
    for tg, st := range prev.TaskGroups {
        n := next.TaskGroups[tg]
        if st != nil && n != nil && st.Promoted && !n.Promoted { return false }
    }
    return true
}

if err := store.UpsertDeployment(idx, dep); err != nil && strings.Contains(err.Error(), "promotion cannot be undone") {
    // refetch deployment, merge promotion flags, retry
}

When it happens

Trigger: Calling StateStore.UpsertDeployment (directly or via UpsertPlanResults) with a Deployment that reuses an existing deployment ID but with Promoted=false on a task group whose stored state has Promoted=true.

Common situations: Buggy automation that writes cached/stale deployment snapshots back to state; custom tooling manipulating deployments via internal state APIs; race between promotion and another writer operating on an outdated copy of the deployment.

Related errors

• deployment id not found: %q
• deployment %q references unknown job %q
• unknown deployment %q
• failed to retrieve latest deployment: %v
• Deployment ID %q couldn't be updated as it does not exist

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/165a15c60493ea23. Report an issue: GitHub.