{"record":{"id":"165a15c60493ea23","repo":"hashicorp/nomad","slug":"deployment-promotion-cannot-be-undone","errorCode":null,"errorMessage":"deployment promotion cannot be undone","messagePattern":"deployment promotion cannot be undone","errorType":"error_code","errorClass":null,"httpStatus":null,"severity":"error","filePath":"nomad/state/state_store.go","lineNumber":595,"sourceCode":"\treturn txn.Commit()\n}\n\nfunc (s *StateStore) upsertDeploymentImpl(index uint64, deployment *structs.Deployment, txn *txn) error {\n\t// Check if the deployment already exists\n\traw, err := txn.First(\"deployment\", \"id\", deployment.ID)\n\tif err != nil {\n\t\treturn fmt.Errorf(\"deployment lookup failed: %v\", err)\n\t}\n\n\t// Setup the indexes and timestamps correctly\n\tif raw != nil {\n\t\texisting := raw.(*structs.Deployment)\n\t\tdeployment.CreateIndex = existing.CreateIndex\n\t\tdeployment.ModifyIndex = index\n\t\tfor tg, dstate := range existing.TaskGroups {\n\t\t\tnewDstate := deployment.TaskGroups[tg]\n\t\t\tif dstate != nil && newDstate != nil && dstate.Promoted && !newDstate.Promoted {\n\t\t\t\treturn errors.New(\"deployment promotion cannot be undone\") // write skew\n\t\t\t}\n\t\t}\n\t} else {\n\t\tdeployment.CreateIndex = index\n\t\tdeployment.ModifyIndex = index\n\t}\n\t// Insert the deployment\n\tif err := txn.Insert(\"deployment\", deployment); err != nil {\n\t\treturn err\n\t}\n\n\t// Update the indexes table for deployment\n\tif err := txn.Insert(\"index\", &IndexEntry{\"deployment\", index}); err != nil {\n\t\treturn fmt.Errorf(\"index update failed: %v\", err)\n\t}\n\n\t// If the deployment is being marked as complete, set the job to stable.\n\tif deployment.Status == structs.DeploymentStatusSuccessful {","sourceCodeStart":577,"sourceCodeEnd":613,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/nomad/state/state_store.go#L577-L613","documentation":"When UpsertDeployment updates an existing deployment, upsertDeploymentImpl detects write skew: the stored deployment has Promoted=true for some task group while the incoming deployment flips it back to false. Since promotions are irreversible by design, the state store rejects the update rather than silently regressing deployment state.","triggerScenarios":"Calling StateStore.UpsertDeployment (directly or via UpsertPlanResults) with a Deployment that reuses an existing deployment ID but with Promoted=false on a task group whose stored state has Promoted=true.","commonSituations":"Buggy automation that writes cached/stale deployment snapshots back to state; custom tooling manipulating deployments via internal state APIs; race between promotion and another writer operating on an outdated copy of the deployment.","solutions":["Never write a deployment object with Promoted downgraded from the stored value; fetch the current deployment first and merge","Use a fresh deployment ID for new deployment lifecycles instead of reusing an existing one","Refetch the deployment inside the same transaction immediately before the update to avoid operating on stale data"],"exampleFix":"// before\ndeployment.TaskGroups[\"web\"].Promoted = false\nstateStore.UpsertDeployment(1000, deployment)\n// after\nexisting, _ := stateStore.DeploymentByID(nil, deployment.ID)\nfor tg, ds := range deployment.TaskGroups {\n    ds.Promoted = existing.TaskGroups[tg].Promoted || ds.Promoted\n}\nstateStore.UpsertDeployment(index, deployment)","handlingStrategy":"validation","validationCode":"func promotable(next, existing *structs.Deployment) bool {\n    for tg, ds := range existing.TaskGroups {\n        if ds != nil && ds.Promoted && next.TaskGroups[tg] != nil && !next.TaskGroups[tg].Promoted {\n            return false\n        }\n    }\n    return true\n}","typeGuard":"func promotionNotUndone(prev, next *structs.Deployment) bool {\n    for tg, st := range prev.TaskGroups {\n        n := next.TaskGroups[tg]\n        if st != nil && n != nil && st.Promoted && !n.Promoted { return false }\n    }\n    return true\n}","tryCatchPattern":"if err := store.UpsertDeployment(idx, dep); err != nil && strings.Contains(err.Error(), \"promotion cannot be undone\") {\n    // refetch deployment, merge promotion flags, retry\n}","preventionTips":["Always read the current deployment before writing an update","Never write stale cached Deployment objects back to state","Treat Promoted=true as immutable in your automation"],"tags":["nomad","state-store","deployment","write-skew","consistency"],"backgroundTag":"state-write-skew","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}