{"record":{"id":"f7371eaf6f87765f","repo":"hashicorp/terraform","slug":"consul-cas-failed-with-transaction-errors-w","errorCode":null,"errorMessage":"consul CAS failed with transaction errors: %w","messagePattern":"consul CAS failed with transaction errors: %w","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"internal/backend/remote-state/consul/client.go","lineNumber":246,"sourceCode":"\t\t\t&consulapi.KVTxnOp{\n\t\t\t\tVerb:  verb,\n\t\t\t\tKey:   c.Path,\n\t\t\t\tValue: payload,\n\t\t\t\tIndex: c.modifyIndex,\n\t\t\t},\n\t\t}\n\n\t\tok, resp, _, err := kv.Txn(txOps, nil)\n\t\tif err != nil {\n\t\t\treturn err\n\t\t}\n\t\t// transaction was rolled back\n\t\tif !ok {\n\t\t\tvar resultErr error\n\t\t\tfor _, respError := range resp.Errors {\n\t\t\t\tresultErr = errors.Join(resultErr, errors.New(respError.What))\n\t\t\t}\n\t\t\treturn fmt.Errorf(\"consul CAS failed with transaction errors: %w\", resultErr)\n\t\t}\n\n\t\tif len(resp.Results) != 1 {\n\t\t\t// this probably shouldn't happen\n\t\t\treturn fmt.Errorf(\"expected on 1 response value, got: %d\", len(resp.Results))\n\t\t}\n\n\t\tc.modifyIndex = resp.Results[0].ModifyIndex\n\n\t\t// We remove all the old chunks\n\t\tcleanupOldChunks()\n\n\t\treturn nil\n\t}\n\n\tif err = store(payload); err == nil {\n\t\t// The payload was small enough to be stored\n\t\treturn diags","sourceCodeStart":228,"sourceCodeEnd":264,"githubUrl":"https://github.com/hashicorp/terraform/blob/d32a084675427f5ac3f7d2868578ef8b2c1dc525/internal/backend/remote-state/consul/client.go#L228-L264","documentation":"In RemoteClient.Put the state is written through a single-op Consul transaction using a CAS verb keyed off c.modifyIndex. If Consul rolls the transaction back (ok==false), each resp.Errors entry is joined into resultErr and wrapped. The dominant cause is a CAS failure: the key's ModifyIndex changed since the client last read it, so the compare-and-set precondition failed.","triggerScenarios":"store(payload) -> kv.Txn(txOps, nil) returns ok==false; the loop joins resp.Errors into resultErr and the function returns the wrapped error.","commonSituations":"Two Terraform processes (or an external writer) updated the same state path concurrently; a long-lived Terraform process has a stale in-memory modifyIndex; Consul ACL denies the kv:write at transaction time; a chunked cleanup or large txn exceeded limits and was rolled back.","solutions":["Ensure no concurrent terraform apply/plan against the same workspace and re-run; Terraform will re-Read, refresh modifyIndex, and retry the CAS.","Inspect the wrapped errors (resp.Errors[].What) in Consul logs for the specific sub-error.","Verify the backend ACL token has kv:write on the state prefix.","If the error names size limits, expect the client to retry in chunked mode; otherwise treat as concurrency."],"exampleFix":null,"handlingStrategy":"retry","validationCode":null,"typeGuard":null,"tryCatchPattern":"// CAS failures are typically transient concurrency; re-read and retry.\nconst maxRetries = 5\nvar lastErr error\nfor i := 0; i < maxRetries; i++ {\n    diags := remoteClient.Put(data)\n    if !diags.HasErrors() {\n        return nil\n    }\n    lastErr = diags.Err()\n    if !strings.Contains(lastErr.Error(), \"CAS failed\") {\n        return lastErr // not a CAS issue\n    }\n    time.Sleep(time.Duration(100<<i) * time.Millisecond) // backoff\n}\nreturn fmt.Errorf(\"state write failed after %d CAS retries: %w\", maxRetries, lastErr)","preventionTips":["Serialize Terraform runs per workspace (CI locks, Buildkite concurrency caps).","Re-run on CAS failures rather than assuming data loss.","Keep the backend ACL token's kv:write privilege current.","Watch Consul transaction error rates as a leading indicator of contention."],"tags":["consul","backend","concurrency","cas","state-write"],"backgroundTag":null,"analyzedSha":"d32a084675427f5ac3f7d2868578ef8b2c1dc525","analyzedAt":"2026-08-11T18:43:52.779Z","contentChangedAt":null,"schemaVersion":2},"datasetVersion":"2026-09-23T08:17:48.524Z"}