{"record":{"id":"f2addee6d4d4c654","repo":"astrid-runtime/astrid","slug":"write-principal-revocation-principal-cas-failed","errorCode":null,"errorMessage":"write principal revocation {principal}: CAS failed: {cas_error}; fail-closed tombstone failed: {fallback_error}","messagePattern":"write principal revocation (.+?): CAS failed: (.+?); fail-closed tombstone failed: (.+?)","errorType":"exception","errorClass":"anyhow::Error","httpStatus":null,"severity":"critical","filePath":"crates/astrid-gateway/src/revocations.rs","lineNumber":163,"sourceCode":"            .await\n        {\n            Ok(true) => return Ok(wanted),\n            Ok(false) => {},\n            Err(cas_error) => {\n                // A successful delete must never become reversible merely\n                // because the monotonic CAS path is unavailable. Persist an\n                // unconditional maximum-epoch tombstone: this intentionally\n                // sacrifices alias reuse until operator repair, but it is\n                // monotonic under every concurrent writer and survives a\n                // restart when this fallback write succeeds. If the fallback\n                // write also fails, no durable fence exists; propagate that\n                // loss so the caller can retain only a process-local fence and\n                // avoid claiming restart durability.\n                store\n                    .set(REVOCATION_NAMESPACE, &key, encode_epoch(u64::MAX))\n                    .await\n                    .map_err(|fallback_error| {\n                        anyhow::anyhow!(\n                            \"write principal revocation {principal}: CAS failed: {cas_error}; fail-closed tombstone failed: {fallback_error}\"\n                        )\n                    })?;\n                return Ok(u64::MAX);\n            },\n        }\n    }\n}\n\n/// Record the maximum device revocation epoch durably using the same CAS/max\n/// rule as principal revocations.\n///\n/// A successful CAS, or a successful maximum-epoch fallback write after a\n/// CAS error, leaves a fence that startup hydration can restore. The function\n/// still returns an error after any CAS error so the HTTP caller withholds\n/// `204`, even when the fallback tombstone succeeded. If both writes fail,\n/// there is no durable fence; the caller may install a process-local maximum,\n/// but a later healthy empty KV cannot reconstruct it.","sourceCodeStart":145,"sourceCodeEnd":181,"githubUrl":"https://github.com/astrid-runtime/astrid/blob/affd8760f44190dbdfbec23403f4c4b642c33112/crates/astrid-gateway/src/revocations.rs#L145-L181","documentation":"After a compare-and-swap (CAS) write of the principal revocation epoch fails, record_principal_max attempts a fail-closed fallback: it writes an epoch of u64::MAX (a tombstone that revokes everything). This error is thrown when BOTH the CAS write and the fallback tombstone write fail, meaning the revocation could not be published at all.","triggerScenarios":"store.compare_exchange (or equivalent) returns cas_error AND the follow-up store.set of encode_epoch(u64::MAX) returns fallback_error, while recording a principal max epoch via migrate_legacy_file or spawn_watcher.","commonSituations":"KV backend going down mid-operation (CAS fails, then the tombstone set also fails); backend in read-only mode (disk full, standby instance); sustained network partition; store auth/permission revocation during operation.","solutions":["Restore the KV backend (health, capacity, write permissions) and re-run record_principal_max to publish the epoch","Treat the principal as NOT durably revoked: re-issue the revocation after recovery, since neither the epoch nor the tombstone landed","Check backend logs for read-only/disk-full/auth errors indicated by the two wrapped errors","Consider an in-process fence for the principal until the durable write succeeds, as suggested by the surrounding code comments"],"exampleFix":"// before: single-shot write\nstore.compare_exchange(...).await?;\n// after: verify durability and re-run on failure\nmatch record_principal_max(&store, principal, epoch).await {\n    Ok(published) => info!(principal, published, \"revocation durable\"),\n    Err(e) => {\n        error!(principal, %e, \"revocation NOT durable; fencing locally and retrying\");\n        local_fence.lock().insert(principal.clone(), epoch);\n    }\n}","handlingStrategy":"fallback","validationCode":"// Verify write access before attempting CAS updates\npub async fn assert_store_writable(store: &dyn KvStore) -> anyhow::Result<()> {\n    let probe_key = \"__write_probe__\";\n    store.set(REVOCATION_NAMESPACE, probe_key, b\"1\".to_vec()).await?;\n    store.delete(REVOCATION_NAMESPACE, probe_key).await\n}","typeGuard":null,"tryCatchPattern":"match record_principal_max(&store, principal, epoch).await {\n    Err(e) if e.to_string().contains(\"tombstone failed\") => {\n        // Neither CAS nor tombstone landed: revocation NOT durable.\n        error!(principal, %e, \"revocation unpublished; retaining process-local fence\");\n        local_fence.lock().insert(principal.clone(), epoch);\n        schedule_republish(principal, epoch);\n    }\n    other => other?,\n}","preventionTips":["Monitor KV backend capacity, read-only mode, and auth so both writes never fail together","Keep a process-local fence for principals until a durable write is confirmed","Alert on any 'tombstone failed' error — it means zero durability for that revocation","Run the CAS retry loop against a store with adequate write QPS to avoid cascading faults"],"tags":["kv-store","cas","revocation","fail-closed"],"backgroundTag":"database-write-failed","analyzedSha":"affd8760f44190dbdfbec23403f4c4b642c33112","analyzedAt":"2026-09-09T21:28:12.402Z","contentChangedAt":"2026-09-09T21:28:12.402Z","schemaVersion":2},"datasetVersion":"2026-09-17T15:17:12.973Z"}