{"record":{"id":"23a62e2b4f7e64e5","repo":"astrid-runtime/astrid","slug":"write-device-revocation-key-id-cas-failed-cas-23a62e","errorCode":null,"errorMessage":"write device revocation {key_id}: CAS failed: {cas_error}; fail-closed tombstone installed","messagePattern":"write device revocation (.+?): CAS failed: (.+?); fail-closed tombstone installed","errorType":"exception","errorClass":"anyhow::Error","httpStatus":null,"severity":"error","filePath":"crates/astrid-gateway/src/revocations.rs","lineNumber":235,"sourceCode":"                // MAX cannot be moved backward by a later CAS writer, so a\n                // successful fallback write is restart-durable. If this set\n                // also fails, no durable fence exists; the caller still gets\n                // the error and can retain only an in-memory MAX for the\n                // current process. Hydration aborts while KV is unavailable or\n                // corrupt, while an empty healthy KV cannot recreate the key.\n                store\n                    .set(REVOCATION_NAMESPACE, &key, encode_epoch(u64::MAX))\n                    .await\n                    .map_err(|fallback_error| {\n                        anyhow::anyhow!(\n                            \"write device revocation {key_id}: CAS failed: {cas_error}; fail-closed tombstone failed: {fallback_error}\"\n                        )\n                    })?;\n                // The tombstone keeps every bearer fail-closed, but the\n                // caller must still observe the publication failure. In\n                // particular, the HTTP revoke path cannot acknowledge 204\n                // when its normal CAS durability path faulted.\n                return Err(anyhow::anyhow!(\n                    \"write device revocation {key_id}: CAS failed: {cas_error}; fail-closed tombstone installed\"\n                ));\n            },\n        }\n    }\n}\n\nfn publish_device_epoch<S: BuildHasher>(\n    revoked_key_ids: &RwLock<HashMap<String, u64, S>>,\n    key_id: &str,\n    epoch: u64,\n) -> u64 {\n    let mut guard = revoked_key_ids\n        .write()\n        .expect(\"revoked-key-id map poisoned — fail-stop\");\n    let previous = guard.get(key_id).copied().unwrap_or(0);\n    let published = previous.max(epoch);\n    if published > previous {","sourceCodeStart":217,"sourceCodeEnd":253,"githubUrl":"https://github.com/astrid-runtime/astrid/blob/affd8760f44190dbdfbec23403f4c4b642c33112/crates/astrid-gateway/src/revocations.rs#L217-L253","documentation":"This error signals a partial failure: the CAS write of the device revocation epoch failed, but the fail-closed tombstone (epoch u64::MAX) WAS successfully installed, so all bearers of the key are rejected even though the intended epoch value is not durable. It is thrown so callers (notably the HTTP revoke path) never acknowledge success while the normal durability path faulted.","triggerScenarios":"record_device_max's store compare-and-swap returns cas_error and the subsequent store.set of encode_epoch(u64::MAX) succeeds — returned as an Err from apply_device_revocation.","commonSituations":"Concurrent writers causing repeated CAS contention plus a genuine CAS fault; transient store error on the conditional write that clears by the time the unconditional tombstone set runs; store instability under load.","solutions":["Return the revocation as failed/pending to the client (do not send 204), even though the key is effectively revoked","After the store stabilizes, re-run record_device_max to publish the intended epoch, replacing the overly broad u64::MAX tombstone","Inspect cas_error for contention or backend faults; reduce hot-key contention or fix backend issues","Document that the key is fail-closed revoked in the meantime so operators don't assume it still works"],"exampleFix":"// before\nreturn Err(anyhow::anyhow!(\"write device revocation {key_id}: CAS failed: {cas_error}; fail-closed tombstone installed\"));\n// after: caller-side handling of the partial-failure error\nmatch apply_device_revocation(state, key_id, epoch).await {\n    Ok(_) => StatusCode::NO_CONTENT,\n    Err(e) if e.to_string().contains(\"fail-closed tombstone installed\") => {\n        warn!(key_id, \"revoked fail-closed; epoch publication pending\");\n        StatusCode::ACCEPTED // retry publishing the exact epoch later\n    }\n    Err(e) => StatusCode::SERVICE_UNAVAILABLE,\n}","handlingStrategy":"try-catch","validationCode":null,"typeGuard":null,"tryCatchPattern":"match apply_device_revocation(state, key_id, epoch).await {\n    Ok(_) => StatusCode::NO_CONTENT,\n    Err(e) if e.to_string().contains(\"fail-closed tombstone installed\") => {\n        warn!(key_id, \"key revoked fail-closed; intended epoch not durable\");\n        queue_epoch_republish(key_id, epoch); // replace u64::MAX tombstone later\n        StatusCode::ACCEPTED\n    }\n    Err(_) => StatusCode::SERVICE_UNAVAILABLE,\n}","preventionTips":["Treat this error as 'revoked but pending publication', not success and not total failure","After store recovery, re-run record_device_max to publish the exact intended epoch","Reduce CAS hot-key contention (shard writers, add jitter) to lower cas_error rates","Log and alert on cas_error even when the tombstone succeeded, to catch backend instability"],"tags":["kv-store","cas","revocation","fail-closed","partial-failure"],"backgroundTag":"database-write-failed","analyzedSha":"affd8760f44190dbdfbec23403f4c4b642c33112","analyzedAt":"2026-09-09T21:28:12.402Z","contentChangedAt":"2026-09-09T21:28:12.402Z","schemaVersion":2},"datasetVersion":"2026-09-17T15:17:12.973Z"}