{"record":{"id":"a571ad69032ac96b","repo":"astrid-runtime/astrid","slug":"write-device-revocation-key-id-cas-failed-cas","errorCode":null,"errorMessage":"write device revocation {key_id}: CAS failed: {cas_error}; fail-closed tombstone failed: {fallback_error}","messagePattern":"write device revocation (.+?): CAS failed: (.+?); fail-closed tombstone failed: (.+?)","errorType":"exception","errorClass":"anyhow::Error","httpStatus":null,"severity":"critical","filePath":"crates/astrid-gateway/src/revocations.rs","lineNumber":227,"sourceCode":"                encode_epoch(wanted),\n            )\n            .await\n        {\n            Ok(true) => return Ok(wanted),\n            Ok(false) => {},\n            Err(cas_error) => {\n                // Same fail-closed durability rule as principal deletion.\n                // MAX cannot be moved backward by a later CAS writer, so a\n                // successful fallback write is restart-durable. If this set\n                // also fails, no durable fence exists; the caller still gets\n                // the error and can retain only an in-memory MAX for the\n                // current process. Hydration aborts while KV is unavailable or\n                // corrupt, while an empty healthy KV cannot recreate the key.\n                store\n                    .set(REVOCATION_NAMESPACE, &key, encode_epoch(u64::MAX))\n                    .await\n                    .map_err(|fallback_error| {\n                        anyhow::anyhow!(\n                            \"write device revocation {key_id}: CAS failed: {cas_error}; fail-closed tombstone failed: {fallback_error}\"\n                        )\n                    })?;\n                // The tombstone keeps every bearer fail-closed, but the\n                // caller must still observe the publication failure. In\n                // particular, the HTTP revoke path cannot acknowledge 204\n                // when its normal CAS durability path faulted.\n                return Err(anyhow::anyhow!(\n                    \"write device revocation {key_id}: CAS failed: {cas_error}; fail-closed tombstone installed\"\n                ));\n            },\n        }\n    }\n}\n\nfn publish_device_epoch<S: BuildHasher>(\n    revoked_key_ids: &RwLock<HashMap<String, u64, S>>,\n    key_id: &str,","sourceCodeStart":209,"sourceCodeEnd":245,"githubUrl":"https://github.com/astrid-runtime/astrid/blob/affd8760f44190dbdfbec23403f4c4b642c33112/crates/astrid-gateway/src/revocations.rs#L209-L245","documentation":"When the CAS write of a device revocation epoch fails, record_device_max falls back to writing a u64::MAX tombstone so every bearer of that key fails closed. This error is thrown when BOTH the CAS write and the fallback tombstone write fail, meaning the device revocation could not be persisted in any form.","triggerScenarios":"store CAS failure (cas_error) followed by a failing store.set of encode_epoch(u64::MAX) (fallback_error), while applying a device revocation through apply_device_revocation.","commonSituations":"KV backend becomes unavailable or read-only between the CAS attempt and the fallback; disk-full or quota limits on the store; sustained partition or store credential rotation mid-request.","solutions":["Restore the store and re-run apply_device_revocation; neither the epoch nor the tombstone was persisted","Do NOT return success (e.g. HTTP 204) to the caller — the code comments explicitly require the caller to observe this failure","Check store-side errors (read-only, disk, auth) revealed by the two wrapped errors","Keep the revoked key rejected in-process until the durable write succeeds"],"exampleFix":"// before: treat any tombstone write as success\nstore.set(REVOCATION_NAMESPACE, &key, encode_epoch(u64::MAX)).await?;\nreturn Ok(());\n// after: propagate failure so the HTTP path does not 204\nmatch record_device_max(&store, key_id, epoch).await {\n    Ok(published) => Ok((StatusCode::NO_CONTENT, published)),\n    Err(e) => Err(StatusCode::SERVICE_UNAVAILABLE), // caller retries revocation\n}","handlingStrategy":"fallback","validationCode":"// Ensure the store accepts writes before revoking devices\npub async fn assert_device_store_writable(store: &dyn KvStore) -> anyhow::Result<()> {\n    store.set(REVOCATION_NAMESPACE, \"__write_probe__\", b\"1\".to_vec()).await.map(|_| ())\n}","typeGuard":null,"tryCatchPattern":"match record_device_max(&store, key_id, epoch).await {\n    Err(e) if e.to_string().contains(\"tombstone failed\") => {\n        // Zero durability: neither epoch nor tombstone persisted.\n        error!(key_id, %e, \"device revocation unpublished; denying key in-process\");\n        local_key_fence.lock().insert(key_id.to_string());\n        Err(StatusCode::SERVICE_UNAVAILABLE) // never 204\n    }\n    other => other.map(|_| StatusCode::NO_CONTENT),\n}","preventionTips":["Never map this error to an HTTP 2xx; the revocation did not persist","Track local fences for keys whose revocation failed and republish after recovery","Monitor store disk/quota/read-only state to catch dual-write failures early","Exercise CAS + tombstone paths in staging fault-injection tests"],"tags":["kv-store","cas","revocation","fail-closed"],"backgroundTag":"database-write-failed","analyzedSha":"affd8760f44190dbdfbec23403f4c4b642c33112","analyzedAt":"2026-09-09T21:28:12.402Z","contentChangedAt":"2026-09-09T21:28:12.402Z","schemaVersion":2},"datasetVersion":"2026-09-17T15:17:12.973Z"}