astrid-runtime/astrid · error · anyhow::Error

write device revocation

Error message

write device revocation {key_id}: CAS failed: {cas_error}; fail-closed tombstone installed

What it means

This error signals a partial failure: the CAS write of the device revocation epoch failed, but the fail-closed tombstone (epoch u64::MAX) WAS successfully installed, so all bearers of the key are rejected even though the intended epoch value is not durable. It is thrown so callers (notably the HTTP revoke path) never acknowledge success while the normal durability path faulted.

Solutions

Return the revocation as failed/pending to the client (do not send 204), even though the key is effectively revoked
After the store stabilizes, re-run record_device_max to publish the intended epoch, replacing the overly broad u64::MAX tombstone
Inspect cas_error for contention or backend faults; reduce hot-key contention or fix backend issues
Document that the key is fail-closed revoked in the meantime so operators don't assume it still works

Example fix

// before
return Err(anyhow::anyhow!("write device revocation {key_id}: CAS failed: {cas_error}; fail-closed tombstone installed"));
// after: caller-side handling of the partial-failure error
match apply_device_revocation(state, key_id, epoch).await {
    Ok(_) => StatusCode::NO_CONTENT,
    Err(e) if e.to_string().contains("fail-closed tombstone installed") => {
        warn!(key_id, "revoked fail-closed; epoch publication pending");
        StatusCode::ACCEPTED // retry publishing the exact epoch later
    }
    Err(e) => StatusCode::SERVICE_UNAVAILABLE,
}

Defensive patterns

Strategy: try-catch

Try / catch

match apply_device_revocation(state, key_id, epoch).await {
    Ok(_) => StatusCode::NO_CONTENT,
    Err(e) if e.to_string().contains("fail-closed tombstone installed") => {
        warn!(key_id, "key revoked fail-closed; intended epoch not durable");
        queue_epoch_republish(key_id, epoch); // replace u64::MAX tombstone later
        StatusCode::ACCEPTED
    }
    Err(_) => StatusCode::SERVICE_UNAVAILABLE,
}

Prevention

Treat this error as 'revoked but pending publication', not success and not total failure
After store recovery, re-run record_device_max to publish the exact intended epoch
Reduce CAS hot-key contention (shard writers, add jitter) to lower cas_error rates
Log and alert on cas_error even when the tombstone succeeded, to catch backend instability

When it happens

Trigger: record_device_max's store compare-and-swap returns cas_error and the subsequent store.set of encode_epoch(u64::MAX) succeeds — returned as an Err from apply_device_revocation.

Common situations: Concurrent writers causing repeated CAS contention plus a genuine CAS fault; transient store error on the conditional write that clears by the time the unconditional tombstone set runs; store instability under load.

Related errors

AI-assisted analysis of astrid-runtime/astrid@affd8760f4 (2026-09-09). Data as JSON: /api/errors/23a62e2b4f7e64e5. Report an issue: GitHub.

Appendix: source

Thrown at crates/astrid-gateway/src/revocations.rs:235

                // MAX cannot be moved backward by a later CAS writer, so a
                // successful fallback write is restart-durable. If this set
                // also fails, no durable fence exists; the caller still gets
                // the error and can retain only an in-memory MAX for the
                // current process. Hydration aborts while KV is unavailable or
                // corrupt, while an empty healthy KV cannot recreate the key.
                store
                    .set(REVOCATION_NAMESPACE, &key, encode_epoch(u64::MAX))
                    .await
                    .map_err(|fallback_error| {
                        anyhow::anyhow!(
                            "write device revocation {key_id}: CAS failed: {cas_error}; fail-closed tombstone failed: {fallback_error}"
                        )
                    })?;
                // The tombstone keeps every bearer fail-closed, but the
                // caller must still observe the publication failure. In
                // particular, the HTTP revoke path cannot acknowledge 204
                // when its normal CAS durability path faulted.
                return Err(anyhow::anyhow!(
                    "write device revocation {key_id}: CAS failed: {cas_error}; fail-closed tombstone installed"
                ));
            },
        }
    }
}

fn publish_device_epoch<S: BuildHasher>(
    revoked_key_ids: &RwLock<HashMap<String, u64, S>>,
    key_id: &str,
    epoch: u64,
) -> u64 {
    let mut guard = revoked_key_ids
        .write()
        .expect("revoked-key-id map poisoned — fail-stop");
    let previous = guard.get(key_id).copied().unwrap_or(0);
    let published = previous.max(epoch);
    if published > previous {

View on GitHub (pinned to affd8760f4)