astrid-runtime/astrid · critical · anyhow::Error

write device revocation

Error message

write device revocation {key_id}: CAS failed: {cas_error}; fail-closed tombstone failed: {fallback_error}

What it means

When the CAS write of a device revocation epoch fails, record_device_max falls back to writing a u64::MAX tombstone so every bearer of that key fails closed. This error is thrown when BOTH the CAS write and the fallback tombstone write fail, meaning the device revocation could not be persisted in any form.

Solutions

1. Restore the store and re-run apply_device_revocation; neither the epoch nor the tombstone was persisted
2. Do NOT return success (e.g. HTTP 204) to the caller — the code comments explicitly require the caller to observe this failure
3. Check store-side errors (read-only, disk, auth) revealed by the two wrapped errors
4. Keep the revoked key rejected in-process until the durable write succeeds

Example fix

// before: treat any tombstone write as success
store.set(REVOCATION_NAMESPACE, &key, encode_epoch(u64::MAX)).await?;
return Ok(());
// after: propagate failure so the HTTP path does not 204
match record_device_max(&store, key_id, epoch).await {
    Ok(published) => Ok((StatusCode::NO_CONTENT, published)),
    Err(e) => Err(StatusCode::SERVICE_UNAVAILABLE), // caller retries revocation
}

// Ensure the store accepts writes before revoking devices
pub async fn assert_device_store_writable(store: &dyn KvStore) -> anyhow::Result<()> {
    store.set(REVOCATION_NAMESPACE, "__write_probe__", b"1".to_vec()).await.map(|_| ())
}

match record_device_max(&store, key_id, epoch).await {
    Err(e) if e.to_string().contains("tombstone failed") => {
        // Zero durability: neither epoch nor tombstone persisted.
        error!(key_id, %e, "device revocation unpublished; denying key in-process");
        local_key_fence.lock().insert(key_id.to_string());
        Err(StatusCode::SERVICE_UNAVAILABLE) // never 204
    }
    other => other.map(|_| StatusCode::NO_CONTENT),
}

When it happens

Trigger: store CAS failure (cas_error) followed by a failing store.set of encode_epoch(u64::MAX) (fallback_error), while applying a device revocation through apply_device_revocation.

Common situations: KV backend becomes unavailable or read-only between the CAS attempt and the fallback; disk-full or quota limits on the store; sustained partition or store credential rotation mid-request.

Related errors

• write device revocation
• write principal revocation
• list device revocations
• list principal revocations
• read device revocation

AI-assisted analysis of astrid-runtime/astrid@affd8760f4 (2026-09-09). Data as JSON: /api/errors/a571ad69032ac96b. Report an issue: GitHub.

Appendix: source

Thrown at crates/astrid-gateway/src/revocations.rs:227

                encode_epoch(wanted),
            )
            .await
        {
            Ok(true) => return Ok(wanted),
            Ok(false) => {},
            Err(cas_error) => {
                // Same fail-closed durability rule as principal deletion.
                // MAX cannot be moved backward by a later CAS writer, so a
                // successful fallback write is restart-durable. If this set
                // also fails, no durable fence exists; the caller still gets
                // the error and can retain only an in-memory MAX for the
                // current process. Hydration aborts while KV is unavailable or
                // corrupt, while an empty healthy KV cannot recreate the key.
                store
                    .set(REVOCATION_NAMESPACE, &key, encode_epoch(u64::MAX))
                    .await
                    .map_err(|fallback_error| {
                        anyhow::anyhow!(
                            "write device revocation {key_id}: CAS failed: {cas_error}; fail-closed tombstone failed: {fallback_error}"
                        )
                    })?;
                // The tombstone keeps every bearer fail-closed, but the
                // caller must still observe the publication failure. In
                // particular, the HTTP revoke path cannot acknowledge 204
                // when its normal CAS durability path faulted.
                return Err(anyhow::anyhow!(
                    "write device revocation {key_id}: CAS failed: {cas_error}; fail-closed tombstone installed"
                ));
            },
        }
    }
}

fn publish_device_epoch<S: BuildHasher>(
    revoked_key_ids: &RwLock<HashMap<String, u64, S>>,
    key_id: &str,

View on GitHub (pinned to affd8760f4)