astrid-runtime/astrid · critical · anyhow::Error

write principal revocation

Error message

write principal revocation {principal}: CAS failed: {cas_error}; fail-closed tombstone failed: {fallback_error}

What it means

After a compare-and-swap (CAS) write of the principal revocation epoch fails, record_principal_max attempts a fail-closed fallback: it writes an epoch of u64::MAX (a tombstone that revokes everything). This error is thrown when BOTH the CAS write and the fallback tombstone write fail, meaning the revocation could not be published at all.

Solutions

1. Restore the KV backend (health, capacity, write permissions) and re-run record_principal_max to publish the epoch
2. Treat the principal as NOT durably revoked: re-issue the revocation after recovery, since neither the epoch nor the tombstone landed
3. Check backend logs for read-only/disk-full/auth errors indicated by the two wrapped errors
4. Consider an in-process fence for the principal until the durable write succeeds, as suggested by the surrounding code comments

Example fix

// before: single-shot write
store.compare_exchange(...).await?;
// after: verify durability and re-run on failure
match record_principal_max(&store, principal, epoch).await {
    Ok(published) => info!(principal, published, "revocation durable"),
    Err(e) => {
        error!(principal, %e, "revocation NOT durable; fencing locally and retrying");
        local_fence.lock().insert(principal.clone(), epoch);
    }
}

// Verify write access before attempting CAS updates
pub async fn assert_store_writable(store: &dyn KvStore) -> anyhow::Result<()> {
    let probe_key = "__write_probe__";
    store.set(REVOCATION_NAMESPACE, probe_key, b"1".to_vec()).await?;
    store.delete(REVOCATION_NAMESPACE, probe_key).await
}

match record_principal_max(&store, principal, epoch).await {
    Err(e) if e.to_string().contains("tombstone failed") => {
        // Neither CAS nor tombstone landed: revocation NOT durable.
        error!(principal, %e, "revocation unpublished; retaining process-local fence");
        local_fence.lock().insert(principal.clone(), epoch);
        schedule_republish(principal, epoch);
    }
    other => other?,
}

When it happens

Trigger: store.compare_exchange (or equivalent) returns cas_error AND the follow-up store.set of encode_epoch(u64::MAX) returns fallback_error, while recording a principal max epoch via migrate_legacy_file or spawn_watcher.

Common situations: KV backend going down mid-operation (CAS fails, then the tombstone set also fails); backend in read-only mode (disk full, standby instance); sustained network partition; store auth/permission revocation during operation.

Related errors

• write device revocation
• write device revocation
• list device revocations
• list principal revocations
• read device revocation

AI-assisted analysis of astrid-runtime/astrid@affd8760f4 (2026-09-09). Data as JSON: /api/errors/f2addee6d4d4c654. Report an issue: GitHub.

Appendix: source

Thrown at crates/astrid-gateway/src/revocations.rs:163

            .await
        {
            Ok(true) => return Ok(wanted),
            Ok(false) => {},
            Err(cas_error) => {
                // A successful delete must never become reversible merely
                // because the monotonic CAS path is unavailable. Persist an
                // unconditional maximum-epoch tombstone: this intentionally
                // sacrifices alias reuse until operator repair, but it is
                // monotonic under every concurrent writer and survives a
                // restart when this fallback write succeeds. If the fallback
                // write also fails, no durable fence exists; propagate that
                // loss so the caller can retain only a process-local fence and
                // avoid claiming restart durability.
                store
                    .set(REVOCATION_NAMESPACE, &key, encode_epoch(u64::MAX))
                    .await
                    .map_err(|fallback_error| {
                        anyhow::anyhow!(
                            "write principal revocation {principal}: CAS failed: {cas_error}; fail-closed tombstone failed: {fallback_error}"
                        )
                    })?;
                return Ok(u64::MAX);
            },
        }
    }
}

/// Record the maximum device revocation epoch durably using the same CAS/max
/// rule as principal revocations.
///
/// A successful CAS, or a successful maximum-epoch fallback write after a
/// CAS error, leaves a fence that startup hydration can restore. The function
/// still returns an error after any CAS error so the HTTP caller withholds
/// `204`, even when the fallback tombstone succeeded. If both writes fail,
/// there is no durable fence; the caller may install a process-local maximum,
/// but a later healthy empty KV cannot reconstruct it.

View on GitHub (pinned to affd8760f4)