What causes "persist device revocation fence for {key_id}: {error}" in astrid-runtime/astrid?

Inspect the inner error chained to this message — it names the underlying apply_device_revocation failure (KV path, permissions, backend connectivity) and fix that first.

astrid-runtime/astrid · critical · GatewayError::Internal

persist device revocation fence for {key_id}: {error}

Error message

persist device revocation fence for {key_id}: {error}

What it means

`acknowledge_device_revocation` throws this GatewayError::Internal when `crate::revocations::apply_device_revocation` fails to durably persist (and install in the live map) the device revocation fence for the given key_id. The kernel has already removed the device's public key, but the gateway-side fence could not be written (e.g. to storage_kv), so the gateway refuses to return 204 because the revocation acknowledgement is incomplete — the device could otherwise still be treated as paired by the gateway.

Source

Thrown at crates/astrid-gateway/src/routes/principals.rs:575

/// The kernel has already removed the public key when this helper runs. The
/// gateway only emits HTTP 204 after the same monotonic device fence is
/// durably published and installed in its live map. Keeping this narrow step
/// separate also lets persistence-failure tests exercise the HTTP boundary
/// without fabricating a kernel admin client.
pub async fn acknowledge_device_revocation(
    state: &GatewayState,
    key_id: &str,
    epoch: u64,
) -> GatewayResult<StatusCode> {
    crate::revocations::apply_device_revocation(
        &state.revoked_key_ids,
        state.storage_kv.as_deref(),
        key_id,
        epoch,
    )
    .await
    .map_err(|error| {
        GatewayError::Internal(anyhow::anyhow!(
            "persist device revocation fence for {key_id}: {error}"
        ))
    })?;
    Ok(StatusCode::NO_CONTENT)
}

// ── Helpers ──────────────────────────────────────────────────────

pub(crate) fn caller_from(req: &Request<axum::body::Body>) -> GatewayResult<&CallerContext> {
    req.extensions()
        .get::<CallerContext>()
        .ok_or(GatewayError::Unauthorized)
}

// Both helpers consume their argument logically (wrap and discard);
// clippy::needless_pass_by_value fires because they only `Display` /
// `Debug` the value. Taking by value keeps `map_err(daemon_internal)`
// usable as a one-line closure replacement throughout the routes —

View on GitHub (pinned to affd8760f4)

Solutions

Inspect the inner error chained to this message — it names the underlying apply_device_revocation failure (KV path, permissions, backend connectivity) and fix that first.
Verify `state.storage_kv` points to a writable, correctly configured KV store; restart the gateway once storage is healthy.
Re-run the device deletion after fixing storage; ensure the fence can be re-applied idempotently rather than leaving the gateway's revoked-key map stale.
Monitor/alert on KV write failures so revocation fences are never left unacknowledged in production.

Defensive patterns

Strategy: try-catch

Validate before calling

// before revoking, confirm durable storage is available
if state.storage_kv.is_none() {
    return Err(anyhow!("storage_kv not configured; cannot persist revocation fence"));
}

Type guard

fn storage_ready(state: &GatewayState) -> bool {
    state.storage_kv.is_some()
}

Try / catch

match acknowledge_device_revocation(&state, &key_id, epoch).await {
    Err(e) if e.to_string().starts_with("persist device revocation fence") => {
        // kernel already revoked; enqueue fence for retry and surface 503
        tracing::warn!(key_id, error = %e, "revocation fence not persisted");
        StatusCode::SERVICE_UNAVAILABLE
    }
    other => other?,
}

Prevention

Health-check the KV backend before serving delete-device requests
Make apply_device_revocation idempotent so a retry after partial failure can complete the fence
Alert on revocation persistence failures — a missed fence lets a revoked device remain usable at the gateway

When it happens

Trigger: DELETE /api/principals/{id}/devices/{key_id} where the kernel admin PairDeviceRevoke succeeds but `apply_device_revocation` errors — typically because `state.storage_kv` is unavailable/misconfigured or the underlying KV write fails (I/O error, permission denied, serialization failure).

Common situations: storage_kv path misconfigured or on a read-only filesystem; KV backend down or disk full; corrupted KV record for the key_id; gateway running with storage_kv = None in an environment that requires durable revocation persistence; transient storage outage during a revoke request.

Related errors

AI-assisted analysis of astrid-runtime/astrid@affd8760f4 (2026-09-09). Data as JSON: /api/errors/9776a3d8330bb4ac. Report an issue: GitHub.