What causes "revocation map poisoned during startup hydration" in astrid-runtime/astrid?

Find and fix the original panic that poisoned the `revoked_at` lock — poisoning is a symptom, not the root cause.

revocation map poisoned during startup hydration

Error message

revocation map poisoned during startup hydration

What it means

`hydrate_revocations` loads principals/ devices revocation lists from the store and swaps them into `RwLock`-guarded maps. The `expect` fires when the `revoked_at` lock is poisoned — i.e. another thread panicked while holding the write lock — so hydration cannot safely install the fresh maps. Panicking during startup hydration is deliberate: an unusable revocation state must not be silently ignored.

Solutions

Find and fix the original panic that poisoned the `revoked_at` lock — poisoning is a symptom, not the root cause.
Avoid panicking while holding revocation-map locks; return `Result` from mutation paths instead.
If hydration must tolerate poison, use `unwrap_or_else(PoisonError::into_inner)` only after auditing the stale state is safe to overwrite.
Ensure hydration runs before other threads acquire these locks to eliminate the poisoning window.

Example fix

// before
*self.revoked_at.write().expect("revocation map poisoned during startup hydration") = principals;
// after
let mut guard = self
    .revoked_at
    .write()
    .unwrap_or_else(std::sync::PoisonError::into_inner);
*guard = principals;

Defensive patterns

Strategy: try-catch

Validate before calling

// Rust: nothing to pre-validate; instead check lock state before hydration
assert!(!is_poisoned(&self.revoked_at), "revoked_at lock poisoned before hydration");

Try / catch

// Rust: tolerate poison when overwriting with fresh store data is safe
let mut guard = self.revoked_at.write()
    .unwrap_or_else(std::sync::PoisonError::into_inner);
*guard = principals;

Prevention

Never panic while holding a lock; return Result from mutations.
Run hydration single-threaded before request handlers start.
Add a CI test that mutates revocation maps concurrently to surface panics under the lock.
Fix root-cause panics, not the poisoning symptom.

When it happens

Trigger: Calling `hydrate_revocations` at startup when the `revoked_at` `RwLock` was previously poisoned by a panic in any earlier reader/writer (a `.write()` returns `Err(PoisonError)`).

Common situations: An earlier panic while mutating/reading revocation maps (e.g. malformed entry causing unwrap failure elsewhere), then a restart/hydration pass hits the poisoned lock; multi-threaded startup where one task panics before hydration runs.

Understand the failure class

Background: "This is a bug, please report it": internal invariant violations, unreachable panics, and SNH errors explained — this error's family across 47 libraries.

Related errors

AI-assisted analysis of astrid-runtime/astrid@affd8760f4 (2026-09-09). Data as JSON: /api/errors/bac0b762439089af. Report an issue: GitHub.

Appendix: source

Thrown at crates/astrid-gateway/src/state.rs:396

    /// # Panics
    ///
    /// Panics if either in-memory revocation lock is poisoned, indicating an
    /// earlier panic while mutating gateway security state.
    pub async fn hydrate_revocations(&self) -> anyhow::Result<()> {
        let Some(store) = self.storage_kv.as_deref() else {
            if crate::revocations::legacy_file_exists()? {
                anyhow::bail!(
                    "gateway revocation storage is unavailable while a legacy revocation file exists"
                );
            }
            return Ok(());
        };
        let _ = crate::revocations::migrate_legacy_file(store).await?;
        let (principals, devices) = crate::revocations::load_from_store(store).await?;
        *self
            .revoked_at
            .write()
            .expect("revocation map poisoned during startup hydration") = principals;
        *self
            .revoked_key_ids
            .write()
            .expect("device revocation map poisoned during startup hydration") = devices;
        Ok(())
    }

    /// Build a bus-direct admin client bound to `caller`. Routes
    /// hosted in this same process talk to the kernel over the
    /// shared event bus rather than the Unix socket — bypasses the
    /// `astrid-capsule-cli` proxy entirely and removes the 19 RPS
    /// admin-throughput ceiling the socket path imposes.
    ///
    /// # Errors
    /// Returns an internal error if the state was built without a
    /// live event bus (the standalone tests-only constructor). In
    /// production the daemon always wires it up.
    pub fn admin_client(

View on GitHub (pinned to affd8760f4)