astrid-runtime/astrid · error · anyhow::Error

recorder lock poisoned

Error message

recorder lock poisoned: {e}

What it means

install_recorder lazily initializes a process-global Prometheus recorder behind a static Mutex. If another thread panicked while holding the lock, the Mutex is poisoned and lock() returns a PoisonError, which is wrapped in this anyhow error. It indicates a panic occurred inside the recorder-init critical section, not a metrics-configuration problem.

Solutions

  1. Find and fix the original panic inside the install_recorder critical section (the first panic's message is the real cause)
  2. Use lock().unwrap_or_else(PoisonError::into_inner) if recovery from poisoning is acceptable
  3. Serialize first-time initialization earlier in startup before spawning concurrent callers
  4. Audit the builder closure/code between lock and unlock for fallible unwraps that can panic

Example fix

// before
let mut guard = HANDLE.lock().map_err(|e| anyhow::anyhow!("recorder lock poisoned: {e}"))?;
// after
let mut guard = HANDLE.lock().unwrap_or_else(std::sync::PoisonError::into_inner); // recover from poisoned lock
Defensive patterns

Strategy: try-catch

Try / catch

match metrics::install_recorder() {
    Ok(handle) => handle,
    Err(e) if e.to_string().contains("lock poisoned") => {
        // a panic happened during init: log the original panic, restart or degrade gracefully
        log::error!("metrics init panicked earlier: {e}");
        fallback_no_metrics()
    }
    Err(e) => return Err(e),
}

Prevention

When it happens

Trigger: Calling install_recorder() after a previous call panicked while holding the HANDLE mutex — the poisoned lock makes every subsequent lock().map_err produce this error.

Common situations: A panic inside PrometheusBuilder installation (e.g. registry conflict) in another thread earlier in the process; tests running install_recorder concurrently with a panicking case; multi-threaded init racing with a failing builder.

Understand the failure class

Background: "This is a bug, please report it": internal invariant violations, unreachable panics, and SNH errors explained — this error's family across 47 libraries.

Related errors

AI-assisted analysis of astrid-runtime/astrid@affd8760f4 (2026-09-09). Data as JSON: /api/errors/161711bc62a9e6c4. Report an issue: GitHub.

Appendix: source

Thrown at crates/astrid-gateway/src/metrics.rs:115

/// allows only one recorder per process, so we serialise the install
/// behind a [`Mutex`] and memoise the handle inside it).
///
/// The Mutex (rather than `OnceLock::get_or_try_init`, which is
/// nightly) is what makes the function safe under concurrent
/// callers: two test binaries that both call `install_recorder` at
/// boot would race the underlying `metrics::set_global_recorder`,
/// the loser would `Err`, and we'd have no way to recover the
/// already-installed handle. Serialising the check + install + store
/// inside one critical section avoids that.
///
/// # Errors
/// Returns an error if `PrometheusBuilder::install_recorder` fails
/// on first call. Subsequent calls cannot fail.
pub fn install_recorder() -> anyhow::Result<PrometheusHandle> {
    static HANDLE: Mutex<Option<PrometheusHandle>> = Mutex::new(None);
    let mut guard = HANDLE
        .lock()
        .map_err(|e| anyhow::anyhow!("recorder lock poisoned: {e}"))?;
    if let Some(existing) = guard.as_ref() {
        return Ok(existing.clone());
    }
    let handle = PrometheusBuilder::new()
        // Set the per-route latency histogram's buckets explicitly.
        // The `Suffix` matcher applies to every series whose name
        // matches the metric base — same buckets for every
        // (method, route, status) combo.
        .set_buckets_for_metric(
            Matcher::Suffix(METRIC_REQUEST_DURATION_SECONDS.to_string()),
            DURATION_BUCKETS_SECONDS,
        )
        .map_err(|e| anyhow::anyhow!("configure histogram buckets: {e}"))?
        .install_recorder()
        .map_err(|e| anyhow::anyhow!("install Prometheus recorder: {e}"))?;

    // Describe every metric the gateway emits so `/metrics` carries
    // `# HELP` + `# TYPE` lines even before the series sees its

View on GitHub (pinned to affd8760f4)