{"record":{"id":"161711bc62a9e6c4","repo":"astrid-runtime/astrid","slug":"recorder-lock-poisoned-e","errorCode":null,"errorMessage":"recorder lock poisoned: {e}","messagePattern":"recorder lock poisoned: (.+?)","errorType":"exception","errorClass":"anyhow::Error","httpStatus":null,"severity":"error","filePath":"crates/astrid-gateway/src/metrics.rs","lineNumber":115,"sourceCode":"/// allows only one recorder per process, so we serialise the install\n/// behind a [`Mutex`] and memoise the handle inside it).\n///\n/// The Mutex (rather than `OnceLock::get_or_try_init`, which is\n/// nightly) is what makes the function safe under concurrent\n/// callers: two test binaries that both call `install_recorder` at\n/// boot would race the underlying `metrics::set_global_recorder`,\n/// the loser would `Err`, and we'd have no way to recover the\n/// already-installed handle. Serialising the check + install + store\n/// inside one critical section avoids that.\n///\n/// # Errors\n/// Returns an error if `PrometheusBuilder::install_recorder` fails\n/// on first call. Subsequent calls cannot fail.\npub fn install_recorder() -> anyhow::Result<PrometheusHandle> {\n    static HANDLE: Mutex<Option<PrometheusHandle>> = Mutex::new(None);\n    let mut guard = HANDLE\n        .lock()\n        .map_err(|e| anyhow::anyhow!(\"recorder lock poisoned: {e}\"))?;\n    if let Some(existing) = guard.as_ref() {\n        return Ok(existing.clone());\n    }\n    let handle = PrometheusBuilder::new()\n        // Set the per-route latency histogram's buckets explicitly.\n        // The `Suffix` matcher applies to every series whose name\n        // matches the metric base — same buckets for every\n        // (method, route, status) combo.\n        .set_buckets_for_metric(\n            Matcher::Suffix(METRIC_REQUEST_DURATION_SECONDS.to_string()),\n            DURATION_BUCKETS_SECONDS,\n        )\n        .map_err(|e| anyhow::anyhow!(\"configure histogram buckets: {e}\"))?\n        .install_recorder()\n        .map_err(|e| anyhow::anyhow!(\"install Prometheus recorder: {e}\"))?;\n\n    // Describe every metric the gateway emits so `/metrics` carries\n    // `# HELP` + `# TYPE` lines even before the series sees its","sourceCodeStart":97,"sourceCodeEnd":133,"githubUrl":"https://github.com/astrid-runtime/astrid/blob/affd8760f44190dbdfbec23403f4c4b642c33112/crates/astrid-gateway/src/metrics.rs#L97-L133","documentation":"install_recorder lazily initializes a process-global Prometheus recorder behind a static Mutex. If another thread panicked while holding the lock, the Mutex is poisoned and lock() returns a PoisonError, which is wrapped in this anyhow error. It indicates a panic occurred inside the recorder-init critical section, not a metrics-configuration problem.","triggerScenarios":"Calling install_recorder() after a previous call panicked while holding the HANDLE mutex — the poisoned lock makes every subsequent lock().map_err produce this error.","commonSituations":"A panic inside PrometheusBuilder installation (e.g. registry conflict) in another thread earlier in the process; tests running install_recorder concurrently with a panicking case; multi-threaded init racing with a failing builder.","solutions":["Find and fix the original panic inside the install_recorder critical section (the first panic's message is the real cause)","Use lock().unwrap_or_else(PoisonError::into_inner) if recovery from poisoning is acceptable","Serialize first-time initialization earlier in startup before spawning concurrent callers","Audit the builder closure/code between lock and unlock for fallible unwraps that can panic"],"exampleFix":"// before\nlet mut guard = HANDLE.lock().map_err(|e| anyhow::anyhow!(\"recorder lock poisoned: {e}\"))?;\n// after\nlet mut guard = HANDLE.lock().unwrap_or_else(std::sync::PoisonError::into_inner); // recover from poisoned lock","handlingStrategy":"try-catch","validationCode":null,"typeGuard":null,"tryCatchPattern":"match metrics::install_recorder() {\n    Ok(handle) => handle,\n    Err(e) if e.to_string().contains(\"lock poisoned\") => {\n        // a panic happened during init: log the original panic, restart or degrade gracefully\n        log::error!(\"metrics init panicked earlier: {e}\");\n        fallback_no_metrics()\n    }\n    Err(e) => return Err(e),\n}","preventionTips":["Never panic inside the recorder-init critical section; use Results end to end","Install the recorder once, early in main, before spawning concurrent tasks","Avoid unwraps in code that runs while holding a shared mutex"],"tags":["mutex","poisoned-lock","metrics","panic"],"backgroundTag":"internal-invariant-violation","analyzedSha":"affd8760f44190dbdfbec23403f4c4b642c33112","analyzedAt":"2026-09-09T21:28:12.402Z","contentChangedAt":"2026-09-09T21:28:12.402Z","schemaVersion":2},"datasetVersion":"2026-09-17T15:17:12.973Z"}