{"record":{"id":"03f055d0e5cbbc3a","repo":"neondatabase/neon","slug":"log-set-id-s-rwlock-poisoned","errorCode":null,"errorMessage":"Log set id's rwlock poisoned: {}","messagePattern":"Log set id's rwlock poisoned: (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"warning","filePath":"compute_tools/src/logger.rs","lineNumber":143,"sourceCode":"        Some(TraceContextPropagator::new().extract(&startup_tracing_carrier))\n    } else {\n        None\n    }\n}\n\n/// Track relevant id's\nconst UNKNOWN_IDS: &str = r#\"\"pg_instance_id\": \"\", \"pg_compute_id\": \"\"\"#;\nstatic IDS: LazyLock<RwLock<String>> = LazyLock::new(|| RwLock::new(UNKNOWN_IDS.to_string()));\n\npub fn update_ids(instance_id: &Option<String>, compute_id: &Option<String>) -> anyhow::Result<()> {\n    let ids = format!(\n        r#\"\"pg_instance_id\": \"{}\", \"pg_compute_id\": \"{}\"\"#,\n        instance_id.as_ref().map(|s| s.as_str()).unwrap_or_default(),\n        compute_id.as_ref().map(|s| s.as_str()).unwrap_or_default()\n    );\n    let mut guard = IDS\n        .write()\n        .map_err(|e| anyhow::anyhow!(\"Log set id's rwlock poisoned: {}\", e))?;\n    *guard = ids;\n    Ok(())\n}\n\n/// Massage compute_ctl logs into PG json log shape so we can use the same Lumberjack setup.\nstruct PgJsonLogShapeFormatter;\nimpl<S, N> fmt::format::FormatEvent<S, N> for PgJsonLogShapeFormatter\nwhere\n    S: Subscriber + for<'a> LookupSpan<'a>,\n    N: for<'a> fmt::format::FormatFields<'a> + 'static,\n{\n    fn format_event(\n        &self,\n        ctx: &fmt::FmtContext<'_, S, N>,\n        mut writer: fmt::format::Writer<'_>,\n        event: &tracing::Event<'_>,\n    ) -> std::fmt::Result {\n        // Format values from the event's metadata, and open message string","sourceCodeStart":125,"sourceCodeEnd":161,"githubUrl":"https://github.com/neondatabase/neon/blob/8f60b04da47ffefe0e52bda2440134b42874eb75/compute_tools/src/logger.rs#L125-L161","documentation":"compute_ctl keeps instance/compute ids for log lines in a global static RwLock<String> (IDS). A RwLock gets poisoned when a thread panics while holding it; every later update_ids() then returns PoisonError, which is mapped to this anyhow error. The poison is only a symptom: the real bug is whatever panicked mid-write, and after that every id update fails (logs keep the stale/unknown ids) until the process restarts.","triggerScenarios":"Some thread panicked between IDS.write() acquiring the lock and dropping the guard (the format! before it cannot panic in practice, so typically a panic in reentrant/extension code on the same thread), after which any update_ids() call returns Err with the PoisonError.","commonSituations":"Almost always secondary damage: a panic elsewhere in compute_ctl (a bug, an unwrap in a logging path) poisons the lock, then this error spams the log; long-running computes where one early panic permanently degrades id logging.","solutions":["Search logs backwards for the original panic ('panicked at') - fix that first","Restart compute_ctl to clear the poisoned static (ids then update normally)","Harden update_ids to recover from poisoning: RwLockWriteGuard::from(PoisonError::into_inner) keeps the last value usable, or switch the static to OnceLock/parking_lot::RwLock which do not poison"],"exampleFix":"// before\nlet mut guard = IDS.write().map_err(|e| anyhow!(\"Log set id's rwlock poisoned: {e}\"))?;\n// after: recover instead of propagating\nlet mut guard = IDS.write().unwrap_or_else(|e| e.into_inner());","handlingStrategy":"fallback","validationCode":null,"typeGuard":"fn ids_lock_alive() -> bool {\n    IDS.try_write().is_ok()\n}","tryCatchPattern":"// Recover from poisoning: keep logging with the last written ids instead of erroring\nlet mut guard = IDS.write().unwrap_or_else(|poison| {\n    warn!(\"ids lock poisoned; recovering with last value\");\n    poison.into_inner()\n});\n*guard = ids;","preventionTips":["Treat any panic in compute_ctl as a bug to fix immediately - poisoning is always secondary damage","Prefer poison-immutable sync primitives for logging state (OnceLock, parking_lot::RwLock) in new code","Test the poison path explicitly: panic under the lock, then assert update_ids still works"],"tags":["rust","compute-ctl","logging","rwlock","poisoned-lock","panic"],"backgroundTag":"lock-poisoned","analyzedSha":"8f60b04da47ffefe0e52bda2440134b42874eb75","analyzedAt":"2026-08-16T23:39:28.135Z","schemaVersion":2},"datasetVersion":"2026-08-17T04:17:16.089Z"}