neondatabase/neon · warning

Log set id's rwlock poisoned: {}

Error message

Log set id's rwlock poisoned: {}

What it means

compute_ctl keeps instance/compute ids for log lines in a global static RwLock<String> (IDS). A RwLock gets poisoned when a thread panics while holding it; every later update_ids() then returns PoisonError, which is mapped to this anyhow error. The poison is only a symptom: the real bug is whatever panicked mid-write, and after that every id update fails (logs keep the stale/unknown ids) until the process restarts.

Source

Thrown at compute_tools/src/logger.rs:143

        Some(TraceContextPropagator::new().extract(&startup_tracing_carrier))
    } else {
        None
    }
}

/// Track relevant id's
const UNKNOWN_IDS: &str = r#""pg_instance_id": "", "pg_compute_id": """#;
static IDS: LazyLock<RwLock<String>> = LazyLock::new(|| RwLock::new(UNKNOWN_IDS.to_string()));

pub fn update_ids(instance_id: &Option<String>, compute_id: &Option<String>) -> anyhow::Result<()> {
    let ids = format!(
        r#""pg_instance_id": "{}", "pg_compute_id": "{}""#,
        instance_id.as_ref().map(|s| s.as_str()).unwrap_or_default(),
        compute_id.as_ref().map(|s| s.as_str()).unwrap_or_default()
    );
    let mut guard = IDS
        .write()
        .map_err(|e| anyhow::anyhow!("Log set id's rwlock poisoned: {}", e))?;
    *guard = ids;
    Ok(())
}

/// Massage compute_ctl logs into PG json log shape so we can use the same Lumberjack setup.
struct PgJsonLogShapeFormatter;
impl<S, N> fmt::format::FormatEvent<S, N> for PgJsonLogShapeFormatter
where
    S: Subscriber + for<'a> LookupSpan<'a>,
    N: for<'a> fmt::format::FormatFields<'a> + 'static,
{
    fn format_event(
        &self,
        ctx: &fmt::FmtContext<'_, S, N>,
        mut writer: fmt::format::Writer<'_>,
        event: &tracing::Event<'_>,
    ) -> std::fmt::Result {
        // Format values from the event's metadata, and open message string

View on GitHub (pinned to 8f60b04da4)

Solutions

1. Search logs backwards for the original panic ('panicked at') - fix that first
2. Restart compute_ctl to clear the poisoned static (ids then update normally)
3. Harden update_ids to recover from poisoning: RwLockWriteGuard::from(PoisonError::into_inner) keeps the last value usable, or switch the static to OnceLock/parking_lot::RwLock which do not poison

Example fix

// before
let mut guard = IDS.write().map_err(|e| anyhow!("Log set id's rwlock poisoned: {e}"))?;
// after: recover instead of propagating
let mut guard = IDS.write().unwrap_or_else(|e| e.into_inner());

fn ids_lock_alive() -> bool {
    IDS.try_write().is_ok()
}

// Recover from poisoning: keep logging with the last written ids instead of erroring
let mut guard = IDS.write().unwrap_or_else(|poison| {
    warn!("ids lock poisoned; recovering with last value");
    poison.into_inner()
});
*guard = ids;

When it happens

Trigger: Some thread panicked between IDS.write() acquiring the lock and dropping the guard (the format! before it cannot panic in practice, so typically a panic in reentrant/extension code on the same thread), after which any update_ids() call returns Err with the PoisonError.

Common situations: Almost always secondary damage: a panic elsewhere in compute_ctl (a bug, an unwrap in a logging path) poisons the lock, then this error spams the log; long-running computes where one early panic permanently degrades id logging.

Related errors

• not implemented
• connection to postgres closed
• pageserver connection information should be provided
• safekeeper connstrings should be provided
• tenant id should be provided

AI-assisted analysis of neondatabase/neon@8f60b04da4 (2026-08-16). Data as JSON: /api/errors/03f055d0e5cbbc3a. Report an issue: GitHub.