neondatabase/neon · critical

rsyslogd is not running after waiting for {} seconds and {}

Error message

rsyslogd is not running after waiting for {} seconds and {} attempts

What it means

restart_rsyslog() kills rsyslogd via pkill and then wait_for_rsyslog_pid() polls pgrep with exponential backoff (2ms doubling) for up to MAX_WAIT = 5 seconds. This error means no rsyslogd process appeared within that budget - it is not installed, nothing restarted it after the kill, or it crashed immediately, typically on the configuration just written under /etc/rsyslog.d/. It is raised from configure_audit_rsyslog()/configure_postgres_logs_export(), whose `?` propagation aborts compute configuration/startup.

Source

Thrown at compute_tools/src/rsyslog.rs:59

        attempts = attempt;
        match get_rsyslog_pid() {
            Some(pid) => return Ok(pid),
            None => {
                if start.elapsed() >= MAX_WAIT {
                    break;
                }
                info!(
                    "rsyslogd is not running, attempt {}. Sleeping for {} ms",
                    attempt,
                    sleep_duration.as_millis()
                );
                std::thread::sleep(sleep_duration);
                sleep_duration *= 2;
            }
        }
    }

    Err(anyhow::anyhow!(
        "rsyslogd is not running after waiting for {} seconds and {} attempts",
        attempts,
        start.elapsed().as_secs()
    ))
}

// Restart rsyslogd to apply the new configuration.
// This is necessary, because there is no other way to reload the rsyslog configuration.
//
// Rsyslogd shouldn't lose any messages, because of the restart,
// because it tracks the last read position in the log files
// and will continue reading from that position.
// TODO: test it properly
//
fn restart_rsyslog() -> Result<()> {
    // kill it to restart
    let _ = Command::new("pkill")
        .arg("rsyslogd")

View on GitHub (pinned to 8f60b04da4)

Solutions

1. Check rsyslogd is installed: `which rsyslogd`; start it manually to surface immediate crash output
2. Validate the generated config syntax: `rsyslogd -N1` (it checks /etc/rsyslog.d/*.conf including the just-written compute_audit_rsyslog.conf or postgres_logs.conf)
3. Inspect the written config for malformed host/port substitutions coming from AUDIT_LOGGING_ENDPOINT / AUDIT_LOGGING_TLS_ENDPOINT / logs_export_host
4. For slow hosts, increase MAX_WAIT in wait_for_rsyslog_pid or retry restart_rsyslog()

Example fix

// before
const MAX_WAIT: Duration = Duration::from_secs(5);

// after
const MAX_WAIT: Duration = Duration::from_secs(15);

// validate generated rsyslog config before restarting the daemon
fn config_is_valid() -> bool {
    Command::new("rsyslogd")
        .args(["-N1"])
        .output()
        .map(|o| o.status.success())
        .unwrap_or(false)
}

match restart_rsyslog() {
    Ok(()) => {}
    Err(e) => {
        warn!("rsyslog restart failed: {e}; retrying once after manual start");
        let _ = Command::new("rsyslogd").output();
        restart_rsyslog()?;
    }
}

When it happens

Trigger: pkill succeeded but rsyslogd never reappears: missing binary in the image, no supervisor to respawn it, instant crash on a malformed generated rsyslog config (bad endpoint substitution), or a host too slow to restart within 5s.

Common situations: Custom compute images that omit rsyslogd; audit-log or logs-export endpoints with bad values producing invalid templates; slow or overloaded nodes during startup.

Related errors

• {} with pid {} did not stop in {:?} seconds
• AUDIT_LOGGING_ENDPOINT and AUDIT_LOGGING_TLS_ENDPOINT are bo
• Error parsing {remote_endpoint}, expected host:port, got {er
• Invalid address format {remote_endpoint}, expected host:port
• Invalid host

AI-assisted analysis of neondatabase/neon@8f60b04da4 (2026-08-16). Data as JSON: /api/errors/a8044a3252eb3574. Report an issue: GitHub.