{"record":{"id":"a8044a3252eb3574","repo":"neondatabase/neon","slug":"rsyslogd-is-not-running-after-waiting-for-secon","errorCode":null,"errorMessage":"rsyslogd is not running after waiting for {} seconds and {} attempts","messagePattern":"rsyslogd is not running after waiting for (.+?) seconds and (.+?) attempts","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"critical","filePath":"compute_tools/src/rsyslog.rs","lineNumber":59,"sourceCode":"        attempts = attempt;\n        match get_rsyslog_pid() {\n            Some(pid) => return Ok(pid),\n            None => {\n                if start.elapsed() >= MAX_WAIT {\n                    break;\n                }\n                info!(\n                    \"rsyslogd is not running, attempt {}. Sleeping for {} ms\",\n                    attempt,\n                    sleep_duration.as_millis()\n                );\n                std::thread::sleep(sleep_duration);\n                sleep_duration *= 2;\n            }\n        }\n    }\n\n    Err(anyhow::anyhow!(\n        \"rsyslogd is not running after waiting for {} seconds and {} attempts\",\n        attempts,\n        start.elapsed().as_secs()\n    ))\n}\n\n// Restart rsyslogd to apply the new configuration.\n// This is necessary, because there is no other way to reload the rsyslog configuration.\n//\n// Rsyslogd shouldn't lose any messages, because of the restart,\n// because it tracks the last read position in the log files\n// and will continue reading from that position.\n// TODO: test it properly\n//\nfn restart_rsyslog() -> Result<()> {\n    // kill it to restart\n    let _ = Command::new(\"pkill\")\n        .arg(\"rsyslogd\")","sourceCodeStart":41,"sourceCodeEnd":77,"githubUrl":"https://github.com/neondatabase/neon/blob/8f60b04da47ffefe0e52bda2440134b42874eb75/compute_tools/src/rsyslog.rs#L41-L77","documentation":"restart_rsyslog() kills rsyslogd via pkill and then wait_for_rsyslog_pid() polls pgrep with exponential backoff (2ms doubling) for up to MAX_WAIT = 5 seconds. This error means no rsyslogd process appeared within that budget - it is not installed, nothing restarted it after the kill, or it crashed immediately, typically on the configuration just written under /etc/rsyslog.d/. It is raised from configure_audit_rsyslog()/configure_postgres_logs_export(), whose `?` propagation aborts compute configuration/startup.","triggerScenarios":"pkill succeeded but rsyslogd never reappears: missing binary in the image, no supervisor to respawn it, instant crash on a malformed generated rsyslog config (bad endpoint substitution), or a host too slow to restart within 5s.","commonSituations":"Custom compute images that omit rsyslogd; audit-log or logs-export endpoints with bad values producing invalid templates; slow or overloaded nodes during startup.","solutions":["Check rsyslogd is installed: `which rsyslogd`; start it manually to surface immediate crash output","Validate the generated config syntax: `rsyslogd -N1` (it checks /etc/rsyslog.d/*.conf including the just-written compute_audit_rsyslog.conf or postgres_logs.conf)","Inspect the written config for malformed host/port substitutions coming from AUDIT_LOGGING_ENDPOINT / AUDIT_LOGGING_TLS_ENDPOINT / logs_export_host","For slow hosts, increase MAX_WAIT in wait_for_rsyslog_pid or retry restart_rsyslog()"],"exampleFix":"// before\nconst MAX_WAIT: Duration = Duration::from_secs(5);\n\n// after\nconst MAX_WAIT: Duration = Duration::from_secs(15);","handlingStrategy":"retry","validationCode":"// validate generated rsyslog config before restarting the daemon\nfn config_is_valid() -> bool {\n    Command::new(\"rsyslogd\")\n        .args([\"-N1\"])\n        .output()\n        .map(|o| o.status.success())\n        .unwrap_or(false)\n}","typeGuard":null,"tryCatchPattern":"match restart_rsyslog() {\n    Ok(()) => {}\n    Err(e) => {\n        warn!(\"rsyslog restart failed: {e}; retrying once after manual start\");\n        let _ = Command::new(\"rsyslogd\").output();\n        restart_rsyslog()?;\n    }\n}","preventionTips":["Install rsyslogd in the compute image and run `rsyslogd -N1` in CI to validate templates","Keep audit/log-export endpoints in strict host:port form so generated configs are valid","Budget startup time on slow hosts: raise MAX_WAIT or retry restart_rsyslog"],"tags":["rsyslog","daemon","timeout","process-management","rust"],"backgroundTag":"service-startup-timeout","analyzedSha":"8f60b04da47ffefe0e52bda2440134b42874eb75","analyzedAt":"2026-08-16T23:39:28.135Z","schemaVersion":2},"datasetVersion":"2026-08-17T04:17:16.089Z"}