{"record":{"id":"dc9627bf276af257","repo":"block/buzz","slug":"server-error-e","errorCode":null,"errorMessage":"Server error: {e}","messagePattern":"Server error: (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"crates/buzz-relay/src/main.rs","lineNumber":1398,"sourceCode":"        return Ok(());\n    }\n\n    #[cfg(not(unix))]\n    if config.uds_path.is_some() {\n        tracing::warn!(\"BUZZ_UDS_PATH set but UDS not supported on this platform\");\n    }\n\n    // TCP-only path.\n    let mut tcp_rx = shutdown_tx.subscribe();\n    axum::serve(\n        tcp_listener,\n        router.into_make_service_with_connect_info::<std::net::SocketAddr>(),\n    )\n    .with_graceful_shutdown(async move {\n        tcp_rx.changed().await.ok();\n    })\n    .await\n    .map_err(|e| anyhow::anyhow!(\"Server error: {e}\"))?;\n\n    let hard_shutdown = shutdown_handle\n        .await\n        .map_err(|e| anyhow::anyhow!(\"Shutdown task failed: {e}\"))?;\n    hard_shutdown.abort();\n    Ok(())\n}\n\n/// Wait for SIGTERM (Unix) or Ctrl+C.\nasync fn shutdown_signal() {\n    #[cfg(unix)]\n    {\n        use tokio::signal::unix::{signal, SignalKind};\n        let mut sigterm = signal(SignalKind::terminate()).expect(\"install SIGTERM handler\");\n        tokio::select! {\n            _ = tokio::signal::ctrl_c() => {},\n            _ = sigterm.recv() => {},\n        }","sourceCodeStart":1380,"sourceCodeEnd":1416,"githubUrl":"https://github.com/block/buzz/blob/f956e6fe06a76e50cbd8fba1a162482e752e7f1a/crates/buzz-relay/src/main.rs#L1380-L1416","documentation":"The TCP-only return path of serve() (no BUZZ_UDS_PATH): this wraps the error returned by axum::serve over the already-bound main TCP listener with graceful shutdown wired to a watch channel. Because the bind at main.rs:1330 succeeded, this error comes from hyper's accept loop or the into_make_service_with_connect_info layer failing mid-serve. Graceful shutdown returns Ok, so hitting this means a real serving-layer failure: fd exhaustion (EMFILE), kernel memory pressure, or a connection-layer bug.","triggerScenarios":"Default (non-UDS) deployments whose fd limit is exhausted by many concurrent WebSocket clients so accept() fails; ENOMEM under memory pressure; an error thrown by the connect-info extraction service for a particular peer. This is the variant every default deployment hits — no BUZZ_UDS_PATH involved.","commonSituations":"Production relays under connection spikes (mass reconnects after network events) with default 1024 fd limits; container orchestrators that do not raise nofile; slowly leaking fds over days of uptime.","solutions":["Raise the fd limit: ulimit -n 65536, LimitNOFILE=65536 (systemd), or nofile in the container spec — EMFILE is the most common cause.","Correlate with connection counts: ls /proc/<pid>/fd | wc -l vs conn_manager occupancy to spot leaked sockets.","Check dmesg/journal for OOM or TCP memory pressure.","Preserve the full error text and report — bind already succeeded, so a recurring serve error is a resource issue or a bug."],"exampleFix":"# before: relay exits under load: \"Server error: Too many open files (os error 24)\"\n\n# after (docker-compose)\nservices:\n  relay:\n    ulimits:\n      nofile:\n        soft: 65536\n        hard: 65536","handlingStrategy":"validation","validationCode":"// Pre-flight: verify fd headroom before the accept loop can hit EMFILE.\nfn fd_headroom_ok(min_fds: usize) -> bool {\n    let mut held = Vec::new();\n    for _ in 0..min_fds {\n        match std::net::TcpListener::bind((\"127.0.0.1\", 0)) {\n            Ok(l) => held.push(l),\n            Err(_) => return false,\n        }\n    }\n    true\n}\n\nassert!(fd_headroom_ok(1024), \"fd limit too low for the accept loop — raise nofile/LimitNOFILE\");","typeGuard":"fn is_fd_exhaustion(e: &std::io::Error) -> bool {\n    matches!(e.raw_os_error(), Some(24) | Some(23)) // EMFILE / ENFILE on Linux\n}","tryCatchPattern":"if let Err(e) = axum::serve(tcp_listener, svc)\n    .with_graceful_shutdown(async move { tcp_rx.changed().await.ok(); })\n    .await\n{\n    let msg = e.to_string();\n    if msg.contains(\"Too many open files\") {\n        tracing::error!(\"fd limit hit mid-serve — raise LimitNOFILE and restart\");\n    }\n    return Err(anyhow!(\"Server error: {e}\"));\n}","preventionTips":["Raise LimitNOFILE/nofile in systemd/container specs for production relays","Alert on open-fd counts approaching the limit","Test mass-reconnect scenarios before deploying listener changes"],"tags":["rust","axum","hyper","accept-loop","file-descriptors","runtime"],"backgroundTag":"tcp-accept-error","analyzedSha":"f956e6fe06a76e50cbd8fba1a162482e752e7f1a","analyzedAt":"2026-08-16T22:11:40.750Z","schemaVersion":2},"datasetVersion":"2026-08-16T23:17:17.608Z"}