{"record":{"id":"f29439ac92c7ce42","repo":"block/buzz","slug":"tcp-server-error-e","errorCode":null,"errorMessage":"TCP server error: {e}","messagePattern":"TCP server error: (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"crates/buzz-relay/src/main.rs","lineNumber":1373,"sourceCode":"        let uds_handle = tokio::spawn(async move {\n            axum::serve(uds_listener, router_uds.into_make_service())\n                .with_graceful_shutdown(async move {\n                    uds_rx.changed().await.ok();\n                })\n                .await\n                .ok();\n        });\n\n        let mut tcp_rx = shutdown_tx.subscribe();\n        axum::serve(\n            tcp_listener,\n            router.into_make_service_with_connect_info::<std::net::SocketAddr>(),\n        )\n        .with_graceful_shutdown(async move {\n            tcp_rx.changed().await.ok();\n        })\n        .await\n        .map_err(|e| anyhow::anyhow!(\"TCP server error: {e}\"))?;\n\n        let hard_shutdown = shutdown_handle\n            .await\n            .map_err(|e| anyhow::anyhow!(\"Shutdown task failed: {e}\"))?;\n        uds_handle.abort();\n        hard_shutdown.abort();\n        return Ok(());\n    }\n\n    #[cfg(not(unix))]\n    if config.uds_path.is_some() {\n        tracing::warn!(\"BUZZ_UDS_PATH set but UDS not supported on this platform\");\n    }\n\n    // TCP-only path.\n    let mut tcp_rx = shutdown_tx.subscribe();\n    axum::serve(\n        tcp_listener,","sourceCodeStart":1355,"sourceCodeEnd":1391,"githubUrl":"https://github.com/block/buzz/blob/f956e6fe06a76e50cbd8fba1a162482e752e7f1a/crates/buzz-relay/src/main.rs#L1355-L1391","documentation":"On the unix path where both UDS and TCP listeners run (BUZZ_UDS_PATH set), this wraps the error returned by axum::serve over the already-bound main TCP listener — i.e. hyper's accept loop or the into_make_service_with_connect_info layer failed while serving, and the error propagated out of serve().await. Graceful shutdown completes with Ok (the watch channel fires first), so this error indicates a genuine accept-layer failure: file-descriptor exhaustion (EMFILE), kernel memory pressure, or a connection-layer bug.","triggerScenarios":"Hitting the process fd limit (ulimit -n, often 1024) under many concurrent WebSocket connections so accept() starts failing; ENOMEM under socket memory pressure; a peer interaction that errors inside the connect-info service. Only reachable when BUZZ_UDS_PATH is set (the UDS+TCP code path).","commonSituations":"Relay absorbing connection storms — mesh peers or mass client reconnects after a network blip — with default container/systemd fd limits; long-lived processes slowly leaking fds.","solutions":["Raise the fd limit: ulimit -n 65536, or LimitNOFILE=65536 in systemd / nofile in container securityContext — EMFILE is the most common accept-loop failure.","Watch fd churn while load runs: ls /proc/<relay-pid>/fd | wc -l; look for leaked sockets the connection manager is not closing.","Check dmesg/journal for OOM kills or TCP memory pressure (tcp_mem, somaxconn tuning).","Capture the full error text and report it — with bind already successful, a persistent serve error is a host-resource issue or a relay bug, not configuration."],"exampleFix":"# before: relay exits under load: \"TCP server error: Too many open files (os error 24)\"\n# (systemd unit with default limits)\n\n# after\n[Service]\nLimitNOFILE=65536","handlingStrategy":"validation","validationCode":"// Pre-flight: verify the process actually has fd headroom before serving.\nfn fd_headroom_ok(min_fds: usize) -> bool {\n    let mut held = Vec::new();\n    for _ in 0..min_fds {\n        match std::net::TcpListener::bind((\"127.0.0.1\", 0)) {\n            Ok(l) => held.push(l),\n            Err(_) => return false,\n        }\n    }\n    true // sockets dropped here; OS limit is at least min_fds above current usage\n}\n\nassert!(fd_headroom_ok(1024), \"fd limit too low for the accept loop — raise nofile/LimitNOFILE\");","typeGuard":"fn is_fd_exhaustion(e: &std::io::Error) -> bool {\n    matches!(e.raw_os_error(), Some(24) | Some(23)) // EMFILE / ENFILE on Linux\n}","tryCatchPattern":"if let Err(e) = axum::serve(tcp_listener, svc)\n    .with_graceful_shutdown(async move { tcp_rx.changed().await.ok(); })\n    .await\n{\n    let msg = e.to_string();\n    if msg.contains(\"Too many open files\") {\n        tracing::error!(\"fd limit hit mid-serve — raise LimitNOFILE and restart\");\n    }\n    return Err(anyhow!(\"TCP server error: {e}\"));\n}","preventionTips":["Raise LimitNOFILE/nofile before serving under high WebSocket connection counts","Monitor open-fd counts (/proc/<pid>/fd) and connection-manager occupancy together","Load-test reconnect storms before shipping listener changes"],"tags":["rust","axum","hyper","accept-loop","file-descriptors","runtime"],"backgroundTag":"tcp-accept-error","analyzedSha":"f956e6fe06a76e50cbd8fba1a162482e752e7f1a","analyzedAt":"2026-08-16T22:11:40.750Z","schemaVersion":2},"datasetVersion":"2026-08-16T23:17:17.608Z"}