{"record":{"id":"f29439ac92c7ce42","repo":"block/buzz","slug":"tcp-server-error-e","errorCode":null,"errorMessage":"TCP server error: {e}","messagePattern":"TCP server error: (.+?)","errorType":"exception","errorClass":"anyhow::Error","httpStatus":null,"severity":"error","filePath":"crates/buzz-relay/src/main.rs","lineNumber":1489,"sourceCode":"        let uds_handle = tokio::spawn(async move {\n            axum::serve(uds_listener, router_uds.into_make_service())\n                .with_graceful_shutdown(async move {\n                    uds_rx.changed().await.ok();\n                })\n                .await\n                .ok();\n        });\n\n        let mut tcp_rx = shutdown_tx.subscribe();\n        axum::serve(\n            tcp_listener,\n            router.into_make_service_with_connect_info::<std::net::SocketAddr>(),\n        )\n        .with_graceful_shutdown(async move {\n            tcp_rx.changed().await.ok();\n        })\n        .await\n        .map_err(|e| anyhow::anyhow!(\"TCP server error: {e}\"))?;\n\n        let hard_shutdown = shutdown_handle\n            .await\n            .map_err(|e| anyhow::anyhow!(\"Shutdown task failed: {e}\"))?;\n        uds_handle.abort();\n        hard_shutdown.abort();\n        return Ok(());\n    }\n\n    #[cfg(not(unix))]\n    if config.uds_path.is_some() {\n        tracing::warn!(\"BUZZ_UDS_PATH set but UDS not supported on this platform\");\n    }\n\n    // TCP-only path.\n    let mut tcp_rx = shutdown_tx.subscribe();\n    axum::serve(\n        tcp_listener,","sourceCodeStart":1471,"sourceCodeEnd":1507,"githubUrl":"https://github.com/block/buzz/blob/ef2aa1ae38fadcc0bc22b8bf6ed96b35933146be/crates/buzz-relay/src/main.rs#L1471-L1507","documentation":"In the UDS+TCP mode of serve(), axum::serve drives the main TCP listener after a successful bind; any error from the accept/serve loop surfaces here as 'TCP server error'. Unlike the bind errors, this happens at runtime — the listener was healthy and then the server future failed. In practice axum/hyper only errors out of serve() on unrecoverable accept errors (e.g. EMFILE fd exhaustion) or a poisoned connection state; ordinary per-connection errors are handled internally.","triggerScenarios":"axum::serve(tcp_listener, router.into_make_service_with_connect_info::<SocketAddr>()) .with_graceful_shutdown(...) returning Err at crates/buzz-relay/src/main.rs:1364-1372: accept() failing persistently with EMFILE/ENFILE (process or system fd limit exhausted under connection churn), ENOMEM, or EBADF after external fd manipulation. It also propagates if the tokio runtime the serve future runs on is dropped unexpectedly.","commonSituations":"Relay under heavy WebSocket connection load hitting the `ulimit -n` cap (EMFILE on accept); a leak of sockets elsewhere in the process starving the accept loop; running under a supervisor that kills and the runtime shuts down mid-serve; extremely rare hyper internal errors on malformed connections.","solutions":["Check fd limits and raise them: `ulimit -n 65536` (systemd: LimitNOFILE=65536) and look for fd leaks: `ls /proc/<pid>/fd | wc -l` over time.","Capture the underlying io::Error kind from the anyhow chain (e.g. RUST_LOG=debug) — EMFILE points to fd exhaustion, EBADF to runtime misuse.","Restart under a supervisor (systemd/K8s restart policy) — serve() returning Err exits main, and the stale-socket cleanup handles UDS reuse on restart.","If it recurs, profile connection churn (`ss -tnp state established '( dport = :<port> )' | wc -l`) and tune connection timeouts/idle draining."],"exampleFix":null,"handlingStrategy":"try-catch","validationCode":"# raise fd limits before launch so accept() never starves\nulimit -n 65536","typeGuard":null,"tryCatchPattern":"// treat serve() failure as terminal but observable, and let the supervisor restart\nif let Err(e) = serve(router, health_router, state).await {\n    tracing::error!(error = %e, \"relay serve failed, exiting\");\n    std::process::exit(1);\n}","preventionTips":["Set LimitNOFILE/ulimit generously (64k+) on relay deployments.","Monitor open-fd counts (`/proc/<pid>/fd`) as a leading indicator of accept-loop EMFILE.","Run under a restart policy — this error path exits main by design.","Correlate occurrences with connection spikes in metrics before assuming a code bug."],"tags":["rust","network","runtime","accept-loop","axum","file-descriptors"],"backgroundTag":"listener-accept-failure","analyzedSha":"ef2aa1ae38fadcc0bc22b8bf6ed96b35933146be","analyzedAt":"2026-08-20T04:38:24.874Z","contentChangedAt":"2026-08-20T04:38:24.874Z","schemaVersion":2},"datasetVersion":"2026-09-23T08:17:48.524Z"}