block/buzz · error

TCP server error: {e}

Error message

TCP server error: {e}

What it means

On the unix path where both UDS and TCP listeners run (BUZZ_UDS_PATH set), this wraps the error returned by axum::serve over the already-bound main TCP listener — i.e. hyper's accept loop or the into_make_service_with_connect_info layer failed while serving, and the error propagated out of serve().await. Graceful shutdown completes with Ok (the watch channel fires first), so this error indicates a genuine accept-layer failure: file-descriptor exhaustion (EMFILE), kernel memory pressure, or a connection-layer bug.

Source

Thrown at crates/buzz-relay/src/main.rs:1373

        let uds_handle = tokio::spawn(async move {
            axum::serve(uds_listener, router_uds.into_make_service())
                .with_graceful_shutdown(async move {
                    uds_rx.changed().await.ok();
                })
                .await
                .ok();
        });

        let mut tcp_rx = shutdown_tx.subscribe();
        axum::serve(
            tcp_listener,
            router.into_make_service_with_connect_info::<std::net::SocketAddr>(),
        )
        .with_graceful_shutdown(async move {
            tcp_rx.changed().await.ok();
        })
        .await
        .map_err(|e| anyhow::anyhow!("TCP server error: {e}"))?;

        let hard_shutdown = shutdown_handle
            .await
            .map_err(|e| anyhow::anyhow!("Shutdown task failed: {e}"))?;
        uds_handle.abort();
        hard_shutdown.abort();
        return Ok(());
    }

    #[cfg(not(unix))]
    if config.uds_path.is_some() {
        tracing::warn!("BUZZ_UDS_PATH set but UDS not supported on this platform");
    }

    // TCP-only path.
    let mut tcp_rx = shutdown_tx.subscribe();
    axum::serve(
        tcp_listener,

View on GitHub (pinned to f956e6fe06)

Solutions

1. Raise the fd limit: ulimit -n 65536, or LimitNOFILE=65536 in systemd / nofile in container securityContext — EMFILE is the most common accept-loop failure.
2. Watch fd churn while load runs: ls /proc/<relay-pid>/fd | wc -l; look for leaked sockets the connection manager is not closing.
3. Check dmesg/journal for OOM kills or TCP memory pressure (tcp_mem, somaxconn tuning).
4. Capture the full error text and report it — with bind already successful, a persistent serve error is a host-resource issue or a relay bug, not configuration.

Example fix

# before: relay exits under load: "TCP server error: Too many open files (os error 24)"
# (systemd unit with default limits)

# after
[Service]
LimitNOFILE=65536

// Pre-flight: verify the process actually has fd headroom before serving.
fn fd_headroom_ok(min_fds: usize) -> bool {
    let mut held = Vec::new();
    for _ in 0..min_fds {
        match std::net::TcpListener::bind(("127.0.0.1", 0)) {
            Ok(l) => held.push(l),
            Err(_) => return false,
        }
    }
    true // sockets dropped here; OS limit is at least min_fds above current usage
}

assert!(fd_headroom_ok(1024), "fd limit too low for the accept loop — raise nofile/LimitNOFILE");

fn is_fd_exhaustion(e: &std::io::Error) -> bool {
    matches!(e.raw_os_error(), Some(24) | Some(23)) // EMFILE / ENFILE on Linux
}

if let Err(e) = axum::serve(tcp_listener, svc)
    .with_graceful_shutdown(async move { tcp_rx.changed().await.ok(); })
    .await
{
    let msg = e.to_string();
    if msg.contains("Too many open files") {
        tracing::error!("fd limit hit mid-serve — raise LimitNOFILE and restart");
    }
    return Err(anyhow!("TCP server error: {e}"));
}

When it happens

Trigger: Hitting the process fd limit (ulimit -n, often 1024) under many concurrent WebSocket connections so accept() starts failing; ENOMEM under socket memory pressure; a peer interaction that errors inside the connect-info service. Only reachable when BUZZ_UDS_PATH is set (the UDS+TCP code path).

Common situations: Relay absorbing connection storms — mesh peers or mass client reconnects after a network blip — with default container/systemd fd limits; long-lived processes slowly leaking fds.

Related errors

• Server error: {e}
• git conformance probe failed: {e}
• Failed to bind health port {}: {e}
• Failed to bind {}: {e}
• BUZZ_UDS_PATH {uds_path} exists but is not a socket

AI-assisted analysis of block/buzz@f956e6fe06 (2026-08-16). Data as JSON: /api/errors/f29439ac92c7ce42. Report an issue: GitHub.