block/buzz · error

Server error: {e}

Error message

Server error: {e}

What it means

The TCP-only return path of serve() (no BUZZ_UDS_PATH): this wraps the error returned by axum::serve over the already-bound main TCP listener with graceful shutdown wired to a watch channel. Because the bind at main.rs:1330 succeeded, this error comes from hyper's accept loop or the into_make_service_with_connect_info layer failing mid-serve. Graceful shutdown returns Ok, so hitting this means a real serving-layer failure: fd exhaustion (EMFILE), kernel memory pressure, or a connection-layer bug.

Source

Thrown at crates/buzz-relay/src/main.rs:1398

        return Ok(());
    }

    #[cfg(not(unix))]
    if config.uds_path.is_some() {
        tracing::warn!("BUZZ_UDS_PATH set but UDS not supported on this platform");
    }

    // TCP-only path.
    let mut tcp_rx = shutdown_tx.subscribe();
    axum::serve(
        tcp_listener,
        router.into_make_service_with_connect_info::<std::net::SocketAddr>(),
    )
    .with_graceful_shutdown(async move {
        tcp_rx.changed().await.ok();
    })
    .await
    .map_err(|e| anyhow::anyhow!("Server error: {e}"))?;

    let hard_shutdown = shutdown_handle
        .await
        .map_err(|e| anyhow::anyhow!("Shutdown task failed: {e}"))?;
    hard_shutdown.abort();
    Ok(())
}

/// Wait for SIGTERM (Unix) or Ctrl+C.
async fn shutdown_signal() {
    #[cfg(unix)]
    {
        use tokio::signal::unix::{signal, SignalKind};
        let mut sigterm = signal(SignalKind::terminate()).expect("install SIGTERM handler");
        tokio::select! {
            _ = tokio::signal::ctrl_c() => {},
            _ = sigterm.recv() => {},
        }

View on GitHub (pinned to f956e6fe06)

Solutions

1. Raise the fd limit: ulimit -n 65536, LimitNOFILE=65536 (systemd), or nofile in the container spec — EMFILE is the most common cause.
2. Correlate with connection counts: ls /proc/<pid>/fd | wc -l vs conn_manager occupancy to spot leaked sockets.
3. Check dmesg/journal for OOM or TCP memory pressure.
4. Preserve the full error text and report — bind already succeeded, so a recurring serve error is a resource issue or a bug.

Example fix

# before: relay exits under load: "Server error: Too many open files (os error 24)"

# after (docker-compose)
services:
  relay:
    ulimits:
      nofile:
        soft: 65536
        hard: 65536

// Pre-flight: verify fd headroom before the accept loop can hit EMFILE.
fn fd_headroom_ok(min_fds: usize) -> bool {
    let mut held = Vec::new();
    for _ in 0..min_fds {
        match std::net::TcpListener::bind(("127.0.0.1", 0)) {
            Ok(l) => held.push(l),
            Err(_) => return false,
        }
    }
    true
}

assert!(fd_headroom_ok(1024), "fd limit too low for the accept loop — raise nofile/LimitNOFILE");

fn is_fd_exhaustion(e: &std::io::Error) -> bool {
    matches!(e.raw_os_error(), Some(24) | Some(23)) // EMFILE / ENFILE on Linux
}

if let Err(e) = axum::serve(tcp_listener, svc)
    .with_graceful_shutdown(async move { tcp_rx.changed().await.ok(); })
    .await
{
    let msg = e.to_string();
    if msg.contains("Too many open files") {
        tracing::error!("fd limit hit mid-serve — raise LimitNOFILE and restart");
    }
    return Err(anyhow!("Server error: {e}"));
}

When it happens

Trigger: Default (non-UDS) deployments whose fd limit is exhausted by many concurrent WebSocket clients so accept() fails; ENOMEM under memory pressure; an error thrown by the connect-info extraction service for a particular peer. This is the variant every default deployment hits — no BUZZ_UDS_PATH involved.

Common situations: Production relays under connection spikes (mass reconnects after network events) with default 1024 fd limits; container orchestrators that do not raise nofile; slowly leaking fds over days of uptime.

Related errors

• TCP server error: {e}
• git conformance probe failed: {e}
• Failed to bind health port {}: {e}
• Failed to bind {}: {e}
• BUZZ_UDS_PATH {uds_path} exists but is not a socket

AI-assisted analysis of block/buzz@f956e6fe06 (2026-08-16). Data as JSON: /api/errors/dc9627bf276af257. Report an issue: GitHub.