linera-io/linera-protocol · error

Error serving metrics: {e}

Error message

Error serving metrics: {e}

What it means

start_metrics spawns a Tokio task that serves the axum-based metrics/prometheus endpoint. If axum::serve on the (already bound) listener returns an error — an accept-loop or socket IO failure — the task panics with 'Error serving metrics'. Because the listener is bound beforehand, this usually reflects socket-level trouble while serving rather than a plain port conflict, though misconfigured addresses and exhausted file descriptors are the everyday culprits.

Source

Thrown at linera-metrics/src/monitoring_server.rs:112

    memory_profiling: MemoryProfiling,
    register_metrics: impl FnOnce(),
) {
    crate::runtime_metrics::register();
    register_metrics();
    let app = metrics_router(memory_profiling);

    tokio::spawn(async move {
        let listener = tokio::net::TcpListener::bind(address)
            .await
            .expect("Failed to bind to address");
        let address = listener.local_addr().expect("Failed to get local address");

        info!("Starting to serve metrics on {:?}", address);
        if let Err(e) = axum::serve(listener, app)
            .with_graceful_shutdown(shutdown_signal.cancelled_owned())
            .await
        {
            panic!("Error serving metrics: {e}");
        }
    });
}

fn metrics_router(memory_profiling: MemoryProfiling) -> Router {
    #[cfg(feature = "jemalloc")]
    if memory_profiling == MemoryProfiling::Enabled {
        match MemoryProfiler::check_prof_ctl() {
            Ok(()) => {
                info!("Memory profiling enabled, registering /debug/pprof and /debug/flamegraph endpoints");
                return Router::new()
                    .route("/metrics", get(serve_metrics))
                    .route("/debug/pprof", get(MemoryProfiler::heap_profile))
                    .route("/debug/flamegraph", get(MemoryProfiler::heap_flamegraph));
            }
            Err(e) => {
                tracing::warn!(
                    "Memory profiling requested but not available: {}, serving metrics-only",

View on GitHub (pinned to 6c226ddcb3)

Solutions

1. Give each process a distinct, free metrics port (check with ss -ltnp)
2. Raise the file-descriptor limit (ulimit -n 4096 or LimitNOFILE= in systemd) if you run many connections
3. Confirm the configured metrics address is a valid, bindable socket address for the host
4. If metrics are not needed, start the process without the metrics endpoint enabled

Example fix

# before
LINERA_METRICS_SERVER=0.0.0.0:9100 ./node-a & LINERA_METRICS_SERVER=0.0.0.0:9100 ./node-b &

# after
LINERA_METRICS_SERVER=0.0.0.0:9100 ./node-a & LINERA_METRICS_SERVER=0.0.0.0:9101 ./node-b &

use std::net::TcpListener;
let listener = TcpListener::bind(metrics_address)
    .unwrap_or_else(|e| panic!("metrics address {metrics_address} unusable: {e}"));
drop(listener); // probe done before handing the address to start_metrics

When it happens

Trigger: Serving the metrics endpoint on an address whose socket errors during operation: fd exhaustion (ulimit -n) causing accept failures, the listener being closed/teared down unexpectedly, or an invalid/duplicated metrics address configuration across processes.

Common situations: Multiple Linera nodes/validators on one host all told to use the same metrics port; containerized deployments with low fd limits; the metrics port colliding with another service so the socket misbehaves.

Related errors

• Only allowed options are grpc and grpcs
• Failed to obtain a port
• HTTP server exited unexpectedly: {result:?}
• network protocol mismatch: cannot have {internal_protocol} a
• Failed to download certificates and update local node to the

AI-assisted analysis of linera-io/linera-protocol@6c226ddcb3 (2026-08-22). Data as JSON: /api/errors/207ab211e288c856. Report an issue: GitHub.