{"record":{"id":"207ab211e288c856","repo":"linera-io/linera-protocol","slug":"error-serving-metrics-e","errorCode":null,"errorMessage":"Error serving metrics: {e}","messagePattern":"Error serving metrics: (.+?)","errorType":"panic","errorClass":null,"httpStatus":null,"severity":"error","filePath":"linera-metrics/src/monitoring_server.rs","lineNumber":112,"sourceCode":"    memory_profiling: MemoryProfiling,\n    register_metrics: impl FnOnce(),\n) {\n    crate::runtime_metrics::register();\n    register_metrics();\n    let app = metrics_router(memory_profiling);\n\n    tokio::spawn(async move {\n        let listener = tokio::net::TcpListener::bind(address)\n            .await\n            .expect(\"Failed to bind to address\");\n        let address = listener.local_addr().expect(\"Failed to get local address\");\n\n        info!(\"Starting to serve metrics on {:?}\", address);\n        if let Err(e) = axum::serve(listener, app)\n            .with_graceful_shutdown(shutdown_signal.cancelled_owned())\n            .await\n        {\n            panic!(\"Error serving metrics: {e}\");\n        }\n    });\n}\n\nfn metrics_router(memory_profiling: MemoryProfiling) -> Router {\n    #[cfg(feature = \"jemalloc\")]\n    if memory_profiling == MemoryProfiling::Enabled {\n        match MemoryProfiler::check_prof_ctl() {\n            Ok(()) => {\n                info!(\"Memory profiling enabled, registering /debug/pprof and /debug/flamegraph endpoints\");\n                return Router::new()\n                    .route(\"/metrics\", get(serve_metrics))\n                    .route(\"/debug/pprof\", get(MemoryProfiler::heap_profile))\n                    .route(\"/debug/flamegraph\", get(MemoryProfiler::heap_flamegraph));\n            }\n            Err(e) => {\n                tracing::warn!(\n                    \"Memory profiling requested but not available: {}, serving metrics-only\",","sourceCodeStart":94,"sourceCodeEnd":130,"githubUrl":"https://github.com/linera-io/linera-protocol/blob/6c226ddcb332ef55118dc8d0aafbd093d5420899/linera-metrics/src/monitoring_server.rs#L94-L130","documentation":"start_metrics spawns a Tokio task that serves the axum-based metrics/prometheus endpoint. If axum::serve on the (already bound) listener returns an error — an accept-loop or socket IO failure — the task panics with 'Error serving metrics'. Because the listener is bound beforehand, this usually reflects socket-level trouble while serving rather than a plain port conflict, though misconfigured addresses and exhausted file descriptors are the everyday culprits.","triggerScenarios":"Serving the metrics endpoint on an address whose socket errors during operation: fd exhaustion (ulimit -n) causing accept failures, the listener being closed/teared down unexpectedly, or an invalid/duplicated metrics address configuration across processes.","commonSituations":"Multiple Linera nodes/validators on one host all told to use the same metrics port; containerized deployments with low fd limits; the metrics port colliding with another service so the socket misbehaves.","solutions":["Give each process a distinct, free metrics port (check with ss -ltnp)","Raise the file-descriptor limit (ulimit -n 4096 or LimitNOFILE= in systemd) if you run many connections","Confirm the configured metrics address is a valid, bindable socket address for the host","If metrics are not needed, start the process without the metrics endpoint enabled"],"exampleFix":"# before\nLINERA_METRICS_SERVER=0.0.0.0:9100 ./node-a & LINERA_METRICS_SERVER=0.0.0.0:9100 ./node-b &\n\n# after\nLINERA_METRICS_SERVER=0.0.0.0:9100 ./node-a & LINERA_METRICS_SERVER=0.0.0.0:9101 ./node-b &","handlingStrategy":"validation","validationCode":"use std::net::TcpListener;\nlet listener = TcpListener::bind(metrics_address)\n    .unwrap_or_else(|e| panic!(\"metrics address {metrics_address} unusable: {e}\"));\ndrop(listener); // probe done before handing the address to start_metrics","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Assign a unique metrics port per process via configuration management","Monitor fd usage (ulimit -n) on hosts running many connections","Smoke-test the /metrics endpoint after startup so serve failures surface immediately"],"tags":["metrics","prometheus","axum","network","observability"],"backgroundTag":"metrics-server-startup-failure","analyzedSha":"6c226ddcb332ef55118dc8d0aafbd093d5420899","analyzedAt":"2026-08-22T22:49:09.787Z","schemaVersion":2},"datasetVersion":"2026-08-23T01:17:44.959Z"}