{"record":{"id":"c955b5e5757d4167","repo":"zhisheng17/flink-learning","slug":"could-not-start-prometheusreporter-http-server-on","errorCode":null,"errorMessage":"Could not start PrometheusReporter HTTP server on any configured port. Ports: ${portsConfig}","messagePattern":"Could not start PrometheusReporter HTTP server on any configured port\\. Ports: (.+?)","errorType":"exception","errorClass":"RuntimeException","httpStatus":null,"severity":"error","filePath":"flink-learning-extends/flink-metrics/flink-metrics-prometheus/src/main/java/org/apache/flink/metrics/prometheus/PrometheusReporter.java","lineNumber":73,"sourceCode":"\t\tsuper.open(config);\n\n\t\tString portsConfig = config.getString(ARG_PORT, DEFAULT_PORT);\n\t\tIterator<Integer> ports = NetUtils.getPortRangeFromString(portsConfig);\n\n\t\twhile (ports.hasNext()) {\n\t\t\tint port = ports.next();\n\t\t\ttry {\n\t\t\t\t// internally accesses CollectorRegistry.defaultRegistry\n\t\t\t\thttpServer = new HTTPServer(port);\n\t\t\t\tthis.port = port;\n\t\t\t\tlog.info(\"Started PrometheusReporter HTTP server on port {}.\", port);\n\t\t\t\tbreak;\n\t\t\t} catch (IOException ioe) { //assume port conflict\n\t\t\t\tlog.debug(\"Could not start PrometheusReporter HTTP server on port {}.\", port, ioe);\n\t\t\t}\n\t\t}\n\t\tif (httpServer == null) {\n\t\t\tthrow new RuntimeException(\"Could not start PrometheusReporter HTTP server on any configured port. Ports: \" + portsConfig);\n\t\t}\n\t}\n\n\t@Override\n\tpublic void close() {\n\t\tif (httpServer != null) {\n\t\t\thttpServer.stop();\n\t\t}\n\n\t\tsuper.close();\n\t}\n\n}\n","sourceCodeStart":55,"sourceCodeEnd":87,"githubUrl":"https://github.com/zhisheng17/flink-learning/blob/d731cee7618021be56d132cc925102ffff8d75e6/flink-learning-extends/flink-metrics/flink-metrics-prometheus/src/main/java/org/apache/flink/metrics/prometheus/PrometheusReporter.java#L55-L87","documentation":"PrometheusReporter starts an embedded HTTP server (com.sun.net.httpserver.HttpServer) on one of the configured ports so Prometheus can scrape it. If binding every configured port fails with IOException (assumed port conflict), it throws RuntimeException listing the attempted ports.","triggerScenarios":"All ports in metrics.reporter.<name>.port (comma-separated list) are already bound by other processes, often multiple Flink TaskManager/Reporter instances on the same host all fighting for the same fixed port.","commonSituations":"Several TaskManagers colocated on one machine with the same single port configured; a stale process still holding the port; containerized deployments sharing the host network without unique ports orPorts config.","solutions":["Configure a port range: metrics.reporter.promgateway.port: 9250-9300 so each instance picks a free port.","Check who holds the ports: ss/lsof -i :9250 and kill stale processes.","Run at most one reporter instance per host/port or use unique ports per TaskManager.","Ensure Prometheus scrapes the dynamic instance ports accordingly."],"exampleFix":"# before\nmetrics.reporter.prom.port: 9250\n# after\nmetrics.reporter.prom.port: 9250-9350","handlingStrategy":"try-catch","validationCode":"int[] ports = {9250};\nfor (int p : ports)\n    try (java.net.ServerSocket s = new java.net.ServerSocket(p)) { }\n    catch (java.io.IOException e) { System.out.println(\"Port \" + p + \" busy, widen reporter port range\"); }","typeGuard":null,"tryCatchPattern":"try {\n    reporter.open(metrics);\n} catch (RuntimeException e) {\n    LOG.warn(\"Prometheus reporter could not bind ports: {}\", e.getMessage());\n    // continue without metrics or retry with different range\n}","preventionTips":["Configure a port range (e.g. 9250-9350) instead of a single fixed port","Check port occupancy with ss/lsof before rollout","Avoid collocating many TaskManagers with identical single-port reporter configs","In containers, use unique ports per instance or separate network namespaces"],"tags":["flink","metrics","prometheus","port-conflict"],"backgroundTag":"address-already-in-use","analyzedSha":"d731cee7618021be56d132cc925102ffff8d75e6","analyzedAt":"2026-09-06T05:35:08.496Z","contentChangedAt":"2026-09-06T05:35:08.496Z","schemaVersion":2},"datasetVersion":"2026-09-14T05:17:10.506Z"}