zhisheng17/flink-learning · error · RuntimeException

Could not start PrometheusReporter HTTP server on any config

Error message

Could not start PrometheusReporter HTTP server on any configured port. Ports: ${portsConfig}

What it means

PrometheusReporter starts an embedded HTTP server (com.sun.net.httpserver.HttpServer) on one of the configured ports so Prometheus can scrape it. If binding every configured port fails with IOException (assumed port conflict), it throws RuntimeException listing the attempted ports.

Source

Thrown at flink-learning-extends/flink-metrics/flink-metrics-prometheus/src/main/java/org/apache/flink/metrics/prometheus/PrometheusReporter.java:73

		super.open(config);

		String portsConfig = config.getString(ARG_PORT, DEFAULT_PORT);
		Iterator<Integer> ports = NetUtils.getPortRangeFromString(portsConfig);

		while (ports.hasNext()) {
			int port = ports.next();
			try {
				// internally accesses CollectorRegistry.defaultRegistry
				httpServer = new HTTPServer(port);
				this.port = port;
				log.info("Started PrometheusReporter HTTP server on port {}.", port);
				break;
			} catch (IOException ioe) { //assume port conflict
				log.debug("Could not start PrometheusReporter HTTP server on port {}.", port, ioe);
			}
		}
		if (httpServer == null) {
			throw new RuntimeException("Could not start PrometheusReporter HTTP server on any configured port. Ports: " + portsConfig);
		}
	}

	@Override
	public void close() {
		if (httpServer != null) {
			httpServer.stop();
		}

		super.close();
	}

}

View on GitHub (pinned to d731cee761)

Solutions

Configure a port range: metrics.reporter.promgateway.port: 9250-9300 so each instance picks a free port.
Check who holds the ports: ss/lsof -i :9250 and kill stale processes.
Run at most one reporter instance per host/port or use unique ports per TaskManager.
Ensure Prometheus scrapes the dynamic instance ports accordingly.

Example fix

# before
metrics.reporter.prom.port: 9250
# after
metrics.reporter.prom.port: 9250-9350

Defensive patterns

Strategy: try-catch

Validate before calling

int[] ports = {9250};
for (int p : ports)
    try (java.net.ServerSocket s = new java.net.ServerSocket(p)) { }
    catch (java.io.IOException e) { System.out.println("Port " + p + " busy, widen reporter port range"); }

Try / catch

try {
    reporter.open(metrics);
} catch (RuntimeException e) {
    LOG.warn("Prometheus reporter could not bind ports: {}", e.getMessage());
    // continue without metrics or retry with different range
}

Prevention

Configure a port range (e.g. 9250-9350) instead of a single fixed port
Check port occupancy with ss/lsof before rollout
Avoid collocating many TaskManagers with identical single-port reporter configs
In containers, use unique ports per instance or separate network namespaces

When it happens

Trigger: All ports in metrics.reporter.<name>.port (comma-separated list) are already bound by other processes, often multiple Flink TaskManager/Reporter instances on the same host all fighting for the same fixed port.

Common situations: Several TaskManagers colocated on one machine with the same single port configured; a stale process still holding the port; containerized deployments sharing the host network without unique ports orPorts config.

Related errors

AI-assisted analysis of zhisheng17/flink-learning@d731cee761 (2026-09-06). Data as JSON: /api/errors/c955b5e5757d4167. Report an issue: GitHub.