apache/hadoop · error · MetricsException

Error creating connection, {}:{}

Error message

Error creating connection, {}:{}

What it means

Graphite.connect() opens a new Socket(serverHost, serverPort). Any failure — connection refused, unknown host, connect timeout — increments a connection-failure counter and throws MetricsException("Error creating connection, host:port", e) with the endpoint embedded. After 5 consecutive failures (MAX_CONNECTION_FAILURES) connect() returns silently, an ERROR is logged once, and metrics are dropped without further attempts until the connection succeeds again or the process restarts.

Source

Thrown at hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/sink/GraphiteSink.java:162

      if (isConnected()) {
        throw new MetricsException("Already connected to Graphite");
      }
      if (tooManyConnectionFailures()) {
        // return silently (there was ERROR in logs when we reached limit for the first time)
        return;
      }
      try {
          // Open a connection to Graphite server.
        socket = new Socket(serverHost, serverPort);
        writer = new OutputStreamWriter(socket.getOutputStream(),
                StandardCharsets.UTF_8);
      } catch (Exception e) {
        connectionFailures++;
        if (tooManyConnectionFailures()) {
          // first time when connection limit reached, report to logs
          LOG.error("Too many connection failures, would not try to connect again.");
        }
        throw new MetricsException("Error creating connection, " +
            serverHost + ":" + serverPort, e);
      }
    }

    public void write(String msg) throws IOException {
      if (!isConnected()) {
        connect();
      }
      if (isConnected()) {
        writer.write(msg);
      }
    }

    public void flush() throws IOException {
      if (isConnected()) {
        writer.flush();
      }
    }

View on GitHub (pinned to 2add963021)

Solutions

1. Verify the endpoint from the Hadoop node: nc -vz <server_host> <server_port> (default 2003)
2. Fix *.sink.graphite.server_host and *.sink.graphite.server_port in hadoop-metrics2.properties
3. Check DNS resolution and firewall/egress rules for the port
4. Restart the daemon after fixing — past 5 consecutive failures the sink stops trying silently

Example fix

# before
*.sink.graphite.server_host=graphite.example.com
*.sink.graphite.server_port=80  # web UI port, carbon not listening here

# after
*.sink.graphite.server_host=graphite.example.com
*.sink.graphite.server_port=2003

try (Socket s = new Socket()) {
  s.connect(new InetSocketAddress(serverHost, serverPort), 3000);
} catch (IOException e) {
  throw new IllegalStateException("Graphite endpoint " + serverHost + ":"
      + serverPort + " unreachable — fix metrics2 config before deploy", e);
}

try {
  graphite.write(lines);
} catch (MetricsException e) {
  // e.getMessage() contains 'Error creating connection, host:port' with the refused/timeout cause
  LOG.error("Cannot reach Graphite at {}: {}", endpoint, e.getCause(), e);
}

When it happens

Trigger: Nothing listening on the configured *.sink.graphite.server_host:server_port (connection refused); typo'd host (UnknownHostException); firewall dropping SYNs (connect timeout); Graphite container restarted on a changed IP.

Common situations: Wrong port for the carbon/Graphite listener (plaintext carbon is 2003, not the web 80/2004 pickled port); graphite host migration; DNS failure from the Hadoop node.

Related errors

• Error closing connection to Graphite
• Error closing connection to Graphite.
• Error connecting to file system: ${basePath} [${ex}]
• Could not find a free port in ${range}
• Illegal tag pattern: {}

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/e5497a38e618e5be. Report an issue: GitHub.