apache/hadoop · error · MetricsException

Error creating connection

Error message

Error creating connection, {}:{}

What it means

Graphite.connect() opens a new Socket(serverHost, serverPort). Any failure — connection refused, unknown host, connect timeout — increments a connection-failure counter and throws MetricsException("Error creating connection, host:port", e) with the endpoint embedded. After 5 consecutive failures (MAX_CONNECTION_FAILURES) connect() returns silently, an ERROR is logged once, and metrics are dropped without further attempts until the connection succeeds again or the process restarts.

Solutions

1. Verify the endpoint from the Hadoop node: nc -vz <server_host> <server_port> (default 2003)
2. Fix *.sink.graphite.server_host and *.sink.graphite.server_port in hadoop-metrics2.properties
3. Check DNS resolution and firewall/egress rules for the port
4. Restart the daemon after fixing — past 5 consecutive failures the sink stops trying silently

Example fix

# before
*.sink.graphite.server_host=graphite.example.com
*.sink.graphite.server_port=80  # web UI port, carbon not listening here

# after
*.sink.graphite.server_host=graphite.example.com
*.sink.graphite.server_port=2003

try (Socket s = new Socket()) {
  s.connect(new InetSocketAddress(serverHost, serverPort), 3000);
} catch (IOException e) {
  throw new IllegalStateException("Graphite endpoint " + serverHost + ":"
      + serverPort + " unreachable — fix metrics2 config before deploy", e);
}

try {
  graphite.write(lines);
} catch (MetricsException e) {
  // e.getMessage() contains 'Error creating connection, host:port' with the refused/timeout cause
  LOG.error("Cannot reach Graphite at {}: {}", endpoint, e.getCause(), e);
}

When it happens

Trigger: Nothing listening on the configured *.sink.graphite.server_host:server_port (connection refused); typo'd host (UnknownHostException); firewall dropping SYNs (connect timeout); Graphite container restarted on a changed IP.

Common situations: Wrong port for the carbon/Graphite listener (plaintext carbon is 2003, not the web 80/2004 pickled port); graphite host migration; DNS failure from the Hadoop node.

Understand the failure class

Background: ECONNREFUSED and "connection refused" / "could not connect to server" errors: what they mean and how to fix them — this error's family across 44 libraries.

Related errors

• Error closing connection to Graphite
• Error closing connection to Graphite.
• Error connecting to file system
• Already connected to Graphite
• Could not find a free port in

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/e5497a38e618e5be. Report an issue: GitHub.

Appendix: source

Thrown at hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/sink/GraphiteSink.java:162

      if (isConnected()) {
        throw new MetricsException("Already connected to Graphite");
      }
      if (tooManyConnectionFailures()) {
        // return silently (there was ERROR in logs when we reached limit for the first time)
        return;
      }
      try {
          // Open a connection to Graphite server.
        socket = new Socket(serverHost, serverPort);
        writer = new OutputStreamWriter(socket.getOutputStream(),
                StandardCharsets.UTF_8);
      } catch (Exception e) {
        connectionFailures++;
        if (tooManyConnectionFailures()) {
          // first time when connection limit reached, report to logs
          LOG.error("Too many connection failures, would not try to connect again.");
        }
        throw new MetricsException("Error creating connection, " +
            serverHost + ":" + serverPort, e);
      }
    }

    public void write(String msg) throws IOException {
      if (!isConnected()) {
        connect();
      }
      if (isConnected()) {
        writer.write(msg);
      }
    }

    public void flush() throws IOException {
      if (isConnected()) {
        writer.flush();
      }
    }

View on GitHub (pinned to 2add963021)