apache/hadoop · warning · MetricsException

Error closing connection to Graphite

Error message

Error closing connection to Graphite

What it means

GraphiteSink.putMetrics writes the formatted metric lines over a TCP socket to the Graphite server. If the write throws (broken or reset connection), the sink logs WARN "Error sending metrics to Graphite." and calls graphite.close() to force a clean reconnect on the next cycle; if that close also throws, this MetricsException("Error closing connection to Graphite", e1) propagates. The exception you observe is therefore the secondary close failure — the original network error is only in the WARN line immediately before it.

Source

Thrown at hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/sink/GraphiteSink.java:105

    // The record timestamp is in milliseconds while Graphite expects an epoc time in seconds.
    long timestamp = record.timestamp() / 1000L;

    // Collect datapoints.
    for (AbstractMetric metric : record.metrics()) {
      lines.append(metricsPathPrefix + "." + metric.name().replace(' ', '.')).append(" ")
           .append(metric.value()).append(" ").append(timestamp)
           .append("\n");
    }

    try {
      graphite.write(lines.toString());
    } catch (Exception e) {
      LOG.warn("Error sending metrics to Graphite.", e);
      try {
        graphite.close();
      } catch (Exception e1) {
        throw new MetricsException("Error closing connection to Graphite", e1);
      }
    }
  }

  @Override
  public void flush() {
    try {
      graphite.flush();
    } catch (Exception e) {
      LOG.warn("Error flushing metrics to Graphite.", e);
      try {
        graphite.close();
      } catch (Exception e1) {
        throw new MetricsException("Error closing connection to Graphite.", e1);
      }
    }
  }

View on GitHub (pinned to 2add963021)

Solutions

1. Read the paired WARN 'Error sending metrics to Graphite.' log line — it carries the root cause exception
2. Restore reachability of the Graphite server (check *.sink.graphite.server_host / server_port) and firewall rules
3. Do nothing for a single occurrence: write() lazily reconnects on the next metrics cycle
4. If failures repeat, note the sink gives up after 5 consecutive connection failures (MAX_CONNECTION_FAILURES) — restart the daemon after fixing the endpoint

Example fix

# before
*.sink.graphite.server_host=graphite-old.prod.example.com  # decommissioned host

# after
*.sink.graphite.server_host=graphite-new.prod.example.com
*.sink.graphite.server_port=2003

// preflight before relying on the sink: verify the graphite endpoint is reachable
try (Socket s = new Socket()) {
  s.connect(new InetSocketAddress(graphiteHost, graphitePort), 3000);
} catch (IOException e) {
  LOG.warn("Graphite not reachable; sink will retry lazily: {}", e.toString());
}

try {
  sink.putMetrics(record);
} catch (MetricsException e) {
  // thrown only when BOTH write and close failed; root cause is the WARN right before
  // no action needed: next cycle lazily reconnects via write() -> connect()
  LOG.warn("Graphite sink write+close failed; will reconnect next cycle", e);
}

When it happens

Trigger: Graphite server stopped or restarted mid-session: the write fails on the stale socket and closing the already-broken socket throws too. Also firewalls/NAT silently dropping idle TCP connections so both write and close fail.

Common situations: Graphite host migrations or restarts; load balancer idle timeouts shorter than the metrics flush period; transient network partitions between the Hadoop node and the Graphite collector.

Related errors

• Error closing connection to Graphite.
• Error creating connection, {}:{}
• Already connected to Graphite
• Error connecting to file system: ${basePath} [${ex}]
• Error writing metric to StatsD

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/1ad9b23f6eb851b3. Report an issue: GitHub.