{"record":{"id":"1ad9b23f6eb851b3","repo":"apache/hadoop","slug":"error-closing-connection-to-graphite","errorCode":null,"errorMessage":"Error closing connection to Graphite","messagePattern":"Error closing connection to Graphite","errorType":"exception","errorClass":"MetricsException","httpStatus":null,"severity":"warning","filePath":"hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/sink/GraphiteSink.java","lineNumber":105,"sourceCode":"\n    // The record timestamp is in milliseconds while Graphite expects an epoc time in seconds.\n    long timestamp = record.timestamp() / 1000L;\n\n    // Collect datapoints.\n    for (AbstractMetric metric : record.metrics()) {\n      lines.append(metricsPathPrefix + \".\" + metric.name().replace(' ', '.')).append(\" \")\n           .append(metric.value()).append(\" \").append(timestamp)\n           .append(\"\\n\");\n    }\n\n    try {\n      graphite.write(lines.toString());\n    } catch (Exception e) {\n      LOG.warn(\"Error sending metrics to Graphite.\", e);\n      try {\n        graphite.close();\n      } catch (Exception e1) {\n        throw new MetricsException(\"Error closing connection to Graphite\", e1);\n      }\n    }\n  }\n\n  @Override\n  public void flush() {\n    try {\n      graphite.flush();\n    } catch (Exception e) {\n      LOG.warn(\"Error flushing metrics to Graphite.\", e);\n      try {\n        graphite.close();\n      } catch (Exception e1) {\n        throw new MetricsException(\"Error closing connection to Graphite.\", e1);\n      }\n    }\n  }\n","sourceCodeStart":87,"sourceCodeEnd":123,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/sink/GraphiteSink.java#L87-L123","documentation":"GraphiteSink.putMetrics writes the formatted metric lines over a TCP socket to the Graphite server. If the write throws (broken or reset connection), the sink logs WARN \"Error sending metrics to Graphite.\" and calls graphite.close() to force a clean reconnect on the next cycle; if that close also throws, this MetricsException(\"Error closing connection to Graphite\", e1) propagates. The exception you observe is therefore the secondary close failure — the original network error is only in the WARN line immediately before it.","triggerScenarios":"Graphite server stopped or restarted mid-session: the write fails on the stale socket and closing the already-broken socket throws too. Also firewalls/NAT silently dropping idle TCP connections so both write and close fail.","commonSituations":"Graphite host migrations or restarts; load balancer idle timeouts shorter than the metrics flush period; transient network partitions between the Hadoop node and the Graphite collector.","solutions":["Read the paired WARN 'Error sending metrics to Graphite.' log line — it carries the root cause exception","Restore reachability of the Graphite server (check *.sink.graphite.server_host / server_port) and firewall rules","Do nothing for a single occurrence: write() lazily reconnects on the next metrics cycle","If failures repeat, note the sink gives up after 5 consecutive connection failures (MAX_CONNECTION_FAILURES) — restart the daemon after fixing the endpoint"],"exampleFix":"# before\n*.sink.graphite.server_host=graphite-old.prod.example.com  # decommissioned host\n\n# after\n*.sink.graphite.server_host=graphite-new.prod.example.com\n*.sink.graphite.server_port=2003","handlingStrategy":"retry","validationCode":"// preflight before relying on the sink: verify the graphite endpoint is reachable\ntry (Socket s = new Socket()) {\n  s.connect(new InetSocketAddress(graphiteHost, graphitePort), 3000);\n} catch (IOException e) {\n  LOG.warn(\"Graphite not reachable; sink will retry lazily: {}\", e.toString());\n}","typeGuard":null,"tryCatchPattern":"try {\n  sink.putMetrics(record);\n} catch (MetricsException e) {\n  // thrown only when BOTH write and close failed; root cause is the WARN right before\n  // no action needed: next cycle lazily reconnects via write() -> connect()\n  LOG.warn(\"Graphite sink write+close failed; will reconnect next cycle\", e);\n}","preventionTips":["Keep Graphite flush/idle timeouts longer than the metrics sink flush period so TCP connections are not dropped between writes","Monitor for the paired WARN 'Error sending metrics to Graphite.' — it carries the root cause, not the thrown exception","Remember the sink stops reconnecting after 5 consecutive failures; alert on repeated errors and restart the daemon after fixing the endpoint"],"tags":["graphite","metrics2","network","socket","sink"],"backgroundTag":"socket-write-failed","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}