apache/hadoop · warning · MetricsException

Error writing metric to StatsD

Error message

Error writing metric to StatsD

What it means

StatsDSink sends each metric as one UDP datagram ('host.service.context.name.metricname:value|type'). writeMetric catches IOException from statsd.write and rethrows it as MetricsException("Error writing metric to StatsD") after logging WARN 'Error sending metrics to StatsD'. The IOException originates in createSocket (building the InetSocketAddress for server.host; failures are wrapped via NetUtils.wrapException) or in DatagramSocket.send. Because the transport is UDP, a merely stopped StatsD server does NOT produce this error — it indicates name-resolution or socket-level problems.

Source

Thrown at hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/sink/StatsDSink.java:155

      }
      StringBuilder line = new StringBuilder();
      line.append(buf.toString())
          .append(metric.name().replace(' ', '_'))
          .append(":")
          .append(metric.value())
          .append("|")
          .append(type);
      writeMetric(line.toString());
    }

  }

  public void writeMetric(String line) {
    try {
      statsd.write(line);
    } catch (IOException e) {
      LOG.warn("Error sending metrics to StatsD", e);
      throw new MetricsException("Error writing metric to StatsD", e);
    }
  }

  @Override
  public void flush() {
  }

  @Override
  public void close() throws IOException {
    statsd.close();
  }

  /**
   * Class that sends UDP packets to StatsD daemon.
   *
   */
  public static class StatsD {

View on GitHub (pinned to 2add963021)

Solutions

1. Verify resolution from the daemon host: getent hosts <server.host> / nslookup <server.host>
2. Fix *.sink.statsd.server.host / *.sink.statsd.server.port values
3. Recovery is automatic once DNS is fixed — write() re-creates the socket when it is null
4. Remember UDP gives no feedback if the server is merely down; do not chase this error for that case

Example fix

# before
*.sink.statsd.server.host=statsd  # unresolvable bare name
*.sink.statsd.server.port=8125

# after
*.sink.statsd.server.host=statsd.prod.example.com
*.sink.statsd.server.port=8125

InetSocketAddress addr = new InetSocketAddress(serverHost, serverPort);
if (addr.isUnresolved()) {
  throw new IllegalStateException("StatsD host does not resolve: " + serverHost
      + " — fix *.sink.statsd.server.host before deploy");
}

try {
  sink.writeMetric(line);
} catch (MetricsException e) {
  // UDP transport: thrown for DNS/socket problems (NetUtils.wrapException cause), not for a down server
  LOG.warn("StatsD write failed ({}); socket re-created lazily on next write",
      e.getCause() == null ? e : e.getCause().toString());
}

When it happens

Trigger: *.sink.statsd.server.host does not resolve (DNS typo, NXDOMAIN) so createSocket throws UnknownHostException; the datagram socket is closed concurrently (close() nulls it, so the next write re-creates it lazily); a send() IOException from the OS.

Common situations: Typo'd or short (unqualified) statsd hostname in hadoop-metrics2.properties; DNS outages in containers; the sink restarted after close() with stale host config.

Related errors

• Error closing connection to Graphite
• Error closing connection to Graphite.
• Error creating connection, {}:{}
• Error connecting to file system: ${basePath} [${ex}]
• Unresolved host: ${inetAddress}

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/e7cbd6c3fb042b97. Report an issue: GitHub.