apache/hadoop · error · MetricsException

Failed to putMetrics

Error message

Failed to putMetrics

What it means

MetricsException('Failed to putMetrics') thrown by GangliaSink30.putMetrics() wrapping any IOException raised while building and emitting metric datagrams (typically from emitToGangliaHosts sending to the gmond collector). The original IOException is preserved as the cause, so the real network failure (send error, unresolved host, closed socket) is visible via getCause(). The metrics2 framework catches MetricsException in MetricsSinkAdapter, logs it, and continues, so it usually appears as a recurring error line rather than a crash.

Source

Thrown at hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/sink/ganglia/GangliaSink30.java:191

            // slope
            metric.visit(gangliaMetricVisitor);
            type = gangliaMetricVisitor.getType();
            slopeFromMetric = gangliaMetricVisitor.getSlope();

            GangliaConf gConf = getGangliaConfForMetric(name);
            calculatedSlope = calculateSlope(gConf, slopeFromMetric);

            // send metric to Ganglia
            emitMetric(groupName, name, type, metric.value().toString(), gConf,
                calculatedSlope);

            // reset the length of the buffer for next iteration
            sb.setLength(sbBaseLen);
          }
        }
      }
    } catch (IOException io) {
      throw new MetricsException("Failed to putMetrics", io);
    }
  }

  // Calculate the slope from properties and metric
  private GangliaSlope calculateSlope(GangliaConf gConf,
      GangliaSlope slopeFromMetric) {
    if (gConf.getSlope() != null) {
      // if slope has been specified in properties, use that
      return gConf.getSlope();
    } else if (slopeFromMetric != null) {
      // slope not specified in properties, use derived from Metric
      return slopeFromMetric;
    } else {
      return DEFAULT_SLOPE;
    }
  }

  /**

View on GitHub (pinned to 2add963021)

Solutions

1. Read the cause: catch MetricsException and inspect getCause() — the wrapped IOException states the real failure
2. Verify gmond is listening on the configured UDP port on the target: 'nc -u -v <gangliaHost> 8649' or check 'ss -lun' on the collector
3. Check firewall rules allow outbound UDP to the ganglia port from every Hadoop node
4. Confirm the '<prefix>.sink.ganglia.servers' host:port values and that the host resolves (see UnknownHostException 'Unresolved host' from the same sink)

Example fix

// before
sink.putMetrics(record);  // throws MetricsException("Failed to putMetrics", io) every flush cycle

// after
// fix the collector side: start gmond on 10.0.0.5:8649 (systemctl start gmond)
// and validate reachability: nc -u -vz 10.0.0.5 8649

// Check UDP reachability before enabling the sink in production
// nc -u -vz <gangliaHost> 8649  (shell-level check, not in-process)

try {
  sink.putMetrics(record);
} catch (MetricsException e) {
  Throwable cause = e.getCause(); // the real IOException
  LOG.warn("Ganglia emit failed: {}", cause, cause instanceof UnknownHostException ? "fix servers property" : "check gmond/network");
}

When it happens

Trigger: datagramSocket.send() failing with network-unreachable / socket-closed; an unresolved ganglia host raising UnknownHostException inside emitToGangliaHosts; gmond not running or not listening on the configured UDP port so ICMP port-unreachable is reflected back to the sender.

Common situations: gmond (Ganglia monitoring daemon) down or restarted; firewall between the Hadoop node and the ganglia collector blocking UDP 8649; wrong port in the servers property; GangliaSink30 multicast configuration pointing at a bad multicast channel.

Related errors

• Unsupported Address type
• Unresolved host: ${inetAddress}
• Unable to write to output stream.
• Error while reading compressed data
• Premature EOF from inputStream

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/91252b0d17906035. Report an issue: GitHub.