apache/hadoop · error · MetricsException

Error sending data

Error message

Error sending data

What it means

In KafkaSink.putMetrics(), after producer.send(data) the sink blocks on future.get(). If the waiting thread is interrupted, InterruptedException is caught and rethrown as MetricsException('Error sending data'). Unlike the ExecutionException path (5098), this variant means the publishing thread was cancelled while waiting for Kafka's ack — not that Kafka rejected the record.

Source

Thrown at hadoop-tools/hadoop-kafka/src/main/java/org/apache/hadoop/metrics2/sink/KafkaSink.java:180

    // Send the data to the Kafka broker. Here is an example of this data:
    // {"hostname": "...", "timestamp": 1436913651516,
    // "date": "2015-6-14","time": "22:40:51","context": "yarn","name":
    // "QueueMetrics, "running_0": "1", "running_60": "0", "running_300": "0",
    // "running_1440": "0", "AppsSubmitted": "1", "AppsRunning": "1",
    // "AppsPending": "0", "AppsCompleted": "0", "AppsKilled": "0",
    // "AppsFailed": "0", "AllocatedMB": "134656", "AllocatedVCores": "132",
    // "AllocatedContainers": "132", "AggregateContainersAllocated": "132",
    // "AggregateContainersReleased": "0", "AvailableMB": "0",
    // "AvailableVCores": "0", "PendingMB": "275456", "PendingVCores": "269",
    // "PendingContainers": "269", "ReservedMB": "0", "ReservedVCores": "0",
    // "ReservedContainers": "0", "ActiveUsers": "1", "ActiveApplications": "1"}
    Future<RecordMetadata> future = producer.send(data);
    jsonLines.setLength(0);
    try {
      future.get();
    } catch (InterruptedException e) {
      throw new MetricsException("Error sending data", e);
    } catch (ExecutionException e) {
      throw new MetricsException("Error sending data", e);
    }
  }

  @Override
  public void flush() {
    LOG.debug("Kafka seems not to have any flush() mechanism!");
  }

  @Override
  public void close() throws IOException {
    // Close the producer and set it to null.
    try {
      producer.close();
    } catch (RuntimeException e) {
      throw new MetricsException("Error closing producer", e);
    } finally {

View on GitHub (pinned to 2add963021)

Solutions

1. Treat as shutdown noise if it appears only during daemon stop/restart — verify Kafka health separately
2. Fix underlying slowness: check broker availability and network from Hadoop nodes (acks latency), since a fast get() rarely overlaps interrupts
3. If it recurs at runtime (not shutdown), capture the stack trace to identify which component is interrupting the metrics thread

try {
  sink.putMetrics(record);
} catch (MetricsException e) {
  if (e.getCause() instanceof InterruptedException) {
    Thread.currentThread().interrupt(); // restore flag
    // metrics thread cancelled (shutdown); stop publishing
    return;
  }
  throw e;
}

When it happens

Trigger: The metrics publishing thread is interrupted while blocked in future.get(): daemon shutdown (metrics system stopping sink threads), long Kafka ack latency combined with thread pool shutdown/now, or explicit cancellation of the metrics timer thread.

Common situations: Rolling restarts or shutdown of NameNode/ResourceManager while metrics are in flight; Kafka slow or unreachable (request.required.acks=0 still involves metadata I/O) so get() blocks until the interrupt arrives; executor shutdownNow() during reconfiguration of metrics.

Related errors

• Kafka topic can not be null
• Error creating Producer, {}
• Producer in KafkaSink is null!
• Error closing producer
• Failed to delete temporary files while closing stream: '%s'

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/799e0a3ff9a50471. Report an issue: GitHub.