apache/hadoop · warning · MetricsException

Error closing producer

Error message

Error closing producer

What it means

KafkaSink.close() calls producer.close() and wraps any RuntimeException as MetricsException('Error closing producer'), always nulling the producer field in finally. A failing close usually means the producer buffer could not be drained: brokers became unreachable while undelivered records were pending, or the producer was already in a broken state from a prior failure.

Source

Thrown at hadoop-tools/hadoop-kafka/src/main/java/org/apache/hadoop/metrics2/sink/KafkaSink.java:197

    } catch (InterruptedException e) {
      throw new MetricsException("Error sending data", e);
    } catch (ExecutionException e) {
      throw new MetricsException("Error sending data", e);
    }
  }

  @Override
  public void flush() {
    LOG.debug("Kafka seems not to have any flush() mechanism!");
  }

  @Override
  public void close() throws IOException {
    // Close the producer and set it to null.
    try {
      producer.close();
    } catch (RuntimeException e) {
      throw new MetricsException("Error closing producer", e);
    } finally {
      producer = null;
    }
  }
}

View on GitHub (pinned to 2add963021)

Solutions

1. Check Kafka health at shutdown time — this error typically accompanies 'Error sending data' entries earlier in the log
2. Set producer close/linger timeouts appropriately (via additional producer props) so buffered records do not block shutdown
3. If shutdown noise only, it can be ignored after confirming metrics otherwise flowed; the producer is nulled regardless, preventing leaks

try {
  sink.close();
} catch (IOException | MetricsException e) {
  // usually shutdown-time flush failure while Kafka is down; producer is nulled
  // regardless, so no leak — log and continue shutdown
  LOG.warn("Kafka sink close failed", e);
}

When it happens

Trigger: Daemon shutdown (metrics system closes sinks) while Kafka is down or slow, so producer.close() times out flushing buffered records and throws InterruptException/TimeoutException (RuntimeExceptions); closing after an earlier send failure left the producer in error state; double-close where the underlying client misbehaves.

Common situations: Rolling restarts of Hadoop daemons during a Kafka outage; close races with in-flight putMetrics on another thread; kafka-clients version quirks around close-timeout behavior.

Related errors

• Kafka topic can not be null
• Error creating Producer, {}
• Producer in KafkaSink is null!
• Error sending data
• Call interrupted

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/9582df4bea89844c. Report an issue: GitHub.