apache/hadoop · error · MetricsException

Kafka topic can not be null

Error message

Kafka topic can not be null

What it means

KafkaSink.init() reads the 'topic' property from the metrics2 sink configuration (hadoop-metrics2.properties, e.g. *.sink.kafka.topic). If it is null or empty (Strings.isNullOrEmpty), init throws MetricsException('Kafka topic can not be null'). The sink publishes every metrics record to this single Kafka topic, so the property is mandatory.

Source

Thrown at hadoop-tools/hadoop-kafka/src/main/java/org/apache/hadoop/metrics2/sink/KafkaSink.java:96

  public void init(SubsetConfiguration conf) {
    // Get Kafka broker configuration.
    Properties props = new Properties();
    brokerList = conf.getString(BROKER_LIST);
    if (LOG.isDebugEnabled()) {
      LOG.debug("Broker list " + brokerList);
    }
    props.put("bootstrap.servers", brokerList);
    if (LOG.isDebugEnabled()) {
      LOG.debug("Kafka brokers: " + brokerList);
    }

    // Get Kafka topic configuration.
    topic = conf.getString(TOPIC);
    if (LOG.isDebugEnabled()) {
      LOG.debug("Kafka topic " + topic);
    }
    if (Strings.isNullOrEmpty(topic)) {
      throw new MetricsException("Kafka topic can not be null");
    }

    // Set the rest of Kafka configuration.
    props.put("key.serializer",
        "org.apache.kafka.common.serialization.ByteArraySerializer");
    props.put("value.serializer",
        "org.apache.kafka.common.serialization.ByteArraySerializer");
    props.put("request.required.acks", "0");

    // Set the hostname once and use it in every message.
    hostname = "null";
    try {
      hostname = InetAddress.getLocalHost().getHostName();
    } catch (Exception e) {
      LOG.warn("Error getting Hostname, going to continue");
    }

    System.setProperty("org.apache.kafka.automatic.config.providers", "none");

View on GitHub (pinned to 2add963021)

Solutions

1. Add the topic key to the sink block: *.sink.kafka.topic=<your-topic> alongside *.sink.kafka.broker_list
2. Check exact spelling: the keys are 'broker_list' and 'topic' under the sink prefix (e.g. resourcemanager.sink.kafka.topic)
3. Verify the topic exists on the Kafka cluster (or auto-create is enabled) so later sends do not fail

Example fix

# before (hadoop-metrics2.properties)
resourcemanager.sink.kafka.class=org.apache.hadoop.metrics2.sink.KafkaSink
resourcemanager.sink.kafka.broker_list=kafka1:9092,kafka2:9092

# after
resourcemanager.sink.kafka.class=org.apache.hadoop.metrics2.sink.KafkaSink
resourcemanager.sink.kafka.broker_list=kafka1:9092,kafka2:9092
resourcemanager.sink.kafka.topic=hadoop-metrics

// Validate hadoop-metrics2.properties before deploy
Properties p = load("hadoop-metrics2.properties");
String topic = p.getProperty("*.sink.kafka.topic");
if (topic == null || topic.trim().isEmpty()) {
  throw new IllegalStateException("KafkaSink requires *.sink.kafka.topic");
}

When it happens

Trigger: Configuring a kafka sink block in hadoop-metrics2.properties that sets broker_list but omits topic (or sets it to an empty string); misspelling the key (topics, kafka.topic) so conf.getString(TOPIC) returns null; enabling the sink via *.sink.kafka.class without any properties.

Common situations: First-time setup of the hadoop-kafka metrics sink; copying an example config that comments out the topic line; property-name drift between Hadoop versions or documentation typos; trailing whitespace-only values are not caught by IsNullOrEmpty and will fail later at send time instead.

Related errors

• Illegal tag pattern: {}
• Error creating plugin: {}
• Error stringify config
• Error creating Producer, {}
• Producer in KafkaSink is null!

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/f7ce04cf2735b532. Report an issue: GitHub.