apache/kafka · error · KafkaException

Unexpected error fetching metadata for topic ${topic}

Error message

Unexpected error fetching metadata for topic ${topic}

What it means

Async-consumer counterpart of error 234. Thrown as a generic KafkaException by TopicMetadataRequestManager.handleTopicMetadataResponse for any per-topic error in the MetadataResponse that is not INVALID_TOPIC_EXCEPTION, UNKNOWN_TOPIC_OR_PARTITION, or a RetriableException. Unlike the legacy path, the async consumer does not loop on retriable errors here — it propagates the retriable exception too, letting the caller decide. The wrapped cause carries the underlying Errors.exception().

Source

Thrown at clients/src/main/java/org/apache/kafka/clients/consumer/internals/TopicMetadataRequestManager.java:268

                // if there were errors, we need to check whether they were fatal or whether
                // we should just retry

                log.debug("Topic metadata fetch included errors: {}", errors);

                for (Map.Entry<String, Errors> errorEntry : errors.entrySet()) {
                    String topic = errorEntry.getKey();
                    Errors error = errorEntry.getValue();

                    if (error == Errors.INVALID_TOPIC_EXCEPTION)
                        throw new InvalidTopicException("Topic '" + topic + "' is invalid");
                    else if (error == Errors.UNKNOWN_TOPIC_OR_PARTITION)
                        // if a requested topic is unknown, we just continue and let it be absent
                        // in the returned map
                        continue;
                    else if (error.exception() instanceof RetriableException)
                        throw error.exception();
                    else
                        throw new KafkaException("Unexpected error fetching metadata for topic " + topic,
                            error.exception());
                }
            }

            HashMap<String, List<PartitionInfo>> topicsPartitionInfos = new HashMap<>();
            for (String topic : cluster.topics())
                topicsPartitionInfos.put(topic, cluster.partitionsForTopic(topic));
            return topicsPartitionInfos;
        }

        public String topic() {
            return topic;
        }
    }
}

View on GitHub (pinned to c31c9215e1)

Solutions

1. Inspect the wrapped cause (exception.getCause()) — the Errors code there is the real signal.
2. If cause is retriable (RetriableException), wrap the call site in a bounded retry with backoff.
3. Align client and broker versions to remove unrecognized-code ambiguity.
4. Enable DEBUG on org.apache.kafka.clients.consumer.internals.TopicMetadataRequestManager — it logs the full error map ('Topic metadata fetch included errors: ...').

Example fix

// before
consumer.partitionsFor(topic).get(); // bubbles up KafkaException with hidden cause

// after
try {
    consumer.partitionsFor(topic).get();
} catch (ExecutionException e) {
    Throwable cause = e.getCause() != null ? e.getCause() : e;
    if (cause instanceof RetriableException) {
        backoffAndRetry();
    } else {
        log.error("metadata failed; cause={} {}", cause.getClass().getSimpleName(), cause.getMessage());
    }
}

// No general client-side precheck; preflight reachability and version compatibility instead.
try (java.net.Socket s = new java.net.Socket()) {
    s.connect(new java.net.InetSocketAddress(bootstrapHost, bootstrapPort), 2000);
}

import org.apache.kafka.common.KafkaException;

/** True iff a throwable is the generic 'Unexpected error fetching metadata' wrapper. */
static boolean isUnexpectedMetadataError(Throwable t) {
    if (!(t instanceof KafkaException)) return false;
    String m = t.getMessage();
    return m != null && m.startsWith("Unexpected error fetching metadata for topic");
}

try {
    consumer.poll(Duration.ofMillis(1000));
} catch (org.apache.kafka.common.KafkaException e) {
    if (!isUnexpectedMetadataError(e)) throw e;
    Throwable cause = e.getCause();
    log.error("Unexpected metadata fetch error; cause={}", cause, e);
    if (cause instanceof org.apache.kafka.common.errors.RetriableException) {
        // backoff and retry
    } else {
        throw e;
    }
}

When it happens

Trigger: MetadataResponse.errors() returns an unexpected error code (e.g. TOPIC_AUTHORIZATION_FAILED when unauthorizedTopics is empty, CLUSTER_AUTHORIZATION_FAILED, NOT_CONTROLLER, or a future error code from a newer broker). Also fires for any RetriableException the async path elects to surface rather than retry.

Common situations: Client/broker version skew surfacing an unrecognized code; rare authorization paths; non-standard broker or sidecar that rewrites error codes; transient retriable errors (LEADER_NOT_AVAILABLE) the async layer surfaces instead of masking.

Related errors

• Unexpected error fetching metadata for topic ${topic}
• Not authorized to access topics: ${unauthorizedTopics}
• Topic '${topic}' is invalid
• Failed to get offsets by times in {}ms
• Not authorized to access topics: ${unauthorizedTopics}

AI-assisted analysis of apache/kafka@c31c9215e1 (2026-08-03). Data as JSON: /data/errors/7e1c0fb023bfee3b.json. Report an issue: GitHub.