apache/kafka · error · IOException

Connection to {node} failed.

Error message

Connection to {node} failed.

What it means

Thrown as IOException by NetworkClientUtils.awaitReady when, while polling for the connection to come up, client.connectionFailed(node) returns true. It reports that the connection attempt to the node failed before the ready condition was reached (as opposed to a timeout, which returns false). Because the underlying NetworkClient may surface a recently disconnected previous connection, the failure can predate the current awaitReady call.

Source

Thrown at clients/src/main/java/org/apache/kafka/clients/NetworkClientUtils.java:71

     * connection timeoutMs, it is possible for this method to raise an `IOException` for a previous connection which
     * has recently disconnected. If authentication to the node fails, an `AuthenticationException` is thrown.
     *
     * This method is useful for implementing blocking behaviour on top of the non-blocking `NetworkClient`, use it with
     * care.
     */
    public static boolean awaitReady(KafkaClient client, Node node, Time time, long timeoutMs) throws IOException {
        if (timeoutMs < 0) {
            throw new IllegalArgumentException("Timeout needs to be greater than 0");
        }
        long startTime = time.milliseconds();

        if (isReady(client, node, startTime) ||  client.ready(node, startTime))
            return true;

        long attemptStartTime = time.milliseconds();
        while (!client.isReady(node, attemptStartTime) && attemptStartTime - startTime < timeoutMs) {
            if (client.connectionFailed(node)) {
                throw new IOException("Connection to " + node + " failed.");
            }
            long pollTimeout = timeoutMs - (attemptStartTime - startTime); // initialize in this order to avoid overflow

            // If the network client is waiting to send data for some reason (eg. throttling or retry backoff),
            // polling longer than that is potentially dangerous as the producer will not attempt to send
            // any pending requests.
            long waitingTime = client.pollDelayMs(node, startTime);
            if (waitingTime > 0 && pollTimeout > waitingTime) {
                // Block only until the next-scheduled time that it's okay to send data to the producer,
                // wake up, and try again. This is the way.
                pollTimeout = waitingTime;
            }

            client.poll(pollTimeout, attemptStartTime);
            if (client.authenticationException(node) != null)
                throw client.authenticationException(node);
            attemptStartTime = time.milliseconds();
        }

View on GitHub (pinned to c31c9215e1)

Solutions

  1. Confirm the broker process is up and the listener is bound: nc -zv brokerHost 9092.
  2. Compare advertised.listeners on the broker against the node address the client received; fix the broker config and roll.
  3. Validate the security.protocol / SSL settings on the client match the listener (PLAINTEXT/SSL/SASL).
  4. Check firewalls, security groups, and DNS for the node host:port.
  5. Inspect broker logs for the period around the connect attempt (auth, SSL, listener errors).

Example fix

# before
advertised.listeners=PLAINTEXT://localhost:9092
# after (advertise a reachable address)
advertised.listeners=PLAINTEXT://broker1.internal:9092
Defensive patterns

Strategy: retry

Validate before calling

// Pre-check connection state to fail fast with actionable context.
if (networkClient.connectionFailed(node)) {
    long delay = networkClient.connectionDelay(node, time.milliseconds());
    throw new java.io.IOException(
        "Node " + node + " is in failed state; reconnect backoff " + delay + " ms remaining.");
}
// Useful when you want a clearer error than the generic IOException from awaitReady.

Type guard

// Predicate separating transient backoff from permanent failure.
public static boolean inReconnectBackoff(KafkaClient client, Node node, Time time) {
    return client.connectionFailed(node)
        && client.connectionDelay(node, time.milliseconds()) > 0;
}
// If true, wait out the backoff rather than hammering awaitReady.

Try / catch

long deadline = time.milliseconds() + totalTimeoutMs;
while (time.milliseconds() < deadline) {
    try {
        if (NetworkClientUtils.awaitReady(client, node, time, Math.max(0, deadline - time.milliseconds()))) {
            return; // connected
        }
    } catch (java.io.IOException e) {
        if (e.getMessage() != null && e.getMessage().startsWith("Connection to ") && e.getMessage().endsWith(" failed.")) {
        // Transient connect failure. Honor reconnect.backoff.ms / reconnect.backoff.max.ms,
        // refresh metadata, then retry until the deadline.
        Thread.sleep(reconnectBackoffMs);
        continue;
        }
        throw e;
    }
}

Prevention

When it happens

Trigger: awaitReady polls the client and observes connectionFailed(node) true for the target Node. This happens when TCP connect is refused, the SSL handshake aborts, the broker rejects the connection, or a prior connection to the same node was lost within the connection-failure window. The Node instance in the message identifies the target.

Common situations: Firewall/security group blocking the broker port; broker down or still starting; wrong port or protocol (e.g. plaintext against an SSL listener); TLS misconfiguration; advertised.listeners pointing at an unreachable address; client connecting to a broker id that has been decommissioned.

Related errors

AI-assisted analysis of apache/kafka@c31c9215e1 (2026-08-03). Data as JSON: /data/errors/d8d13fd86778842c.json. Report an issue: GitHub.