apache/kafka · error · IOException

Connection to {destination} was disconnected before the resp

Error message

Connection to {destination} was disconnected before the response was read

What it means

Thrown as IOException by NetworkClientUtils.sendAndReceive when a ClientResponse matching the request's correlation id is found but its wasDisconnected() flag is true. It means the request was sent successfully, then the connection to the destination node was lost before the full response was read back. The destination in the message is the node id the request was targeted at.

Source

Thrown at clients/src/main/java/org/apache/kafka/clients/NetworkClientUtils.java:111

    /**
     * Invokes `client.send` followed by 1 or more `client.poll` invocations until a response is received or a
     * disconnection happens (which can happen for a number of reasons including a request timeout).
     *
     * In case of a disconnection, an `IOException` is thrown.
     * If shutdown is initiated on the client during this method, an IOException is thrown.
     *
     * This method is useful for implementing blocking behaviour on top of the non-blocking `NetworkClient`, use it with
     * care.
     */
    public static ClientResponse sendAndReceive(KafkaClient client, ClientRequest request, Time time) throws IOException {
        try {
            client.send(request, time.milliseconds());
            while (client.active()) {
                List<ClientResponse> responses = client.poll(Long.MAX_VALUE, time.milliseconds());
                for (ClientResponse response : responses) {
                    if (response.requestHeader().correlationId() == request.correlationId()) {
                        if (response.wasDisconnected()) {
                            throw new IOException("Connection to " + response.destination() + " was disconnected before the response was read");
                        }
                        if (response.versionMismatch() != null) {
                            throw response.versionMismatch();
                        }
                        return response;
                    }
                }
            }
            throw new IOException("Client was shutdown before response was read");
        } catch (DisconnectException e) {
            if (client.active())
                throw e;
            else
                throw new IOException("Client was shutdown before response was read");

        }
    }

View on GitHub (pinned to c31c9215e1)

Solutions

1. Increase request.timeout.ms and delivery.timeout.ms to exceed the slowest expected broker operation.
2. Retry idempotent operations; ensure the broker is healthy and not restarting.
3. Check for intermediaries (LB, NAT) with idle timeouts shorter than the operation duration and raise them.
4. Inspect broker logs for the disconnection cause (e.g. TooManyRequestsException, OOM, restart).
5. Verify the network path is stable between client and broker hosts.

Example fix

// before
props.put("request.timeout.ms", "5000");
// after
props.put("request.timeout.ms", "30000");
props.put("delivery.timeout.ms", "120000");

// Pre-check readiness and liveness before sendAndReceive.
if (!networkClient.isReady(node, time.milliseconds())) {
    throw new org.apache.kafka.common.errors.DisconnectException(
        "Node " + node + " not ready; refusing to send to avoid a mid-flight disconnect.");
}

// Predicate that captures 'safe to send': ready AND not currently disconnecting.
public static boolean safeToSend(KafkaClient client, Node node, long now) {
    return client.isReady(node, now) && !client.connectionFailed(node);
}

int attempt = 0;
while (attempt < maxAttempts) {
    try {
        return NetworkClientUtils.sendAndReceive(client, request, time);
    } catch (java.io.IOException e) {
        if (e.getMessage() != null && e.getMessage().contains("was disconnected before the response was read")) {
        // In-flight request lost (broker restart, idle disconnect, request timeout).
        // Refresh metadata, re-establish readiness, then retry with a new ClientRequest.
        attempt++;
        continue;
        }
        throw e;
    }
}

When it happens

Trigger: sendAndReceive sends the request and loops on poll(); a matching ClientResponse returns with wasDisconnected()==true. The NetworkClient marks in-flight responses disconnected when the selector reports the channel closed mid-read. Triggers include broker-side close, idle-timeout expiry, network interruption, or a request that exceeded its request.timeout.ms while in flight.

Common situations: Broker restarted or rolled during a long-running request; request.timeout.ms shorter than broker processing time; idle connection reaped by an intermediary LB; network blip / pod migration; broker under backpressure closing sockets; client sending to a node that is being decommissioned.

Related errors

• Unknown host in bootstrap.servers: {url}
• Failed to create new NetworkClient
• NetworkClient is no longer active, state is {state}
• Buffer underflow while parsing response for request with hea
• Connection to {node} failed.

AI-assisted analysis of apache/kafka@c31c9215e1 (2026-08-03). Data as JSON: /data/errors/dc78aa239f424f08.json. Report an issue: GitHub.