{"id":"dc78aa239f424f08","repo":"apache/kafka","slug":"connection-to-destination-was-disconnected-befor","errorCode":null,"errorMessage":"Connection to {destination} was disconnected before the response was read","messagePattern":"Connection to (.+?) was disconnected before the response was read","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"clients/src/main/java/org/apache/kafka/clients/NetworkClientUtils.java","lineNumber":111,"sourceCode":"    /**\n     * Invokes `client.send` followed by 1 or more `client.poll` invocations until a response is received or a\n     * disconnection happens (which can happen for a number of reasons including a request timeout).\n     *\n     * In case of a disconnection, an `IOException` is thrown.\n     * If shutdown is initiated on the client during this method, an IOException is thrown.\n     *\n     * This method is useful for implementing blocking behaviour on top of the non-blocking `NetworkClient`, use it with\n     * care.\n     */\n    public static ClientResponse sendAndReceive(KafkaClient client, ClientRequest request, Time time) throws IOException {\n        try {\n            client.send(request, time.milliseconds());\n            while (client.active()) {\n                List<ClientResponse> responses = client.poll(Long.MAX_VALUE, time.milliseconds());\n                for (ClientResponse response : responses) {\n                    if (response.requestHeader().correlationId() == request.correlationId()) {\n                        if (response.wasDisconnected()) {\n                            throw new IOException(\"Connection to \" + response.destination() + \" was disconnected before the response was read\");\n                        }\n                        if (response.versionMismatch() != null) {\n                            throw response.versionMismatch();\n                        }\n                        return response;\n                    }\n                }\n            }\n            throw new IOException(\"Client was shutdown before response was read\");\n        } catch (DisconnectException e) {\n            if (client.active())\n                throw e;\n            else\n                throw new IOException(\"Client was shutdown before response was read\");\n\n        }\n    }\n","sourceCodeStart":93,"sourceCodeEnd":129,"githubUrl":"https://github.com/apache/kafka/blob/c31c9215e131f8c17e79f8901b48c13ee6aa8e7a/clients/src/main/java/org/apache/kafka/clients/NetworkClientUtils.java#L93-L129","documentation":"Thrown as IOException by NetworkClientUtils.sendAndReceive when a ClientResponse matching the request's correlation id is found but its wasDisconnected() flag is true. It means the request was sent successfully, then the connection to the destination node was lost before the full response was read back. The destination in the message is the node id the request was targeted at.","triggerScenarios":"sendAndReceive sends the request and loops on poll(); a matching ClientResponse returns with wasDisconnected()==true. The NetworkClient marks in-flight responses disconnected when the selector reports the channel closed mid-read. Triggers include broker-side close, idle-timeout expiry, network interruption, or a request that exceeded its request.timeout.ms while in flight.","commonSituations":"Broker restarted or rolled during a long-running request; request.timeout.ms shorter than broker processing time; idle connection reaped by an intermediary LB; network blip / pod migration; broker under backpressure closing sockets; client sending to a node that is being decommissioned.","solutions":["Increase request.timeout.ms and delivery.timeout.ms to exceed the slowest expected broker operation.","Retry idempotent operations; ensure the broker is healthy and not restarting.","Check for intermediaries (LB, NAT) with idle timeouts shorter than the operation duration and raise them.","Inspect broker logs for the disconnection cause (e.g. TooManyRequestsException, OOM, restart).","Verify the network path is stable between client and broker hosts."],"exampleFix":"// before\nprops.put(\"request.timeout.ms\", \"5000\");\n// after\nprops.put(\"request.timeout.ms\", \"30000\");\nprops.put(\"delivery.timeout.ms\", \"120000\");","handlingStrategy":"retry","validationCode":"// Pre-check readiness and liveness before sendAndReceive.\nif (!networkClient.isReady(node, time.milliseconds())) {\n    throw new org.apache.kafka.common.errors.DisconnectException(\n        \"Node \" + node + \" not ready; refusing to send to avoid a mid-flight disconnect.\");\n}","typeGuard":"// Predicate that captures 'safe to send': ready AND not currently disconnecting.\npublic static boolean safeToSend(KafkaClient client, Node node, long now) {\n    return client.isReady(node, now) && !client.connectionFailed(node);\n}","tryCatchPattern":"int attempt = 0;\nwhile (attempt < maxAttempts) {\n    try {\n        return NetworkClientUtils.sendAndReceive(client, request, time);\n    } catch (java.io.IOException e) {\n        if (e.getMessage() != null && e.getMessage().contains(\"was disconnected before the response was read\")) {\n        // In-flight request lost (broker restart, idle disconnect, request timeout).\n        // Refresh metadata, re-establish readiness, then retry with a new ClientRequest.\n        attempt++;\n        continue;\n        }\n        throw e;\n    }\n}","preventionTips":["Set request.timeout.ms and connections.max.idle.ms so disconnects are bounded and predictable.","Make requests idempotent on the server side (idempotent producer, transactions) so retry-after-disconnect is safe.","Refresh metadata after a disconnect in case the broker went away (partition leadership moved).","Watch for DisconnectException specifically; it is the upstream signal this IOException wraps."],"tags":["network","disconnect","request-timeout","retriable"],"analyzedSha":"c31c9215e131f8c17e79f8901b48c13ee6aa8e7a","analyzedAt":"2026-08-03T12:34:05.770Z","schemaVersion":2}