{"id":"967f7bcc28fe69ae","repo":"apache/kafka","slug":"failed-to-make-progress-reading-messages-at","errorCode":null,"errorMessage":"Failed to make progress reading messages at {}={}. Received a non-empty fetch response from the server, but no complete records were found.","messagePattern":"Failed to make progress reading messages at (.+?)=(.+?)\\. Received a non-empty fetch response from the server, but no complete records were found\\.","errorType":"exception","errorClass":"KafkaException","httpStatus":null,"severity":"error","filePath":"clients/src/main/java/org/apache/kafka/clients/consumer/internals/FetchCollector.java","lineNumber":272,"sourceCode":"        final long fetchOffset = completedFetch.nextFetchOffset();\n\n        // we are interested in this fetch only if the beginning offset matches the\n        // current consumed position\n        SubscriptionState.FetchPosition position = subscriptions.positionOrNull(tp);\n        if (position == null || position.offset != fetchOffset) {\n            log.debug(\"Discarding stale fetch response for partition {} since its offset {} does not match \" +\n                \"the expected offset {} or the partition has been unassigned\", tp, fetchOffset, position);\n            return null;\n        }\n\n        final FetchResponseData.PartitionData partition = completedFetch.partitionData;\n        log.trace(\"Preparing to read {} bytes of data for partition {} with offset {}\",\n                FetchResponse.recordsSize(partition), tp, position);\n        Iterator<? extends RecordBatch> batches = FetchResponse.recordsOrFail(partition).batches().iterator();\n\n        if (!batches.hasNext() && FetchResponse.recordsSize(partition) > 0) {\n            // This should not happen with brokers that support FetchRequest/Response V4 or higher (i.e. KIP-74)\n            throw new KafkaException(\"Failed to make progress reading messages at \" + tp + \"=\" +\n                    fetchOffset + \". Received a non-empty fetch response from the server, but no \" +\n                    \"complete records were found.\");\n        }\n\n        if (!updatePartitionState(partition, tp)) {\n            return null;\n        }\n\n        completedFetch.setInitialized();\n        return completedFetch;\n    }\n\n    private boolean updatePartitionState(final FetchResponseData.PartitionData partitionData,\n                                         final TopicPartition tp) {\n        if (partitionData.highWatermark() >= 0) {\n            log.trace(\"Updating high watermark for partition {} to {}\", tp, partitionData.highWatermark());\n            if (!subscriptions.tryUpdatingHighWatermark(tp, partitionData.highWatermark())) {\n                return false;","sourceCodeStart":254,"sourceCodeEnd":290,"githubUrl":"https://github.com/apache/kafka/blob/c31c9215e131f8c17e79f8901b48c13ee6aa8e7a/clients/src/main/java/org/apache/kafka/clients/consumer/internals/FetchCollector.java#L254-L290","documentation":"KafkaException raised in FetchCollector.handleInitializeSuccess when the fetch response carries a non-empty records payload (bytes > 0) but iterating the record batches yields zero batches. Per KIP-74 this is impossible for brokers that support FetchRequest/Response v4+, so it indicates a malformed response, a broker bug, or an intermediary (proxy/serde) that corrupted the bytes. The client refuses to silently lose data, so it throws rather than advance the offset.","triggerScenarios":"Triggered at FetchCollector.java:272 when FetchResponse.recordsSize(partition) > 0 but FetchResponse.recordsOrFail(partition).batches().iterator() has no next element.","commonSituations":"Talking to a down-level or buggy broker (< 0.10.0 / FetchResponse v3 or earlier); a misbehaving L7 proxy or sidecar that rewrites the fetch response; on-the-wire corruption from a faulty NIC / TLS terminator; broker version mismatch in a heterogeneous cluster; old broker mixed with new client during an upgrade.","solutions":["Ensure all brokers are at version 0.10.0 or higher (FetchResponse v4+); decommission legacy brokers.","Remove or fix any HTTP/TCP proxy, sidecar, or capture tool between the client and brokers that may alter the response body.","Upgrade kafka-clients to the latest release to benefit from stricter response validation and fixes.","Capture a network trace (or enable TRACE logging of org.apache.kafka.clients.FetchCollector) to confirm the response bytes are intact end-to-end."],"exampleFix":"// before: client points at mixed cluster with one 0.9 broker\nbootstrap.servers=broker-v9:9092,broker-v3:9092\n\n// after: all brokers >= 0.10 and client aligned to cluster version\nbootstrap.servers=broker-v3:9092,broker-v3b:9092\n# (decommission the 0.9 node)","handlingStrategy":"retry","validationCode":"// This is a broker-version compatibility issue (KIP-74): the server returned bytes\n// but no complete RecordBatch. You cannot validate it client-side. Ensure broker and\n// client versions are aligned to avoid triggering it:\nProperties p = new Properties();\np.put(ProducerConfig.BROKER_VERSION_COMPATIBILITY_ENABLE, \"true\"); // producer side\n// On the consumer side, verify the broker reports a FetchRequest V4+ capable version:\nNode node = consumer.partitionsFor(\"myTopic\").get(0).leader();\n// If node is null or the cluster is mid-upgrade, defer heavy consumption until stable.","typeGuard":null,"tryCatchPattern":"int attempts = 0;\nwhile (attempts++ < 3) {\n    try {\n        return consumer.poll(Duration.ofMillis(500));\n    } catch (org.apache.kafka.common.KafkaException e) {\n        if (e.getMessage() != null && e.getMessage().startsWith(\"Failed to make progress reading messages\")) {\n            // FetchCollector.java:272: non-empty response, zero complete batches.\n            // Almost always a broker < FetchRequest V4 or a mid-upgrade/transient corruption.\n            log.warn(\"Broker returned unparseable records for partition; retrying after metadata refresh (attempt {})\", attempts);\n            consumer.requestOffsetReset(consumer.assignment()); // optional: force re-fetch from a clean offset\n            continue;\n        }\n        throw e;\n    }\n}\nthrow new IllegalStateException(\"Could not make fetch progress after retries\");","preventionTips":["Keep broker and client versions aligned; the condition that raises this is essentially impossible with brokers supporting FetchRequest V4+.","During a rolling upgrade, expect transient occurrences — retry with bounded backoff rather than aborting.","Do not catch this as a generic Exception; isolate the 'Failed to make progress reading messages' message and retry, surface anything else.","If it persists on a stable cluster, suspect on-disk corruption or an incompatible interceptor/serializer — capture the partition/offset from the message."],"tags":["consumer","broker-mismatch","protocol","data-corruption"],"analyzedSha":"c31c9215e131f8c17e79f8901b48c13ee6aa8e7a","analyzedAt":"2026-08-03T12:34:05.770Z","schemaVersion":2}