{"id":"874c0f7d4f1afe01","repo":"apache/kafka","slug":"encountered-corrupt-message-when-fetching-offset","errorCode":null,"errorMessage":"Encountered corrupt message when fetching offset {} for topic-partition {}","messagePattern":"Encountered corrupt message when fetching offset (.+?) for topic-partition (.+?)","errorType":"exception","errorClass":"KafkaException","httpStatus":null,"severity":"error","filePath":"clients/src/main/java/org/apache/kafka/clients/consumer/internals/FetchCollector.java","lineNumber":377,"sourceCode":"                        throw new OffsetOutOfRangeException(errorMessage,\n                                Collections.singletonMap(tp, position.offset));\n                    }\n                }\n            } else {\n                log.debug(\"Unset the preferred read replica {} for partition {} since we got {} when fetching {}\",\n                        clearedReplicaId.get(), tp, error, fetchOffset);\n            }\n        } else if (error == Errors.TOPIC_AUTHORIZATION_FAILED) {\n            //we log the actual partition and not just the topic to help with ACL propagation issues in large clusters\n            log.warn(\"Not authorized to read from partition {}.\", tp);\n            throw new TopicAuthorizationException(Collections.singleton(tp.topic()));\n        } else if (error == Errors.UNKNOWN_LEADER_EPOCH) {\n            log.debug(\"Received unknown leader epoch error in fetch for partition {}\", tp);\n        } else if (error == Errors.UNKNOWN_SERVER_ERROR) {\n            log.warn(\"Unknown server error while fetching offset {} for topic-partition {}\",\n                    fetchOffset, tp);\n        } else if (error == Errors.CORRUPT_MESSAGE) {\n            throw new KafkaException(\"Encountered corrupt message when fetching offset \"\n                    + fetchOffset\n                    + \" for topic-partition \"\n                    + tp);\n        } else {\n            throw new IllegalStateException(\"Unexpected error code \"\n                    + error.code()\n                    + \" while fetching at offset \"\n                    + fetchOffset\n                    + \" from topic-partition \" + tp);\n        }\n    }\n}\n","sourceCodeStart":359,"sourceCodeEnd":390,"githubUrl":"https://github.com/apache/kafka/blob/c31c9215e131f8c17e79f8901b48c13ee6aa8e7a/clients/src/main/java/org/apache/kafka/clients/consumer/internals/FetchCollector.java#L359-L390","documentation":"Thrown as a KafkaException when the broker returns Errors.CORRUPT_MESSAGE for a fetch response in FetchCollector.handleInitializeErrors. It signals that the broker's CRC validation of a record batch at the requested fetch offset failed, so the consumer refuses to hand the malformed data to the application. Unlike transient fetch errors (NOT_LEADER_OR_FOLLOWER, OFFSET_OUT_OF_RANGE) which are handled inline, CORRUPT_MESSAGE is treated as unrecoverable for that position and bubbles up to the caller.","triggerScenarios":"Returned by KafkaConsumer.poll() when the leader's log read at the consumer's fetch offset produces a record batch whose checksum does not match. Occurs specifically in the fetch error path that handles Errors.CORRUPT_MESSAGE after a successful FetchRequest; not raised by producer or admin operations.","commonSituations":"Disk corruption or bit-rot on a broker segment, a partially written segment after a hard broker crash, faulty storage hardware, memory/disk errors during log flush, or a broker version downgrade where record framing differs. Rare under normal operation; often surfaces on one specific partition/offset while others fetch fine.","solutions":["Identify the topic-partition and offset from the exception message, then use kafka-console-consumer or kafka-dump-log on the broker segment to confirm the corrupt batch.","Skip the corrupt record by seeking the consumer past the bad offset with consumer.seek(tp, corruptOffset + 1) and resume polling, or use a producer to write a compensating record.","If corruption is widespread, delete and recreate the affected partition's log segment on the broker (it will rebuild from ISR), or restore from a known-good backup/tiered storage.","Replace or diagnose the underlying disk/hardware on the affected broker to prevent recurrence."],"exampleFix":"// before\nconsumer.subscribe(Collections.singleton(\"orders\"));\nwhile (true) {\n    for (ConsumerRecord<String,String> r : consumer.poll(Duration.ofMillis(500))) {\n        process(r);\n    }\n}\n\n// after - skip past the corrupt offset reported by the exception\ntry {\n    consumer.subscribe(Collections.singleton(\"orders\"));\n    while (true)\n        consumer.poll(Duration.ofMillis(500)).forEach(this::process);\n} catch (org.apache.kafka.common.KafkaException e) {\n    if (e.getMessage().contains(\"corrupt message\")) {\n        TopicPartition tp = parseTp(e);            // from message text\n        long bad = parseOffset(e);\n        consumer.seek(tp, bad + 1);                 // skip the bad batch\n    } else throw e;\n}","handlingStrategy":"try-catch","validationCode":null,"typeGuard":null,"tryCatchPattern":"try {\n    ConsumerRecords<K,V> records = consumer.poll(Duration.ofMillis(timeoutMs));\n} catch (org.apache.kafka.common.KafkaException e) {\n    if (e.getMessage() != null && e.getMessage().contains(\"corrupt message\")) {\n        // option 1: seek past the bad offset on the offending partition\n        consumer.seek(badPartition, badOffset + 1);\n        // option 2: reset to earliest/latest if data is disposable\n        consumer.seek(badPartition, consumer.beginningOffsets(Collections.singleton(badPartition)).get(badPartition));\n    } else {\n        throw e;\n    }\n}","preventionTips":["Run with broker log recovery / kafka-dump-log to identify and truncate the corrupt segment on the broker side.","Keep crc.check.enabled at its default true so the broker rejects bad records before they reach consumers.","Monitor broker disk health and filesystem errors; corrupt messages usually stem from disk faults or unclean shutdowns.","Compacted or compacted+deleted topics can surface older corrupt records on rewind; isolate affected partitions and quarantine them.","If corruption is persistent, recreate the partition and re-produce from a validated source rather than skipping offsets blindly."],"tags":["kafka","consumer","fetch","corruption","crc"],"analyzedSha":"c31c9215e131f8c17e79f8901b48c13ee6aa8e7a","analyzedAt":"2026-08-03T12:34:05.770Z","schemaVersion":2}