{"id":"6ceafa54eb6fcfd3","repo":"apache/kafka","slug":"record-size-exceeds-the-largest-allowable-message","errorCode":null,"errorMessage":"Record size exceeds the largest allowable message size (%d).","messagePattern":"Record size exceeds the largest allowable message size \\((.+?)\\)\\.","errorType":"exception","errorClass":"CorruptRecordException","httpStatus":null,"severity":"error","filePath":"clients/src/main/java/org/apache/kafka/common/record/internal/AbstractLegacyRecordBatch.java","lineNumber":303,"sourceCode":"\n        DataLogInputStream(InputStream stream, int maxMessageSize) {\n            this.stream = stream;\n            this.maxMessageSize = maxMessageSize;\n            this.offsetAndSizeBuffer = ByteBuffer.allocate(Records.LOG_OVERHEAD);\n        }\n\n        public AbstractLegacyRecordBatch nextBatch() throws IOException {\n            offsetAndSizeBuffer.clear();\n            Utils.readFully(stream, offsetAndSizeBuffer);\n            if (offsetAndSizeBuffer.hasRemaining())\n                return null;\n\n            long offset = offsetAndSizeBuffer.getLong(Records.OFFSET_OFFSET);\n            int size = offsetAndSizeBuffer.getInt(Records.SIZE_OFFSET);\n            if (size < LegacyRecord.RECORD_OVERHEAD_V0)\n                throw new CorruptRecordException(String.format(\"Record size is less than the minimum record overhead (%d)\", LegacyRecord.RECORD_OVERHEAD_V0));\n            if (size > maxMessageSize)\n                throw new CorruptRecordException(String.format(\"Record size exceeds the largest allowable message size (%d).\", maxMessageSize));\n\n            ByteBuffer batchBuffer = ByteBuffer.allocate(size);\n            Utils.readFully(stream, batchBuffer);\n            if (batchBuffer.hasRemaining())\n                return null;\n            batchBuffer.flip();\n\n            return new BasicLegacyRecordBatch(offset, new LegacyRecord(batchBuffer));\n        }\n    }\n\n    private static class DeepRecordsIterator extends AbstractIterator<Record> implements CloseableIterator<Record> {\n        private final ArrayDeque<AbstractLegacyRecordBatch> innerEntries;\n        private final long absoluteBaseOffset;\n        private final byte wrapperMagic;\n\n        private DeepRecordsIterator(AbstractLegacyRecordBatch wrapperEntry,\n                                    boolean ensureMatchingMagic,","sourceCodeStart":285,"sourceCodeEnd":321,"githubUrl":"https://github.com/apache/kafka/blob/c31c9215e131f8c17e79f8901b48c13ee6aa8e7a/clients/src/main/java/org/apache/kafka/common/record/internal/AbstractLegacyRecordBatch.java#L285-L321","documentation":"Thrown by DataLogInputStream.nextBatch() (line 303) as a CorruptRecordException when the size field of a legacy record frame exceeds the maxMessageSize cap passed to the stream. This guards against unbounded allocations and DoS when materializing a record from an input stream. The limit is the configured maximum record/message size for the reading context.","triggerScenarios":"A producer sent a message larger than the reader's max.message.size / message.max.bytes / max.message.bytes, and the reader hits it while iterating compressed inner records (DeepRecordsIterator uses Integer.MAX_VALUE, but a direct DataLogInputStream over a fetch response is bounded). Also when broker message.max.bytes is larger than what the consumer or client is willing to buffer.","commonSituations":"Producer compression.disabled sending a single large payload, or a compressed wrapper that decompresses to something whose declared size is huge. Broker config message.max.bytes raised without raising replica.fetch.max.bytes / fetch.max.bytes on consumers. Cross-cluster mirroring where the target has a smaller size cap than source.","solutions":["Raise the reader-side limit (fetch.max.bytes / max.partition.fetch.bytes for consumers, message.max.bytes for brokers) to at least the producer's max.request.size.","Reduce the payload size on the producer, or enable compression so the on-wire frame is smaller.","Keep producer max.request.size, broker message.max.bytes, and consumer fetch.max.bytes in a consistent hierarchy (consumer >= broker >= producer).","Verify no custom serializer is emitting unexpectedly large values (e.g. unbounded collections)."],"exampleFix":"// before\nprops.put(ConsumerConfig.MAX_PARTITION_FETCH_BYTES_CONFIG, 1024 * 1024);\n\n// after\nprops.put(ConsumerConfig.MAX_PARTITION_FETCH_BYTES_CONFIG, 10 * 1024 * 1024);\nprops.put(ConsumerConfig.FETCH_MAX_BYTES_CONFIG, 50 * 1024 * 1024);","handlingStrategy":"try-catch","validationCode":null,"typeGuard":null,"tryCatchPattern":"// Thrown when a legacy record's declared size exceeds maxMessageSize used to\n// construct the iterator (e.g. MemoryRecords.Builder or LogInputStream).\nimport org.apache.kafka.common.errors.CorruptRecordException;\n\ntry {\n    for (Record r : records.records()) { /* process */ }\n} catch (CorruptRecordException e) {\n    // declared size > maxMessageSize: either a genuinely oversized/corrupt record\n    // or the limit you passed is too small. Log the configured limit for diagnosis.\n    log.error(\"Legacy record exceeds maxMessageSize={}; possible corruption or misconfigured limit\", maxMessageSize, e);\n}","preventionTips":["Configure max.message.bytes on the broker and max.request.size / message.max.bytes on the client consistently so producers cannot send records that exceed the size limit used to read them.","Keep producers on the v2 message format; the legacy-size-check path only runs for v0/v1 batches.","When reading legacy data, size maxMessageSize to at least the broker's absolute max to avoid false positives from legitimately large (but valid) records.","Treat this as data corruption first (skip/advance) and only revisit the limit after confirming the bytes are well-formed.","Log maxMessageSize alongside the failure so you can distinguish 'corrupt record' from 'limit too small'."],"tags":["message-size","config","consumer","limits","legacy-record"],"analyzedSha":"c31c9215e131f8c17e79f8901b48c13ee6aa8e7a","analyzedAt":"2026-08-03T12:34:05.770Z","schemaVersion":2}