{"id":"28fa3ace0f6cfc86","repo":"apache/kafka","slug":"found-invalid-record-structure","errorCode":null,"errorMessage":"Found invalid record structure","messagePattern":"Found invalid record structure","errorType":"exception","errorClass":"InvalidRecordException","httpStatus":null,"severity":"error","filePath":"clients/src/main/java/org/apache/kafka/common/record/internal/DefaultRecord.java","lineNumber":358,"sourceCode":"                throw new InvalidRecordException(\"Found invalid number of record headers \" + numHeaders);\n            if (numHeaders > buffer.remaining())\n                throw new InvalidRecordException(\"Found invalid number of record headers. \" + numHeaders + \" is larger than the remaining size of the buffer\");\n\n            final Header[] headers;\n            if (numHeaders == 0)\n                headers = Record.EMPTY_HEADERS;\n            else\n                headers = readHeaders(buffer, numHeaders);\n\n            // validate whether we have read all header bytes in the current record\n            if (buffer.position() - recordStart != sizeOfBodyInBytes)\n                throw new InvalidRecordException(\"Invalid record size: expected to read \" + sizeOfBodyInBytes +\n                        \" bytes in record payload, but instead read \" + (buffer.position() - recordStart));\n\n            int totalSizeInBytes = ByteUtils.sizeOfVarint(sizeOfBodyInBytes) + sizeOfBodyInBytes;\n            return new DefaultRecord(totalSizeInBytes, attributes, offset, timestamp, sequence, key, value, headers);\n        } catch (BufferUnderflowException | IllegalArgumentException e) {\n            throw new InvalidRecordException(\"Found invalid record structure\", e);\n        }\n    }\n\n    public static PartialDefaultRecord readPartiallyFrom(InputStream input,\n                                                         long baseOffset,\n                                                         long baseTimestamp,\n                                                         int baseSequence,\n                                                         Long logAppendTime) throws IOException {\n        int sizeOfBodyInBytes = ByteUtils.readVarint(input);\n        int totalSizeInBytes = ByteUtils.sizeOfVarint(sizeOfBodyInBytes) + sizeOfBodyInBytes;\n\n        return readPartiallyFrom(input, totalSizeInBytes, baseOffset, baseTimestamp,\n            baseSequence, logAppendTime);\n    }\n\n    private static PartialDefaultRecord readPartiallyFrom(InputStream input,\n                                                          int sizeInBytes,\n                                                          long baseOffset,","sourceCodeStart":340,"sourceCodeEnd":376,"githubUrl":"https://github.com/apache/kafka/blob/c31c9215e131f8c17e79f8901b48c13ee6aa8e7a/clients/src/main/java/org/apache/kafka/common/record/internal/DefaultRecord.java#L340-L376","documentation":"A catch-all InvalidRecordException wrapping a BufferUnderflowException or IllegalArgumentException raised while reading the per-record fields from a ByteBuffer. It signals that the parser ran off the end of the buffer or hit an invalid varint while decoding attributes, timestampDelta, offsetDelta, key, value, or headers, but no specific guard caught it first.","triggerScenarios":"Raised at DefaultRecord.java:357-359 (the try/catch around the body of the private readFrom(ByteBuffer, ...)). Triggered by buffer underflow when Utils.readBytes asks for more key/value bytes than remain, by an illegal-argument from ByteUtils when a varint overflows, or by any other RuntimeException from the field-decoding helpers inside the try block.","commonSituations":"Truncated fetch response (network cut, broker returned a partial batch), disk/page-cache corruption, an off-spec producer, or a custom consumer that re-slices MemoryRecords incorrectly before iterating. Frequently co-occurs with checksum failures in the broker log.","solutions":["Check the broker and client logs for preceding CRC or fetch-size errors on the same partition; raise fetch.message.maxBytes if batches are being truncated in flight.","Confirm the buffer handed to DefaultRecord.readFrom is the one returned by MemoryRecords / DefaultRecordBatch and has not been re-sliced by application code.","Run kafka-dump-log on the source segment; if it fails the same way the record is corrupt on disk and must be recovered from a replica.","Verify client and broker are on mutually supported versions and both use magic-v2 batches (Apache Kafka 0.11+)."],"exampleFix":null,"handlingStrategy":"try-catch","validationCode":"// Generic corruption (BufferUnderflowException / IllegalArgumentException\n// while parsing). No single pre-check; validate the whole slice is well-formed\n// by attempting a strict parse on a duplicated, slice()ed buffer first.","typeGuard":"// Cheap structural sanity check before delegating.\nprivate static boolean looksLikeValidRecord(ByteBuffer b, int sizeOfBodyInBytes) {\n    return b != null && b.remaining() >= sizeOfBodyInBytes && sizeOfBodyInBytes > 0;\n}","tryCatchPattern":"try {\n    DefaultRecord r = DefaultRecord.readFrom(buffer, baseOffset, baseTimestamp, baseSequence, logAppendTime);\n} catch (InvalidRecordException e) {\n    // wraps BufferUnderflowException or IllegalArgumentException\n    LOG.warn(\"Corrupt record structure near offset {}\", baseOffset, e);\n}","preventionTips":["Wrap every batch-deserialization loop in a single try-catch for InvalidRecordException so a single malformed record cannot kill the consumer.","Prefer consuming via KafkaConsumer.poll(), which isolates corruption as SerializationException, over calling DefaultRecord directly.","If you read log files offline (DumpLogSegments), pass --max-message-size and verify file integrity (CRC) first."],"tags":["kafka","record-format","deserialization","invalid-record","buffer-underflow"],"analyzedSha":"c31c9215e131f8c17e79f8901b48c13ee6aa8e7a","analyzedAt":"2026-08-03T12:34:05.770Z","schemaVersion":2}