{"id":"88f0ee5009cd4abf","repo":"apache/kafka","slug":"invalid-record-size-expected-bytes-in-record-p-88f0ee","errorCode":null,"errorMessage":"Invalid record size: expected {} bytes in record payload, but instead the buffer has only {} remaining bytes.","messagePattern":"Invalid record size: expected (.+?) bytes in record payload, but instead the buffer has only (.+?) remaining bytes\\.","errorType":"exception","errorClass":"InvalidRecordException","httpStatus":null,"severity":"error","filePath":"clients/src/main/java/org/apache/kafka/common/record/internal/DefaultRecord.java","lineNumber":313,"sourceCode":"\n    public static DefaultRecord readFrom(ByteBuffer buffer,\n                                         long baseOffset,\n                                         long baseTimestamp,\n                                         int baseSequence,\n                                         Long logAppendTime) {\n        int sizeOfBodyInBytes = ByteUtils.readVarint(buffer);\n        return readFrom(buffer, sizeOfBodyInBytes, baseOffset, baseTimestamp,\n            baseSequence, logAppendTime);\n    }\n\n    private static DefaultRecord readFrom(ByteBuffer buffer,\n                                          int sizeOfBodyInBytes,\n                                          long baseOffset,\n                                          long baseTimestamp,\n                                          int baseSequence,\n                                          Long logAppendTime) {\n        if (buffer.remaining() < sizeOfBodyInBytes)\n            throw new InvalidRecordException(\"Invalid record size: expected \" + sizeOfBodyInBytes +\n                \" bytes in record payload, but instead the buffer has only \" + buffer.remaining() +\n                \" remaining bytes.\");\n        try {\n            int recordStart = buffer.position();\n            byte attributes = buffer.get();\n            long timestampDelta = ByteUtils.readVarlong(buffer);\n            long timestamp = baseTimestamp + timestampDelta;\n            if (logAppendTime != null)\n                timestamp = logAppendTime;\n\n            int offsetDelta = ByteUtils.readVarint(buffer);\n            long offset = baseOffset + offsetDelta;\n            int sequence = baseSequence >= 0 ?\n                    DefaultRecordBatch.incrementSequence(baseSequence, offsetDelta) :\n                    RecordBatch.NO_SEQUENCE;\n\n            // read key\n            int keySize = ByteUtils.readVarint(buffer);","sourceCodeStart":295,"sourceCodeEnd":331,"githubUrl":"https://github.com/apache/kafka/blob/c31c9215e131f8c17e79f8901b48c13ee6aa8e7a/clients/src/main/java/org/apache/kafka/common/record/internal/DefaultRecord.java#L295-L331","documentation":"Thrown by DefaultRecord.readFrom(ByteBuffer, ...) when the varint-prefixed body length declares more bytes than the buffer actually has remaining. It is an InvalidRecordException surfaced during magic-v2 record decoding: the on-wire size prefix is inconsistent with the bytes the caller handed to the parser. The library throws it to refuse to manufacture a record out of a truncated or rewritten batch.","triggerScenarios":"Raised at DefaultRecord.java:312-315 when the per-record body size (read via ByteUtils.readVarint at line 301) exceeds buffer.remaining() at the moment of the second private readFrom entry. Hit whenever a caller slices a ByteBuffer too short (e.g. DefaultRecordBatch iterating past its declared batch length), or when an upstream caller hands in a buffer whose limit was manually trimmed.","commonSituations":"Corruption of an in-memory batch after a partial network read, a custom serializer that emits the wrong size prefix, a producer/consumer client version mismatch where the wire format changed (pre-magic-v2 vs v2), or a log segment that was truncated mid-record after a hard broker crash. Also seen with interceptors/clamps that rewrite batches without recomputing the size varint.","solutions":["Inspect the caller that produced the ByteBuffer (e.g. DefaultRecordBatch, MemoryRecords, or a custom Decoder) and confirm the slice handed to DefaultRecord.readFrom spans the full declared batch length.","Verify the broker and client are on compatible Kafka versions and both use record-batch magic v2 (introduced in 0.11.0); downgrade or upgrade together.","If reading from a log segment, run kafka-dump-log on the segment to confirm the size prefix matches the payload, and restore from a replica if the segment is truncated.","Reproduce with a unit test using DefaultRecord.writeTo on the same key/value/headers and diff the produced bytes against the input to locate the size mismatch."],"exampleFix":"// before: caller slices a buffer shorter than the declared body\nint bodySize = ByteUtils.readVarint(buf);\nDefaultRecord r = DefaultRecord.readFrom(buf, baseOffset, baseTimestamp, baseSequence, lat);\n\n// after: slice exactly bodySize bytes for the inner record, validate first\nint bodySize = ByteUtils.readVarint(buf);\nif (buf.remaining() < bodySize) {\n    throw new InvalidRecordException(\"Truncated batch: need \" + bodySize + \", have \" + buf.remaining());\n}\nDefaultRecord r = DefaultRecord.readFrom(buf, baseOffset, baseTimestamp, baseSequence, lat);","handlingStrategy":"try-catch","validationCode":"// If you hold the ByteBuffer and the declared body size yourself:\nint sizeOfBodyInBytes = ByteUtils.readVarint(buffer);\nif (buffer.remaining() < sizeOfBodyInBytes) {\n    // truncated frame: do not call DefaultRecord.readFrom(...); log and skip\n    return;\n}","typeGuard":"// Narrow to a readable record slice before delegating.\nprivate static boolean isReadableRecordSlice(ByteBuffer b, int sizeOfBodyInBytes) {\n    return b != null && b.remaining() >= sizeOfBodyInBytes && sizeOfBodyInBytes >= 0;\n}","tryCatchPattern":"try {\n    DefaultRecord r = DefaultRecord.readFrom(buffer, baseOffset, baseTimestamp, baseSequence, logAppendTime);\n} catch (InvalidRecordException e) {\n    // corrupt/truncated record payload; advance past it or skip the batch\n    LOG.warn(\"Skipping malformed record at offset {}\", baseOffset, e);\n}","preventionTips":["Never hand a ByteBuffer to readFrom that has fewer bytes than the varint-declared body size.","When reading from a stream, read exactly sizeOfBodyInBytes bytes (Utils.readFully) and verify the returned count before flipping the buffer.","Treat any buffer returned by the broker as untrusted; validate frame length at the batch level (DefaultRecordBatch) before descending into records.","If you broker records yourself, enable CRC32C checks on the batch so truncation is caught upstream."],"tags":["kafka","record-format","deserialization","invalid-record","bytebuffer"],"analyzedSha":"c31c9215e131f8c17e79f8901b48c13ee6aa8e7a","analyzedAt":"2026-08-03T12:34:05.770Z","schemaVersion":2}