{"id":"9681b52e0148b34f","repo":"apache/kafka","slug":"invalid-record-size-expected-to-read-bytes-in","errorCode":null,"errorMessage":"Invalid record size: expected to read {} bytes in record payload, but instead read {}","messagePattern":"Invalid record size: expected to read (.+?) bytes in record payload, but instead read (.+?)","errorType":"exception","errorClass":"InvalidRecordException","httpStatus":null,"severity":"error","filePath":"clients/src/main/java/org/apache/kafka/common/record/internal/DefaultRecord.java","lineNumber":352,"sourceCode":"            // read value\n            int valueSize = ByteUtils.readVarint(buffer);\n            ByteBuffer value = Utils.readBytes(buffer, valueSize);\n\n            int numHeaders = ByteUtils.readVarint(buffer);\n            if (numHeaders < 0)\n                throw new InvalidRecordException(\"Found invalid number of record headers \" + numHeaders);\n            if (numHeaders > buffer.remaining())\n                throw new InvalidRecordException(\"Found invalid number of record headers. \" + numHeaders + \" is larger than the remaining size of the buffer\");\n\n            final Header[] headers;\n            if (numHeaders == 0)\n                headers = Record.EMPTY_HEADERS;\n            else\n                headers = readHeaders(buffer, numHeaders);\n\n            // validate whether we have read all header bytes in the current record\n            if (buffer.position() - recordStart != sizeOfBodyInBytes)\n                throw new InvalidRecordException(\"Invalid record size: expected to read \" + sizeOfBodyInBytes +\n                        \" bytes in record payload, but instead read \" + (buffer.position() - recordStart));\n\n            int totalSizeInBytes = ByteUtils.sizeOfVarint(sizeOfBodyInBytes) + sizeOfBodyInBytes;\n            return new DefaultRecord(totalSizeInBytes, attributes, offset, timestamp, sequence, key, value, headers);\n        } catch (BufferUnderflowException | IllegalArgumentException e) {\n            throw new InvalidRecordException(\"Found invalid record structure\", e);\n        }\n    }\n\n    public static PartialDefaultRecord readPartiallyFrom(InputStream input,\n                                                         long baseOffset,\n                                                         long baseTimestamp,\n                                                         int baseSequence,\n                                                         Long logAppendTime) throws IOException {\n        int sizeOfBodyInBytes = ByteUtils.readVarint(input);\n        int totalSizeInBytes = ByteUtils.sizeOfVarint(sizeOfBodyInBytes) + sizeOfBodyInBytes;\n\n        return readPartiallyFrom(input, totalSizeInBytes, baseOffset, baseTimestamp,","sourceCodeStart":334,"sourceCodeEnd":370,"githubUrl":"https://github.com/apache/kafka/blob/c31c9215e131f8c17e79f8901b48c13ee6aa8e7a/clients/src/main/java/org/apache/kafka/common/record/internal/DefaultRecord.java#L334-L370","documentation":"Thrown after the record is fully parsed when the bytes consumed from recordStart to current position do not equal the declared body size. It is a self-consistency check on the magic-v2 record framing: every field read correctly but the total length disagrees with the size prefix, so the structure is corrupt.","triggerScenarios":"Raised at DefaultRecord.java:351-353 inside readFrom(ByteBuffer, ...) when (buffer.position() - recordStart) != sizeOfBodyInBytes after key, value, and headers were all read. Happens when one of the varint length fields (keySize, valueSize, numHeaders, headerKeySize, headerValueSize) was decoded successfully but consumed a number of bytes that does not sum to the declared body length.","commonSituations":"Custom interceptor or Converter that rewrites a record body without updating the outer size varint, a serialization library that mis-encodes a varint, version skew between a broker using one record-batch framing and a client expecting another, or partial on-disk corruption that left individual fields readable but the total length wrong.","solutions":["Identify the producing client for the failing partition/offset and confirm it goes through the Apache Kafka producer path; remove any interceptors that mutate record bytes.","Round-trip the suspected record through DefaultRecord.writeTo / readFrom in a unit test to find which field's length prefix disagrees with its payload.","Validate the batch CRC with kafka-dump-log; if CRC fails the data is corrupt on disk or in transit, otherwise the producer itself emitted a malformed frame.","Recover the affected log segment from a healthy replica or skip the corrupt record with the consumer's isolation/auto-reset policy if data loss is acceptable."],"exampleFix":null,"handlingStrategy":"try-catch","validationCode":"// You would have to re-walk attributes+timestampDelta+offsetDelta+key+value+\n// headers to predict total bytes. Not practical; rely on the parser's own\n// recordStart/position() check and catch the exception.","typeGuard":null,"tryCatchPattern":"try {\n    DefaultRecord r = DefaultRecord.readFrom(buffer, baseOffset, baseTimestamp, baseSequence, logAppendTime);\n} catch (InvalidRecordException e) {\n    // bytes consumed != sizeOfBodyInBytes declared in the varint length prefix\n    LOG.warn(\"Record length mismatch near offset {}\", baseOffset, e);\n}","preventionTips":["This signals a length-prefix / payload mismatch; ensure the producer and broker agree on the message-format version (v2 since 0.11).","Do not splice records across batches; respect the batch length prefix.","Run integration tests with the same client versions on producer and consumer to avoid format drift."],"tags":["kafka","record-format","deserialization","invalid-record","framing"],"analyzedSha":"c31c9215e131f8c17e79f8901b48c13ee6aa8e7a","analyzedAt":"2026-08-03T12:34:05.770Z","schemaVersion":2}