{"id":"9240a84adafb8dc8","repo":"apache/kafka","slug":"found-invalid-number-of-record-headers-is-larg","errorCode":null,"errorMessage":"Found invalid number of record headers. {} is larger than the remaining size of the buffer","messagePattern":"Found invalid number of record headers\\. (.+?) is larger than the remaining size of the buffer","errorType":"exception","errorClass":"InvalidRecordException","httpStatus":null,"severity":"error","filePath":"clients/src/main/java/org/apache/kafka/common/record/internal/DefaultRecord.java","lineNumber":342,"sourceCode":"            int offsetDelta = ByteUtils.readVarint(buffer);\n            long offset = baseOffset + offsetDelta;\n            int sequence = baseSequence >= 0 ?\n                    DefaultRecordBatch.incrementSequence(baseSequence, offsetDelta) :\n                    RecordBatch.NO_SEQUENCE;\n\n            // read key\n            int keySize = ByteUtils.readVarint(buffer);\n            ByteBuffer key = Utils.readBytes(buffer, keySize);\n\n            // read value\n            int valueSize = ByteUtils.readVarint(buffer);\n            ByteBuffer value = Utils.readBytes(buffer, valueSize);\n\n            int numHeaders = ByteUtils.readVarint(buffer);\n            if (numHeaders < 0)\n                throw new InvalidRecordException(\"Found invalid number of record headers \" + numHeaders);\n            if (numHeaders > buffer.remaining())\n                throw new InvalidRecordException(\"Found invalid number of record headers. \" + numHeaders + \" is larger than the remaining size of the buffer\");\n\n            final Header[] headers;\n            if (numHeaders == 0)\n                headers = Record.EMPTY_HEADERS;\n            else\n                headers = readHeaders(buffer, numHeaders);\n\n            // validate whether we have read all header bytes in the current record\n            if (buffer.position() - recordStart != sizeOfBodyInBytes)\n                throw new InvalidRecordException(\"Invalid record size: expected to read \" + sizeOfBodyInBytes +\n                        \" bytes in record payload, but instead read \" + (buffer.position() - recordStart));\n\n            int totalSizeInBytes = ByteUtils.sizeOfVarint(sizeOfBodyInBytes) + sizeOfBodyInBytes;\n            return new DefaultRecord(totalSizeInBytes, attributes, offset, timestamp, sequence, key, value, headers);\n        } catch (BufferUnderflowException | IllegalArgumentException e) {\n            throw new InvalidRecordException(\"Found invalid record structure\", e);\n        }\n    }","sourceCodeStart":324,"sourceCodeEnd":360,"githubUrl":"https://github.com/apache/kafka/blob/c31c9215e131f8c17e79f8901b48c13ee6aa8e7a/clients/src/main/java/org/apache/kafka/common/record/internal/DefaultRecord.java#L324-L360","documentation":"Thrown on the ByteBuffer path when numHeaders is non-negative but greater than buffer.remaining(). Since each header needs at least one byte, the count cannot fit in the remaining payload; the library treats this as structural corruption rather than trying to read past the buffer limit.","triggerScenarios":"Raised at DefaultRecord.java:341-342 when ByteUtils.readVarint returns a header count larger than buffer.remaining() after the key/value sections have already been consumed. Common when the headers count field was overwritten with a large value or when the key/value length prefixes were misread, leaving too few bytes for the claimed header array.","commonSituations":"Producer client producing records where key/value lengths disagree with the actual bytes (off-by-one in a custom partitioner/serializer), corruption of the batch in the page cache or on disk, or a consumer pointed at a topic that another vendor's broker wrote in a subtly different record format.","solutions":["Confirm the producer is on a supported client version and uses the standard record builder; capture a single failing record with kafka-console-consumer --max-messages 1 and hex-inspect the header section.","Check the broker log for CRC validation failures on the same partition/offset to distinguish in-flight corruption from on-disk corruption.","If only one partition is affected, stop the affected producer, validate its header construction, and re-publish the bad batch.","Recover the segment from an in-sync replica if disk corruption is confirmed by kafka-dump-log."],"exampleFix":null,"handlingStrategy":"validation","validationCode":"// After you decode numHeaders, before letting the parser continue:\nif (numHeaders > buffer.remaining()) {\n    // declared header count cannot fit; record is corrupt or truncated\n    throw new InvalidRecordException(\"declared headers exceed remaining buffer\");\n}","typeGuard":"// Bounds-check the header count against the remaining payload.\nprivate static boolean headersFitInBuffer(int numHeaders, ByteBuffer buffer) {\n    return numHeaders >= 0 && numHeaders <= buffer.remaining();\n}","tryCatchPattern":"try {\n    DefaultRecord r = DefaultRecord.readFrom(buffer, baseOffset, baseTimestamp, baseSequence, logAppendTime);\n} catch (InvalidRecordException e) {\n    LOG.warn(\"Header count exceeds remaining bytes near offset {}\", baseOffset, e);\n}","preventionTips":["Remember numHeaders is an upper bound only if each header is >=1 byte; always also trust the parser's later per-header size checks.","When authoring records, cap header count and total header bytes well below the record size limit.","Catch InvalidRecordException at the batch iterator level so one bad record does not abort the whole poll."],"tags":["kafka","record-format","deserialization","headers","invalid-record"],"analyzedSha":"c31c9215e131f8c17e79f8901b48c13ee6aa8e7a","analyzedAt":"2026-08-03T12:34:05.770Z","schemaVersion":2}