{"id":"d0a4b7c917345940","repo":"apache/kafka","slug":"error-reading-array-of-size-size-only-remain-d0a4b7","errorCode":null,"errorMessage":"Error reading array of size ${size}, only ${remaining} bytes available","messagePattern":"Error reading array of size (.+?), only (.+?) bytes available","errorType":"exception","errorClass":"SchemaException","httpStatus":null,"severity":"error","filePath":"clients/src/main/java/org/apache/kafka/common/protocol/types/CompactArrayOf.java","lineNumber":83,"sourceCode":"        ByteUtils.writeUnsignedVarint(size + 1, buffer);\n\n        for (Object obj : objs)\n            type.write(buffer, obj);\n    }\n\n    @Override\n    public Object read(ByteBuffer buffer) {\n        int n = ByteUtils.readUnsignedVarint(buffer);\n        if (n == 0) {\n            if (isNullable()) {\n                return null;\n            } else {\n                throw new SchemaException(\"This array is not nullable.\");\n            }\n        }\n        int size = n - 1;\n        if (size > buffer.remaining())\n            throw new SchemaException(\"Error reading array of size \" + size + \", only \" + buffer.remaining() + \" bytes available\");\n        Object[] objs = new Object[size];\n        for (int i = 0; i < size; i++)\n            objs[i] = type.read(buffer);\n        return objs;\n    }\n\n    @Override\n    public int sizeOf(Object o) {\n        if (o == null) {\n            return 1;\n        }\n        Object[] objs = (Object[]) o;\n        int size = ByteUtils.sizeOfUnsignedVarint(objs.length + 1);\n        for (Object obj : objs) {\n            size += type.sizeOf(obj);\n        }\n        return size;\n    }","sourceCodeStart":65,"sourceCodeEnd":101,"githubUrl":"https://github.com/apache/kafka/blob/c31c9215e131f8c17e79f8901b48c13ee6aa8e7a/clients/src/main/java/org/apache/kafka/common/protocol/types/CompactArrayOf.java#L65-L101","documentation":"Thrown by CompactArrayOf.read after computing `size = n - 1` when size exceeds buffer.remaining(). Same underflow guard as ArrayOf but applied to varint-length compact arrays introduced with flexible message versions. It blocks allocating an array larger than the available payload and aborts the read.","triggerScenarios":"CompactArrayOf.read decodes varint N, sets size = N - 1, and checks `size > buffer.remaining()`. Produced by truncated flexible-version payloads, a wrong API version schema (so the varint is actually part of another field), or a sliced buffer that does not cover the full struct.","commonSituations":"Reading a v3+ (flexible) request/response body with a v2 (non-flexible) schema or vice versa; ByteBuffer slice mis-sized in a custom network codec; log replay after a partial write or segment corruption.","solutions":["Verify the request/response API version on the wire matches the schema (and hence the compact-array encoding) used to decode it.","Ensure the ByteBuffer limit spans the entire serialized struct, not a sub-slice.","For recorded/test payloads, confirm they were produced with the same flexible-version setting; regenerate fixtures from current schemas.","Run kafka-dump-log or hex-inspect the region to rule out on-disk corruption."],"exampleFix":"// before - decoding a flexible body with a non-flexible schema\nStruct s = (Struct) LEGACY_SCHEMA.read(buf);\n\n// after - pick schema by API version\nSchema schema = apiVersion >= 3 ? FLEXIBLE_SCHEMA : LEGACY_SCHEMA;\nStruct s = (Struct) schema.read(buf);","handlingStrategy":"try-catch","validationCode":"// The (size-1) length is decoded from the varint inside read, so only a\n// minimum-feasible pre-check is possible. Verify the buffer can hold the\n// varint length prefix before calling read.\nif (buffer == null || buffer.remaining() < 1) {\n    throw new IllegalArgumentException(\"buffer too small for compact-array length varint\");\n}","typeGuard":null,"tryCatchPattern":"try {\n    Object[] arr = (Object[]) compactArrayOf.read(buffer);\n} catch (org.apache.kafka.common.protocol.types.SchemaException e) {\n    if (e.getMessage().contains(\"only\") && e.getMessage().contains(\"bytes available\")) {\n        // declared size exceeds remaining bytes -> truncated/corrupt compact array\n        failFrame(e);\n    } else {\n        throw e;\n    }\n}","preventionTips":["Verify the framing layer delivered the full declared payload before invoking the compact-array reader.","Remember compact length is encoded as N+1 via varint; off-by-one in custom framing causes this.","On short reads, discard the buffer and reconnect rather than re-reading from a tainted position.","Cross-check that the varint decoder (ByteUtils) and the framing agree on byte boundaries."],"tags":["protocol","serialization","compact-array","buffer-underflow","flexible-versions"],"analyzedSha":"c31c9215e131f8c17e79f8901b48c13ee6aa8e7a","analyzedAt":"2026-08-03T12:34:05.770Z","schemaVersion":2}