{"record":{"id":"2778b73c91ca0656","repo":"apache/druid","slug":"buffer-for-list-is-too-small-was-s-bytes-but","errorCode":null,"errorMessage":"buffer for list is too small, was [%s] bytes, but need [%s] bytes.","messagePattern":"buffer for list is too small, was \\[(.+?)\\] bytes, but need \\[(.+?)\\] bytes\\.","errorType":"exception","errorClass":"IllegalArgumentException","httpStatus":null,"severity":"error","filePath":"processing/src/main/java/org/apache/druid/query/groupby/epinephelinae/ByteBufferIntList.java","lineNumber":46,"sourceCode":"{\n  private final ByteBuffer buffer;\n  private final int maxElements;\n  private int numElements;\n\n  private int maxMergeBufferUsedBytes;\n\n  public ByteBufferIntList(\n      ByteBuffer buffer,\n      int maxElements\n  )\n  {\n    this.buffer = buffer;\n    this.maxElements = maxElements;\n    this.numElements = 0;\n    this.maxMergeBufferUsedBytes = 0;\n\n    if (buffer.capacity() < (maxElements * Integer.BYTES)) {\n      throw new IAE(\n          \"buffer for list is too small, was [%s] bytes, but need [%s] bytes.\",\n          buffer.capacity(),\n          maxElements * Integer.BYTES\n      );\n    }\n  }\n\n  public void add(int val)\n  {\n    if (numElements == maxElements) {\n      throw new IndexOutOfBoundsException(StringUtils.format(\"List is full with %d elements.\", maxElements));\n    }\n    buffer.putInt(numElements * Integer.BYTES, val);\n    numElements++;\n    maxMergeBufferUsedBytes = Math.max(maxMergeBufferUsedBytes, numElements * Integer.BYTES);\n  }\n\n  public void set(int index, int val)","sourceCodeStart":28,"sourceCodeEnd":64,"githubUrl":"https://github.com/apache/druid/blob/9b90983fd291f26935af934383ce360473179e4d/processing/src/main/java/org/apache/druid/query/groupby/epinephelinae/ByteBufferIntList.java#L28-L64","documentation":"ByteBufferIntList wraps an existing ByteBuffer to store up to maxElements ints. The constructor validates the backing buffer can hold maxElements * 4 bytes; if it is smaller, construction fails with an IAE because writes would corrupt adjacent memory regions.","triggerScenarios":"Creating a ByteBufferIntList with a buffer allocated smaller than maxElements * Integer.BYTES — e.g. merging dictionaries/bitmap lists in the groupBy merge buffer where maxElements was computed from a different (larger) sizing pass than the buffer allocation.","commonSituations":"Miscalculated merge buffer sizing after config change to druid.query.groupBy.maxMergeBuffer... or column value counts; off-by-one/×4 byte errors when pre-allocating buffers; reused buffers from a pool with smaller capacity.","solutions":["Allocate the buffer with at least maxElements * Integer.BYTES capacity","Reduce maxElements to buffer.capacity() / Integer.BYTES","Check byte-vs-element unit mistakes in the allocation code","Increase druid.query.groupBy max merge buffer settings if this comes from merge-buffer sizing"],"exampleFix":"// before\nByteBuffer buffer = ByteBuffer.allocateDirect(maxElements); // bytes, not ints\nByteBufferIntList list = new ByteBufferIntList(buffer, maxElements);\n// after\nByteBuffer buffer = ByteBuffer.allocateDirect(maxElements * Integer.BYTES);\nByteBufferIntList list = new ByteBufferIntList(buffer, maxElements);","handlingStrategy":"validation","validationCode":"if (buffer.capacity() < maxElements * Integer.BYTES) {\n  buffer = ByteBuffer.allocateDirect(maxElements * Integer.BYTES);\n}","typeGuard":"boolean fitsList(ByteBuffer b, int maxElements) { return b.capacity() >= maxElements * Integer.BYTES; }","tryCatchPattern":"try { new ByteBufferIntList(buffer, maxElements); } catch (IAE e) { if (e.getMessage().contains(\"too small\")) { reallocate(buffer.capacity() * 2); } }","preventionTips":["Allocate in bytes: capacity = maxElements * 4","Keep buffer pools sized for the max element count actually used","Unit-test buffer sizing helpers with boundary values"],"tags":["druid","groupby","buffer-sizing","memory"],"backgroundTag":"invalid-argument-value","analyzedSha":"9b90983fd291f26935af934383ce360473179e4d","analyzedAt":"2026-09-07T13:32:30.957Z","contentChangedAt":"2026-09-07T13:32:30.957Z","schemaVersion":2},"datasetVersion":"2026-09-17T15:17:12.973Z"}