apache/druid · error · IllegalArgumentException

buffer for list is too small, was

Error message

buffer for list is too small, was [%s] bytes, but need [%s] bytes.

What it means

ByteBufferIntList wraps an existing ByteBuffer to store up to maxElements ints. The constructor validates the backing buffer can hold maxElements * 4 bytes; if it is smaller, construction fails with an IAE because writes would corrupt adjacent memory regions.

Solutions

1. Allocate the buffer with at least maxElements * Integer.BYTES capacity
2. Reduce maxElements to buffer.capacity() / Integer.BYTES
3. Check byte-vs-element unit mistakes in the allocation code
4. Increase druid.query.groupBy max merge buffer settings if this comes from merge-buffer sizing

Example fix

// before
ByteBuffer buffer = ByteBuffer.allocateDirect(maxElements); // bytes, not ints
ByteBufferIntList list = new ByteBufferIntList(buffer, maxElements);
// after
ByteBuffer buffer = ByteBuffer.allocateDirect(maxElements * Integer.BYTES);
ByteBufferIntList list = new ByteBufferIntList(buffer, maxElements);

if (buffer.capacity() < maxElements * Integer.BYTES) {
  buffer = ByteBuffer.allocateDirect(maxElements * Integer.BYTES);
}

boolean fitsList(ByteBuffer b, int maxElements) { return b.capacity() >= maxElements * Integer.BYTES; }

try { new ByteBufferIntList(buffer, maxElements); } catch (IAE e) { if (e.getMessage().contains("too small")) { reallocate(buffer.capacity() * 2); } }

When it happens

Trigger: Creating a ByteBufferIntList with a buffer allocated smaller than maxElements * Integer.BYTES — e.g. merging dictionaries/bitmap lists in the groupBy merge buffer where maxElements was computed from a different (larger) sizing pass than the buffer allocation.

Common situations: Miscalculated merge buffer sizing after config change to druid.query.groupBy.maxMergeBuffer... or column value counts; off-by-one/×4 byte errors when pre-allocating buffers; reused buffers from a pool with smaller capacity.

Understand the failure class

Background: "Must be a positive integer", "Invalid value", "Unsupported": the invalid-argument-value error family, when a library rejects the value you pass — this error's family across 35 libraries.

Related errors

• Not enough capacity for even one row! Need[%,d] but…
• Group key should be a single dimension
• Invalid maxLoadFactor
• Invalid value[ ] for[ ]
• List is full with elements.

AI-assisted analysis of apache/druid@9b90983fd2 (2026-09-07). Data as JSON: /api/errors/2778b73c91ca0656. Report an issue: GitHub.

Appendix: source

Thrown at processing/src/main/java/org/apache/druid/query/groupby/epinephelinae/ByteBufferIntList.java:46

{
  private final ByteBuffer buffer;
  private final int maxElements;
  private int numElements;

  private int maxMergeBufferUsedBytes;

  public ByteBufferIntList(
      ByteBuffer buffer,
      int maxElements
  )
  {
    this.buffer = buffer;
    this.maxElements = maxElements;
    this.numElements = 0;
    this.maxMergeBufferUsedBytes = 0;

    if (buffer.capacity() < (maxElements * Integer.BYTES)) {
      throw new IAE(
          "buffer for list is too small, was [%s] bytes, but need [%s] bytes.",
          buffer.capacity(),
          maxElements * Integer.BYTES
      );
    }
  }

  public void add(int val)
  {
    if (numElements == maxElements) {
      throw new IndexOutOfBoundsException(StringUtils.format("List is full with %d elements.", maxElements));
    }
    buffer.putInt(numElements * Integer.BYTES, val);
    numElements++;
    maxMergeBufferUsedBytes = Math.max(maxMergeBufferUsedBytes, numElements * Integer.BYTES);
  }

  public void set(int index, int val)

View on GitHub (pinned to 9b90983fd2)