apache/druid · error · IllegalStateException

scratch buffer to big to write buckets

Error message

scratch buffer to big to write buckets

What it means

growScratch() doubles the internal ByteBuffer used to assemble front-coded buckets; it may only grow up to MAX_LOG_BUFFER_SIZE. When the scratch buffer is already at the maximum and a flush still needs more room, it throws IllegalStateException, indicating the bucket contents cannot be encoded within the allowed buffer size.

Solutions

1. Reduce bucketSize (e.g. from 128 to 8 or 16) so each bucket fits in the scratch buffer.
2. Re-encode data to keep individual int[] values smaller.
3. If this appears despite normal data, report as a bug; the limit is internal and not user-configurable.

Example fix

// before
new FrontCodedIntArrayIndexedWriter(medium, order, 128); // huge values overflow scratch
// after
new FrontCodedIntArrayIndexedWriter(medium, order, 16); // smaller buckets fit scratch

// Estimate encoded bucket size before writing; use smaller bucketSize when values are large
int bucketSize = valuesAreLarge ? 8 : 128;

try { writer.flush(); } catch (IllegalStateException e) { /* retry with smaller bucketSize */ }

When it happens

Trigger: Flushing or writing a bucket whose encoded size exceeds the maximum scratch capacity (1 << MAX_LOG_BUFFER_SIZE), typically with very large int[] values and a large bucketSize (e.g. 128 huge arrays).

Common situations: Ingesting columns with extremely long int arrays (huge dimension cardinalities per row or very wide multi-value values) combined with maximum bucket size 128.

Understand the failure class

Background: "This is a bug, please report it": internal invariant violations, unreachable panics, and SNH errors explained — this error's family across 47 libraries.

Related errors

• bucketSize must be a power of two (from 1 up to 128) but…
• no value split found with fileSizeLimit
• Values must be sorted and unique. Element
• 08001
• A batch appenderator was already created for this peon's…

AI-assisted analysis of apache/druid@9b90983fd2 (2026-09-07). Data as JSON: /api/errors/731aaf86e5170941. Report an issue: GitHub.

Appendix: source

Thrown at processing/src/main/java/org/apache/druid/segment/data/FrontCodedIntArrayIndexedWriter.java:292

    } while (written < 0);
    scratch.flip();
    Channels.writeFully(valuesOut, scratch);
    resetScratch();
    isClosed = true;
  }

  private void resetScratch()
  {
    scratch.position(0);
    scratch.limit(scratch.capacity());
  }

  private void growScratch()
  {
    if (logScratchSize < MAX_LOG_BUFFER_SIZE) {
      this.scratch = ByteBuffer.allocate(1 << ++logScratchSize).order(byteOrder);
    } else {
      throw new IllegalStateException("scratch buffer to big to write buckets");
    }
  }

  /**
   * Write bucket of values to a {@link ByteBuffer}. The first value is written completely, subsequent values are
   * written with an integer to indicate how much of the first value in the bucket is a prefix of the value, followed
   * by the remaining bytes of the value.
   *
   * Uses {@link VByte} encoded integers to indicate prefix length and value length.
   */
  public static int writeBucket(ByteBuffer buffer, int[][] values, int numValues)
  {
    int written = 0;
    int[] prev = null;
    while (written < numValues) {
      int[] next = values[written];
      if (written == 0) {
        prev = next;

View on GitHub (pinned to 9b90983fd2)