{"record":{"id":"731aaf86e5170941","repo":"apache/druid","slug":"scratch-buffer-to-big-to-write-buckets-731aaf","errorCode":null,"errorMessage":"scratch buffer to big to write buckets","messagePattern":"scratch buffer to big to write buckets","errorType":"exception","errorClass":"IllegalStateException","httpStatus":null,"severity":"error","filePath":"processing/src/main/java/org/apache/druid/segment/data/FrontCodedIntArrayIndexedWriter.java","lineNumber":292,"sourceCode":"    } while (written < 0);\n    scratch.flip();\n    Channels.writeFully(valuesOut, scratch);\n    resetScratch();\n    isClosed = true;\n  }\n\n  private void resetScratch()\n  {\n    scratch.position(0);\n    scratch.limit(scratch.capacity());\n  }\n\n  private void growScratch()\n  {\n    if (logScratchSize < MAX_LOG_BUFFER_SIZE) {\n      this.scratch = ByteBuffer.allocate(1 << ++logScratchSize).order(byteOrder);\n    } else {\n      throw new IllegalStateException(\"scratch buffer to big to write buckets\");\n    }\n  }\n\n  /**\n   * Write bucket of values to a {@link ByteBuffer}. The first value is written completely, subsequent values are\n   * written with an integer to indicate how much of the first value in the bucket is a prefix of the value, followed\n   * by the remaining bytes of the value.\n   *\n   * Uses {@link VByte} encoded integers to indicate prefix length and value length.\n   */\n  public static int writeBucket(ByteBuffer buffer, int[][] values, int numValues)\n  {\n    int written = 0;\n    int[] prev = null;\n    while (written < numValues) {\n      int[] next = values[written];\n      if (written == 0) {\n        prev = next;","sourceCodeStart":274,"sourceCodeEnd":310,"githubUrl":"https://github.com/apache/druid/blob/9b90983fd291f26935af934383ce360473179e4d/processing/src/main/java/org/apache/druid/segment/data/FrontCodedIntArrayIndexedWriter.java#L274-L310","documentation":"growScratch() doubles the internal ByteBuffer used to assemble front-coded buckets; it may only grow up to MAX_LOG_BUFFER_SIZE. When the scratch buffer is already at the maximum and a flush still needs more room, it throws IllegalStateException, indicating the bucket contents cannot be encoded within the allowed buffer size.","triggerScenarios":"Flushing or writing a bucket whose encoded size exceeds the maximum scratch capacity (1 << MAX_LOG_BUFFER_SIZE), typically with very large int[] values and a large bucketSize (e.g. 128 huge arrays).","commonSituations":"Ingesting columns with extremely long int arrays (huge dimension cardinalities per row or very wide multi-value values) combined with maximum bucket size 128.","solutions":["Reduce bucketSize (e.g. from 128 to 8 or 16) so each bucket fits in the scratch buffer.","Re-encode data to keep individual int[] values smaller.","If this appears despite normal data, report as a bug; the limit is internal and not user-configurable."],"exampleFix":"// before\nnew FrontCodedIntArrayIndexedWriter(medium, order, 128); // huge values overflow scratch\n// after\nnew FrontCodedIntArrayIndexedWriter(medium, order, 16); // smaller buckets fit scratch","handlingStrategy":"fallback","validationCode":"// Estimate encoded bucket size before writing; use smaller bucketSize when values are large\nint bucketSize = valuesAreLarge ? 8 : 128;","typeGuard":null,"tryCatchPattern":"try { writer.flush(); } catch (IllegalStateException e) { /* retry with smaller bucketSize */ }","preventionTips":["Choose smaller bucketSize when individual int[] values can be very large","Avoid bucketSize=128 for wide multi-value dimension columns","If reproducible with ordinary data, file a bug — the scratch cap is internal"],"tags":["java","segment-writer","buffer-overflow"],"backgroundTag":"internal-invariant-violation","analyzedSha":"9b90983fd291f26935af934383ce360473179e4d","analyzedAt":"2026-09-07T13:32:30.957Z","contentChangedAt":"2026-09-07T13:32:30.957Z","schemaVersion":2},"datasetVersion":"2026-09-17T15:17:12.973Z"}