apache/druid · error · IllegalStateException

no value split found with fileSizeLimit

Error message

no value split found with fileSizeLimit [%d], avgObjectSize [%d]

What it means

GenericIndexedWriter converts generic-indexed data into files by choosing a power-of-two 'bags per file' split so each file stays under fileSizeLimit. bagSizePower() iterates candidate splits and verifies each actually fits; if no split satisfies the limit it throws ISE. In practice this means a single object is larger than the file size limit, so it can never fit.

Solutions

Increase the segment max file size config (e.g. druid.segment.progressPeriod/maxBytesPerFileBytes) so the largest value fits.
Reduce the size of individual values (truncate, hash, or split oversized dimension values before ingestion).
Verify serialization isn't inflating a single object unexpectedly (e.g. accidental concatenation of all values into one).

Example fix

// before
maxBytesPerFileBytes = 10_000; // single value is 50 KB
// after
maxBytesPerFileBytes = Math.max(maxBytesPerFileBytes, largestSerializedValueSize);

Defensive patterns

Strategy: validation

Validate before calling

long largest = maxSizeOfSerializedValue;
if (largest > maxBytesPerFileBytes) {
  throw new IllegalArgumentException("maxBytesPerFileBytes (" + maxBytesPerFileBytes + ") must exceed largest value (" + largest + ")");
}

Try / catch

try { writer.writeToMultiFiles(...); } catch (ISE e) {
  if (e.getMessage().startsWith("no value split found")) { /* raise maxBytesPerFileBytes and retry */ }
  throw e;
}

Prevention

Set maxBytesPerFileBytes comfortably above the largest serialized value you ingest
Reject or truncate oversized dimension values at ingestion time
Re-estimate file-size tuning after schema changes that widen values

When it happens

Trigger: writeToMultiFiles when avgObjectSize is so large that even (1 << 0) * avgObjectSize > fileSizeLimit, or actuallyFits(i) fails for every candidate — i.e. one serialized value exceeds maxFileSize.

Common situations: Very large individual dimension values (huge strings/blobs) combined with a small druid.segment.cache.initialSize / maxBytesPerFileBytes tuning, or misconfigured maxBytesPerFile smaller than any single value.

Understand the failure class

Background: "File too large" / "file size exceeds limit" errors: why libraries cap file sizes and how to fix them — this error's family across 46 libraries.

Related errors

AI-assisted analysis of apache/druid@9b90983fd2 (2026-09-07). Data as JSON: /api/errors/1551fa664ef66eb2. Report an issue: GitHub.

Appendix: source

Thrown at processing/src/main/java/org/apache/druid/segment/data/GenericIndexedWriter.java:453

  /**
   * Tries to get best value split(number of elements in each value file) which can be expressed as power of 2.
   *
   * @return Returns the size of value file splits as power of 2.
   *
   * @throws IOException
   */
  private int bagSizePower()
  {
    long avgObjectSize = (valuesOut.size() + numWritten - 1) / numWritten;

    for (int i = 31; i >= 0; --i) {
      if ((1L << i) * avgObjectSize <= fileSizeLimit) {
        if (actuallyFits(i)) {
          return i;
        }
      }
    }
    throw new ISE(
        "no value split found with fileSizeLimit [%d], avgObjectSize [%d]",
        fileSizeLimit,
        avgObjectSize
    );
  }

  /**
   * Checks if candidate value splits can divide value file in such a way no object/element crosses the value splits.
   *
   * @param powerTwo candidate value split expressed as power of 2.
   *
   * @return true if candidate value split can hold all splits.
   *
   * @throws IOException
   */
  private boolean actuallyFits(int powerTwo)
  {
    long lastValueOffset = 0;

View on GitHub (pinned to 9b90983fd2)