apache/hadoop · error · IOException

Too many splits

Error message

Too many splits

What it means

CompositeInputSplit is a fixed-capacity container for child InputSplits used by Hadoop's map-side join framework; add() throws IOException("Too many splits") once the number of add() calls reaches the capacity given at construction (new CompositeInputSplit(capacity)). The framework itself builds these in Parser.CNode.getSplits with capacity exactly equal to the number of child InputFormats in the join expression, then adds one split per child. Hitting it means your code (or a custom ComposableInputFormat) adds more children than it allocated.

Source

Thrown at hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/join/CompositeInputSplit.java:61

  private InputSplit[] splits;

  public CompositeInputSplit() { }

  public CompositeInputSplit(int capacity) {
    splits = new InputSplit[capacity];
  }

  /**
   * Add an InputSplit to this collection.
   * @throws IOException If capacity was not specified during construction
   *                     or if capacity has been reached.
   */
  public void add(InputSplit s) throws IOException {
    if (null == splits) {
      throw new IOException("Uninitialized InputSplit");
    }
    if (fill == splits.length) {
      throw new IOException("Too many splits");
    }
    splits[fill++] = s;
    totsize += s.getLength();
  }

  /**
   * Get ith child InputSplit.
   */
  public InputSplit get(int i) {
    return splits[i];
  }

  /**
   * Return the aggregate length of all child InputSplits currently added.
   */
  public long getLength() throws IOException {
    return totsize;
  }

View on GitHub (pinned to 2add963021)

Solutions

1. Construct the split with capacity equal to the exact number of children you will add: new CompositeInputSplit(children.length)
2. In custom ComposableInputFormat code, track add() calls against the capacity and fail loudly on mismatch before calling add()
3. When deserializing, read into a fresh CompositeInputSplit instance instead of adding to an already-filled one
4. Mirror Parser.CNode.getSplits: allocate new CompositeInputSplit(splits.length) and add splits[j][i] in the same loop that counts children

Example fix

// before
CompositeInputSplit cis = new CompositeInputSplit(2);
cis.add(a);
cis.add(b);
cis.add(c); // IOException: Too many splits

// after
InputSplit[] children = new InputSplit[] { a, b, c };
CompositeInputSplit cis = new CompositeInputSplit(children.length);
for (InputSplit s : children) {
  cis.add(s);
}

List<InputSplit> children = Arrays.asList(a, b, c);
CompositeInputSplit cis = new CompositeInputSplit(children.size());
for (InputSplit s : children) {
  cis.add(s); // count can never exceed capacity by construction
}

try {
  cis.add(split);
} catch (IOException e) {
  throw new IOException("composite split capacity " + cis.getLength()
      + " region exceeded; too many child splits", e);
}

When it happens

Trigger: Calling add() a (capacity+1)th time on a CompositeInputSplit; a custom join node that allocates new CompositeInputSplit(n) but adds n+1 splits; adding to a deserialized split whose backing array already matches the read cardinality, since readFields() does not reset the fill counter and fill stays equal to splits.length.

Common situations: Hand-building composite splits in unit tests; custom ComposableInputFormat implementations whose capacity calculation drifts from the actual child count when the mapred.join.expr arity changes; reusing one split instance across write/readFields and further mutation.

Related errors

• Inconsistent split cardinality from child {} ({}/{})
• Invalid split type:{}
• Uninitialized InputSplit
• Child key classes fail to agree
• Child value classes fail to agree

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/765ebbf54066151d. Report an issue: GitHub.