apache/hadoop · error · IOException

Too many splits

Error message

Too many splits

What it means

Error "Too many splits" thrown in apache/hadoop.

Source

Thrown at hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/join/CompositeInputSplit.java:67

  private Configuration conf = new Configuration();

  public CompositeInputSplit() { }

  public CompositeInputSplit(int capacity) {
    splits = new InputSplit[capacity];
  }

  /**
   * Add an InputSplit to this collection.
   * @throws IOException If capacity was not specified during construction
   *                     or if capacity has been reached.
   */
  public void add(InputSplit s) throws IOException, InterruptedException {
    if (null == splits) {
      throw new IOException("Uninitialized InputSplit");
    }
    if (fill == splits.length) {
      throw new IOException("Too many splits");
    }
    splits[fill++] = s;
    totsize += s.getLength();
  }

  /**
   * Get ith child InputSplit.
   */
  public InputSplit get(int i) {
    return splits[i];
  }

  /**
   * Return the aggregate length of all child InputSplits currently added.
   */
  public long getLength() throws IOException {
    return totsize;
  }

View on GitHub (pinned to 2add963021)

Solutions

1. Reduce the number of child splits combined into a single CompositeInputSplit; the count must fit the internal representation.
2. Increase the split size (mapreduce.input.fileinputformat.split.maxsize) or reduce the number of input paths so fewer splits are created.

Example fix

job.getConfiguration().setLong("mapreduce.input.fileinputformat.split.maxsize", 256L*1024*1024);

When it happens

Trigger: Thrown by CompositeInputSplit.add() when more child splits are added than the split was constructed to hold (capacity set at construction). Ensure the number of child input formats matches the CompositeInputSplit capacity.

Common situations: Joining a very large number of small input files/paths in one composite split.

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/5d81c3aaadf6594e. Report an issue: GitHub.