{"record":{"id":"765ebbf54066151d","repo":"apache/hadoop","slug":"too-many-splits","errorCode":null,"errorMessage":"Too many splits","messagePattern":"Too many splits","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/join/CompositeInputSplit.java","lineNumber":61,"sourceCode":"  private InputSplit[] splits;\n\n  public CompositeInputSplit() { }\n\n  public CompositeInputSplit(int capacity) {\n    splits = new InputSplit[capacity];\n  }\n\n  /**\n   * Add an InputSplit to this collection.\n   * @throws IOException If capacity was not specified during construction\n   *                     or if capacity has been reached.\n   */\n  public void add(InputSplit s) throws IOException {\n    if (null == splits) {\n      throw new IOException(\"Uninitialized InputSplit\");\n    }\n    if (fill == splits.length) {\n      throw new IOException(\"Too many splits\");\n    }\n    splits[fill++] = s;\n    totsize += s.getLength();\n  }\n\n  /**\n   * Get ith child InputSplit.\n   */\n  public InputSplit get(int i) {\n    return splits[i];\n  }\n\n  /**\n   * Return the aggregate length of all child InputSplits currently added.\n   */\n  public long getLength() throws IOException {\n    return totsize;\n  }","sourceCodeStart":43,"sourceCodeEnd":79,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/join/CompositeInputSplit.java#L43-L79","documentation":"CompositeInputSplit is a fixed-capacity container for child InputSplits used by Hadoop's map-side join framework; add() throws IOException(\"Too many splits\") once the number of add() calls reaches the capacity given at construction (new CompositeInputSplit(capacity)). The framework itself builds these in Parser.CNode.getSplits with capacity exactly equal to the number of child InputFormats in the join expression, then adds one split per child. Hitting it means your code (or a custom ComposableInputFormat) adds more children than it allocated.","triggerScenarios":"Calling add() a (capacity+1)th time on a CompositeInputSplit; a custom join node that allocates new CompositeInputSplit(n) but adds n+1 splits; adding to a deserialized split whose backing array already matches the read cardinality, since readFields() does not reset the fill counter and fill stays equal to splits.length.","commonSituations":"Hand-building composite splits in unit tests; custom ComposableInputFormat implementations whose capacity calculation drifts from the actual child count when the mapred.join.expr arity changes; reusing one split instance across write/readFields and further mutation.","solutions":["Construct the split with capacity equal to the exact number of children you will add: new CompositeInputSplit(children.length)","In custom ComposableInputFormat code, track add() calls against the capacity and fail loudly on mismatch before calling add()","When deserializing, read into a fresh CompositeInputSplit instance instead of adding to an already-filled one","Mirror Parser.CNode.getSplits: allocate new CompositeInputSplit(splits.length) and add splits[j][i] in the same loop that counts children"],"exampleFix":"// before\nCompositeInputSplit cis = new CompositeInputSplit(2);\ncis.add(a);\ncis.add(b);\ncis.add(c); // IOException: Too many splits\n\n// after\nInputSplit[] children = new InputSplit[] { a, b, c };\nCompositeInputSplit cis = new CompositeInputSplit(children.length);\nfor (InputSplit s : children) {\n  cis.add(s);\n}","handlingStrategy":"validation","validationCode":"List<InputSplit> children = Arrays.asList(a, b, c);\nCompositeInputSplit cis = new CompositeInputSplit(children.size());\nfor (InputSplit s : children) {\n  cis.add(s); // count can never exceed capacity by construction\n}","typeGuard":null,"tryCatchPattern":"try {\n  cis.add(split);\n} catch (IOException e) {\n  throw new IOException(\"composite split capacity \" + cis.getLength()\n      + \" region exceeded; too many child splits\", e);\n}","preventionTips":["Derive the constructor capacity from the children collection size, never a hardcoded number","In custom join nodes, keep the capacity expression adjacent to the add loop so both change together","Do not add to a CompositeInputSplit after readFields; deserialize into a fresh instance"],"tags":["hadoop","mapreduce","join","inputsplit","capacity"],"backgroundTag":"capacity-limit-exceeded","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}