apache/hadoop · error · IOException

Inconsistent split cardinality from child {} ({}/{})

Error message

Inconsistent split cardinality from child {} ({}/{})

What it means

Parser.CNode.getSplits (Parser.java:444) requires every child input of a composite join node to return the same number of splits, because the ith split of each child is zipped into the ith CompositeInputSplit. When child i returns a different count than the previous child, this IOException names the offending child and both counts.

Source

Thrown at hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/join/Parser.java:444

      }
    }

    /**
     * Combine InputSplits from child InputFormats into a
     * {@link CompositeInputSplit}.
     */
    @SuppressWarnings("unchecked")
	public List<InputSplit> getSplits(JobContext job)
        throws IOException, InterruptedException {
      List<List<InputSplit>> splits = 
        new ArrayList<List<InputSplit>>(kids.size());
      for (int i = 0; i < kids.size(); ++i) {
        List<InputSplit> tmp = kids.get(i).getSplits(job);
        if (null == tmp) {
          throw new IOException("Error gathering splits from child RReader");
        }
        if (i > 0 && splits.get(i-1).size() != tmp.size()) {
          throw new IOException("Inconsistent split cardinality from child " +
              i + " (" + splits.get(i-1).size() + "/" + tmp.size() + ")");
        }
        splits.add(i, tmp);
      }
      final int size = splits.get(0).size();
      List<InputSplit> ret = new ArrayList<InputSplit>();
      for (int i = 0; i < size; ++i) {
        CompositeInputSplit split = new CompositeInputSplit(splits.size());
        for (int j = 0; j < splits.size(); ++j) {
          split.add(splits.get(j).get(i));
        }
        ret.add(split);
      }
      return ret;
    }

    @SuppressWarnings("unchecked") // child types unknowable
    public ComposableRecordReader 

View on GitHub (pinned to 2add963021)

Solutions

1. Make each join side contain the same number of files (repartition one side, e.g. run a re-partitioning job or use -Dfs.blocksize/combine small files into N files)
2. If a custom InputFormat controls splitting, override its split policy so both sides yield equal counts (the framework already forces one-split-per-file via minsize=Long.MAX_VALUE)
3. If cardinality genuinely differs, pre-sort and partition both sides by key into the same number of parts before joining
4. Pre-validate with a dry run: compute getSplits(job).size() for each child InputFormat over the same config and assert equality before submitting

Example fix

// before: sides with unequal file counts
// /join/a  -> part-00000, part-00001, part-00002   (3 splits)
// /join/b  -> part-00000, part-00001, part-00002, part-00003 (4 splits)
String expr = CompositeInputFormat.compose("inner", TextInputFormat.class, "/join/a", "/join/b");

// after: repartition side b into exactly 3 files (or set both sides to 1 file)
// e.g. hadoop fs -getmerge /join/b /tmp/b && hadoop fs -put /tmp/b /join/b_fixed/part-00000
String expr = CompositeInputFormat.compose("inner", TextInputFormat.class, "/join/a", "/join/b_fixed");

void assertEqualSplitCardinality(List<String> sides, Configuration conf) throws Exception {
  CompositeInputFormat<?> stub = new CompositeInputFormat<>();
  int prev = -1;
  for (String side : sides) {
    Job job = Job.getInstance(conf);
    org.apache.hadoop.mapreduce.lib.input.FileInputFormat.setInputPaths(job, side);
    int n = stub.getSplits(job).size();
    if (prev >= 0 && prev != n) throw new IOException("split counts differ: " + prev + " vs " + n);
    prev = n;
  }
}

try { root.getSplits(job); } catch (IOException e) { if (e.getMessage().startsWith("Inconsistent split cardinality")) { /* repartition inputs to equal file counts, then resubmit */ } throw e; }

When it happens

Trigger: CompositeInputFormat.getSplits first forces mapreduce.input.fileinputformat.split.minsize=Long.MAX_VALUE (CompositeInputFormat.java:128), making each child produce one split per input file. Joins therefore fail when sides have different numbers of files (e.g. 3 files on the left, 4 on the right → 3/4). Also triggered by custom InputFormats that ignore minsize or return fixed split counts.

Common situations: Reduce-side joins over directories with unequal file counts (one side written by a different-parallelism job); data refreshed on one side only; custom InputFormats with their own splitting logic; users assuming Hadoop joins work like SQL joins over arbitrary inputs.

Related errors

• Uninitialized InputSplit
• Error gathering splits from child RReader
• Invalid split type:{}
• Input only available on map
• Too many splits

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/06df9bed6c69bf06. Report an issue: GitHub.