{"record":{"id":"06df9bed6c69bf06","repo":"apache/hadoop","slug":"inconsistent-split-cardinality-from-child-06df9b","errorCode":null,"errorMessage":"Inconsistent split cardinality from child {} ({}/{})","messagePattern":"Inconsistent split cardinality from child (.+?) \\((.+?)/(.+?)\\)","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/join/Parser.java","lineNumber":444,"sourceCode":"      }\n    }\n\n    /**\n     * Combine InputSplits from child InputFormats into a\n     * {@link CompositeInputSplit}.\n     */\n    @SuppressWarnings(\"unchecked\")\n\tpublic List<InputSplit> getSplits(JobContext job)\n        throws IOException, InterruptedException {\n      List<List<InputSplit>> splits = \n        new ArrayList<List<InputSplit>>(kids.size());\n      for (int i = 0; i < kids.size(); ++i) {\n        List<InputSplit> tmp = kids.get(i).getSplits(job);\n        if (null == tmp) {\n          throw new IOException(\"Error gathering splits from child RReader\");\n        }\n        if (i > 0 && splits.get(i-1).size() != tmp.size()) {\n          throw new IOException(\"Inconsistent split cardinality from child \" +\n              i + \" (\" + splits.get(i-1).size() + \"/\" + tmp.size() + \")\");\n        }\n        splits.add(i, tmp);\n      }\n      final int size = splits.get(0).size();\n      List<InputSplit> ret = new ArrayList<InputSplit>();\n      for (int i = 0; i < size; ++i) {\n        CompositeInputSplit split = new CompositeInputSplit(splits.size());\n        for (int j = 0; j < splits.size(); ++j) {\n          split.add(splits.get(j).get(i));\n        }\n        ret.add(split);\n      }\n      return ret;\n    }\n\n    @SuppressWarnings(\"unchecked\") // child types unknowable\n    public ComposableRecordReader ","sourceCodeStart":426,"sourceCodeEnd":462,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/join/Parser.java#L426-L462","documentation":"Parser.CNode.getSplits (Parser.java:444) requires every child input of a composite join node to return the same number of splits, because the ith split of each child is zipped into the ith CompositeInputSplit. When child i returns a different count than the previous child, this IOException names the offending child and both counts.","triggerScenarios":"CompositeInputFormat.getSplits first forces mapreduce.input.fileinputformat.split.minsize=Long.MAX_VALUE (CompositeInputFormat.java:128), making each child produce one split per input file. Joins therefore fail when sides have different numbers of files (e.g. 3 files on the left, 4 on the right → 3/4). Also triggered by custom InputFormats that ignore minsize or return fixed split counts.","commonSituations":"Reduce-side joins over directories with unequal file counts (one side written by a different-parallelism job); data refreshed on one side only; custom InputFormats with their own splitting logic; users assuming Hadoop joins work like SQL joins over arbitrary inputs.","solutions":["Make each join side contain the same number of files (repartition one side, e.g. run a re-partitioning job or use -Dfs.blocksize/combine small files into N files)","If a custom InputFormat controls splitting, override its split policy so both sides yield equal counts (the framework already forces one-split-per-file via minsize=Long.MAX_VALUE)","If cardinality genuinely differs, pre-sort and partition both sides by key into the same number of parts before joining","Pre-validate with a dry run: compute getSplits(job).size() for each child InputFormat over the same config and assert equality before submitting"],"exampleFix":"// before: sides with unequal file counts\n// /join/a  -> part-00000, part-00001, part-00002   (3 splits)\n// /join/b  -> part-00000, part-00001, part-00002, part-00003 (4 splits)\nString expr = CompositeInputFormat.compose(\"inner\", TextInputFormat.class, \"/join/a\", \"/join/b\");\n\n// after: repartition side b into exactly 3 files (or set both sides to 1 file)\n// e.g. hadoop fs -getmerge /join/b /tmp/b && hadoop fs -put /tmp/b /join/b_fixed/part-00000\nString expr = CompositeInputFormat.compose(\"inner\", TextInputFormat.class, \"/join/a\", \"/join/b_fixed\");","handlingStrategy":"validation","validationCode":"void assertEqualSplitCardinality(List<String> sides, Configuration conf) throws Exception {\n  CompositeInputFormat<?> stub = new CompositeInputFormat<>();\n  int prev = -1;\n  for (String side : sides) {\n    Job job = Job.getInstance(conf);\n    org.apache.hadoop.mapreduce.lib.input.FileInputFormat.setInputPaths(job, side);\n    int n = stub.getSplits(job).size();\n    if (prev >= 0 && prev != n) throw new IOException(\"split counts differ: \" + prev + \" vs \" + n);\n    prev = n;\n  }\n}","typeGuard":null,"tryCatchPattern":"try { root.getSplits(job); } catch (IOException e) { if (e.getMessage().startsWith(\"Inconsistent split cardinality\")) { /* repartition inputs to equal file counts, then resubmit */ } throw e; }","preventionTips":["Give every join side the same number of files before submitting (note: CompositeInputFormat forces one split per file via minsize=Long.MAX_VALUE)","Pre-check split counts per side with a dry getSplits run in the driver","Repartition/merge small files on the larger side so counts match"],"tags":["hadoop","mapreduce","join","input-split","cardinality"],"backgroundTag":"input-split-count-mismatch","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-23T01:17:44.959Z"}