{"record":{"id":"c7785fabf376a2d6","repo":"apache/hadoop","slug":"inconsistent-split-cardinality-from-child","errorCode":null,"errorMessage":"Inconsistent split cardinality from child {} ({}/{})","messagePattern":"Inconsistent split cardinality from child (.+?) \\((.+?)/(.+?)\\)","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/join/Parser.java","lineNumber":385,"sourceCode":"      for (Node n : kids) {\n        n.setKeyComparator(cmpcl);\n      }\n    }\n\n    /**\n     * Combine InputSplits from child InputFormats into a\n     * {@link CompositeInputSplit}.\n     */\n    public InputSplit[] getSplits(JobConf job, int numSplits)\n        throws IOException {\n      InputSplit[][] splits = new InputSplit[kids.size()][];\n      for (int i = 0; i < kids.size(); ++i) {\n        final InputSplit[] tmp = kids.get(i).getSplits(job, numSplits);\n        if (null == tmp) {\n          throw new IOException(\"Error gathering splits from child RReader\");\n        }\n        if (i > 0 && splits[i-1].length != tmp.length) {\n          throw new IOException(\"Inconsistent split cardinality from child \" +\n              i + \" (\" + splits[i-1].length + \"/\" + tmp.length + \")\");\n        }\n        splits[i] = tmp;\n      }\n      final int size = splits[0].length;\n      CompositeInputSplit[] ret = new CompositeInputSplit[size];\n      for (int i = 0; i < size; ++i) {\n        ret[i] = new CompositeInputSplit(splits.length);\n        for (int j = 0; j < splits.length; ++j) {\n          ret[i].add(splits[j][i]);\n        }\n      }\n      return ret;\n    }\n\n    @SuppressWarnings(\"unchecked\") // child types unknowable\n    public ComposableRecordReader getRecordReader(\n        InputSplit split, JobConf job, Reporter reporter) throws IOException {","sourceCodeStart":367,"sourceCodeEnd":403,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/join/Parser.java#L367-L403","documentation":"CNode.getSplits zips the ith split of every child into one CompositeInputSplit, so every source must return exactly the same number of splits; otherwise it throws IOException(\"Inconsistent split cardinality from child i (a/b)\") with the two counts. CompositeInputFormat.getSplits defends against this by forcing job.setLong(\"mapred.min.split.size\", Long.MAX_VALUE) so each file-based child yields a single split — hitting the error means that defense was bypassed or a child InputFormat ignores the setting.","triggerScenarios":"Driving Parser nodes directly instead of through CompositeInputFormat.getSplits; a child InputFormat (custom, or database-style) that computes splits from its own hints and ignores mapred.min.split.size; children reading inputs with very different file sizes or block counts so their split counts differ.","commonSituations":"Joining one large source with one small source and expecting the framework to re-shard them; custom input formats inside tbl(...); a hand-rolled driver replacing CompositeInputFormat; inputs with different replication of many small files.","solutions":["Use CompositeInputFormat (mapred) as the job's InputFormat so its min-split-size forcing applies to all children","If driving children yourself, set mapred.min.split.size to Long.MAX_VALUE in the JobConf passed to getSplits so each source yields one split","Make custom child InputFormats honor mapred.min.split.size or otherwise return equal split counts","Pre-partition all sources identically — same number of similarly sized, identically sorted parts"],"exampleFix":"// before: custom driver, unequal split counts\nInputSplit[] a = fmtA.getSplits(job, 4); // 4 splits\nInputSplit[] b = fmtB.getSplits(job, 1); // 1 split -> inconsistent cardinality\n\n// after: one split per source, exactly what CompositeInputFormat.getSplits does\njob.setLong(\"mapred.min.split.size\", Long.MAX_VALUE);\nInputSplit[] a = fmtA.getSplits(job, 1);\nInputSplit[] b = fmtB.getSplits(job, 1);","handlingStrategy":"validation","validationCode":"// preflight: force one split per source (exactly what CompositeInputFormat does)\njob.setLong(\"mapred.min.split.size\", Long.MAX_VALUE);\nfor (int i = 0; i < kids; i++) {\n  InputSplit[] s = childFormats.get(i).getSplits(job, 1);\n  if (i > 0 && s.length != prevLen) {\n    throw new IOException(\"child \" + i + \" yields \" + s.length\n        + \" splits vs \" + prevLen + \" from child 0\");\n  }\n  prevLen = s.length;\n}","typeGuard":null,"tryCatchPattern":"try {\n  return cif.getSplits(job, numSplits);\n} catch (IOException e) {\n  if (e.getMessage() != null && e.getMessage().contains(\"cardinality\")) {\n    throw new IOException(\"join sources produced unequal split counts; \"\n        + \"set mapred.min.split.size=Long.MAX_VALUE or repartition inputs\", e);\n  }\n  throw e;\n}","preventionTips":["Use CompositeInputFormat as the InputFormat so its min-split-size forcing applies","Make custom child InputFormats honor mapred.min.split.size","Keep join inputs partitioned identically (same part count and similar sizes)"],"tags":["hadoop","mapreduce","join","inputsplit","partitioning","map-side-join"],"backgroundTag":"unequal-partition-count","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}