apache/hadoop · error · IOException

Error gathering splits from child RReader

Error message

Error gathering splits from child RReader

What it means

Parser.CNode.getSplits (Parser.java:441) gathers splits from each child input of a composite join node; if a child InputFormat's getSplits(...) returns null (rather than a list), this IOException is thrown. Standard Hadoop InputFormats throw or return an (possibly empty) list, never null, so this almost always indicates a custom InputFormat breaking the API contract.

Source

Thrown at hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/join/Parser.java:441

      super.setKeyComparator(cmpcl);
      for (Node n : kids) {
        n.setKeyComparator(cmpcl);
      }
    }

    /**
     * Combine InputSplits from child InputFormats into a
     * {@link CompositeInputSplit}.
     */
    @SuppressWarnings("unchecked")
	public List<InputSplit> getSplits(JobContext job)
        throws IOException, InterruptedException {
      List<List<InputSplit>> splits = 
        new ArrayList<List<InputSplit>>(kids.size());
      for (int i = 0; i < kids.size(); ++i) {
        List<InputSplit> tmp = kids.get(i).getSplits(job);
        if (null == tmp) {
          throw new IOException("Error gathering splits from child RReader");
        }
        if (i > 0 && splits.get(i-1).size() != tmp.size()) {
          throw new IOException("Inconsistent split cardinality from child " +
              i + " (" + splits.get(i-1).size() + "/" + tmp.size() + ")");
        }
        splits.add(i, tmp);
      }
      final int size = splits.get(0).size();
      List<InputSplit> ret = new ArrayList<InputSplit>();
      for (int i = 0; i < size; ++i) {
        CompositeInputSplit split = new CompositeInputSplit(splits.size());
        for (int j = 0; j < splits.size(); ++j) {
          split.add(splits.get(j).get(i));
        }
        ret.add(split);
      }
      return ret;
    }

View on GitHub (pinned to 2add963021)

Solutions

1. Fix the custom child InputFormat to return an empty list (or throw) from getSplits — never null
2. If the child is third-party, wrap it in an adapter InputFormat that converts null to an empty list
3. Test the child InputFormat.getSplits standalone before using it in a join expression
4. For empty-input handling prefer returning an empty list so the join framework can reason about cardinality

Example fix

// before (custom InputFormat)
public List<InputSplit> getSplits(JobContext ctx) throws IOException {
  if (inputs.isEmpty()) return null; // triggers 'Error gathering splits from child RReader'
  ...
}

// after
public List<InputSplit> getSplits(JobContext ctx) throws IOException {
  if (inputs.isEmpty()) return Collections.emptyList();
  ...
}

static List<InputSplit> safeSplits(InputFormat<?,?> inf, JobContext ctx) throws IOException {
  List<InputSplit> s = inf.getSplits(ctx);
  return (s == null) ? java.util.Collections.emptyList() : s; // enforce non-null contract in wrappers
}

try { return root.getSplits(job); } catch (IOException e) { if (e.getMessage().contains("Error gathering splits")) { throw new IOException("Child InputFormat returned null splits — check custom InputFormat contract", e); } throw e; }

When it happens

Trigger: A child InputFormat used in tbl(...) overrides getSplits and returns null (e.g. on empty input or error paths); a stub/mock InputFormat in tests returns null; a third-party InputFormat implemented against an older/different contract.

Common situations: Custom or third-party InputFormats plugged into CompositeInputFormat joins; unit tests with mocked InputFormats that forget to return Collections.emptyList(); InputFormat wrappers that swallow exceptions and return null instead of propagating.

Related errors

• Uninitialized InputSplit
• Inconsistent split cardinality from child {} ({}/{})
• Invalid split type:{}
• Input only available on map
• Too many splits

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/77bf43289a015df1. Report an issue: GitHub.