What causes "Error gathering splits from child RReader" in apache/hadoop?

Fix the child InputFormat to return an empty array instead of null

apache/hadoop · error · IOException

Error gathering splits from child RReader

Error message

Error gathering splits from child RReader

What it means

Parser.CNode.getSplits asks each child node for its splits and throws IOException("Error gathering splits from child RReader") when a child returns null instead of an array. The InputFormat.getSplits contract says implementations return an array (possibly empty), never null, so in practice this means a custom or third-party InputFormat inside a tbl(...) node violates that contract. The message text is misleading — the null comes from the child InputFormat, not a RecordReader.

Source

Thrown at hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/join/Parser.java:382

    public void setKeyComparator(Class<? extends WritableComparator> cmpcl) {
      super.setKeyComparator(cmpcl);
      for (Node n : kids) {
        n.setKeyComparator(cmpcl);
      }
    }

    /**
     * Combine InputSplits from child InputFormats into a
     * {@link CompositeInputSplit}.
     */
    public InputSplit[] getSplits(JobConf job, int numSplits)
        throws IOException {
      InputSplit[][] splits = new InputSplit[kids.size()][];
      for (int i = 0; i < kids.size(); ++i) {
        final InputSplit[] tmp = kids.get(i).getSplits(job, numSplits);
        if (null == tmp) {
          throw new IOException("Error gathering splits from child RReader");
        }
        if (i > 0 && splits[i-1].length != tmp.length) {
          throw new IOException("Inconsistent split cardinality from child " +
              i + " (" + splits[i-1].length + "/" + tmp.length + ")");
        }
        splits[i] = tmp;
      }
      final int size = splits[0].length;
      CompositeInputSplit[] ret = new CompositeInputSplit[size];
      for (int i = 0; i < size; ++i) {
        ret[i] = new CompositeInputSplit(splits.length);
        for (int j = 0; j < splits.length; ++j) {
          ret[i].add(splits[j][i]);
        }
      }
      return ret;
    }

View on GitHub (pinned to 2add963021)

Solutions

Fix the child InputFormat to return an empty array instead of null
Wrap the offending format in a delegating InputFormat that null-guards getSplits
Unit-test the custom format standalone with 0-, 1-, and N-file inputs before using it in a join

Example fix

// before (custom InputFormat)
public InputSplit[] getSplits(JobConf job, int numSplits) throws IOException {
  List<InputSplit> l = computeSplits(job); // may be null
  return l == null ? null : l.toArray(new InputSplit[0]);
}

// after
public InputSplit[] getSplits(JobConf job, int numSplits) throws IOException {
  List<InputSplit> l = computeSplits(job);
  return (l == null) ? new InputSplit[0] : l.toArray(new InputSplit[0]);
}

Defensive patterns

Strategy: validation

Validate before calling

// preflight: every child InputFormat must return a non-null split array
for (Path p : joinSources) {
  InputFormat<?, ?> child = getChildFormat(p);
  InputSplit[] s = child.getSplits(job, 1);
  if (s == null) {
    throw new IOException("child InputFormat " + child.getClass()
        + " returned null from getSplits");
  }
}

Try / catch

try {
  return cnodeOrFormat.getSplits(job, numSplits);
} catch (IOException e) {
  if (e.getMessage() != null && e.getMessage().contains("gathering splits")) {
    throw new IOException("a child InputFormat returned null splits; fix its getSplits()", e);
  }
  throw e;
}

Prevention

Custom InputFormats must return an array, empty if needed, never null
Unit-test custom formats with empty inputs before joining them
Wrap third-party formats in a null-guarding delegator

When it happens

Trigger: A custom InputFormat whose getSplits returns null on edge cases (empty input, swallowed error); a delegating wrapper InputFormat that forwards a null from its inner format without checking.

Common situations: Custom input formats bolted into a join expression; formats written for a client that tolerated null splits; formats that return null for empty directories instead of an empty array.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/2b6f5ab56f5935c7. Report an issue: GitHub.