apache/hadoop · error · IOException

Invalid split type:{}

Error message

Invalid split type:{}

What it means

CNode.getRecordReader expects the split produced by its own getSplits — a CompositeInputSplit bundling one child split per source — and throws IOException("Invalid split type:<class>") for anything else. It means a foreign split (typically a plain FileSplit from a file-based InputFormat) reached a composite join node, i.e. getSplits and getRecordReader are being served by different InputFormats or the split tree is hand-assembled.

Source

Thrown at hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/join/Parser.java:405

        }
        splits[i] = tmp;
      }
      final int size = splits[0].length;
      CompositeInputSplit[] ret = new CompositeInputSplit[size];
      for (int i = 0; i < size; ++i) {
        ret[i] = new CompositeInputSplit(splits.length);
        for (int j = 0; j < splits.length; ++j) {
          ret[i].add(splits[j][i]);
        }
      }
      return ret;
    }

    @SuppressWarnings("unchecked") // child types unknowable
    public ComposableRecordReader getRecordReader(
        InputSplit split, JobConf job, Reporter reporter) throws IOException {
      if (!(split instanceof CompositeInputSplit)) {
        throw new IOException("Invalid split type:" +
                              split.getClass().getName());
      }
      final CompositeInputSplit spl = (CompositeInputSplit)split;
      final int capacity = kids.size();
      CompositeRecordReader ret = null;
      try {
        if (!rrCstrMap.containsKey(ident)) {
          throw new IOException("No RecordReader for " + ident);
        }
        ret = (CompositeRecordReader)
          rrCstrMap.get(ident).newInstance(id, job, capacity, cmpcl);
      } catch (IllegalAccessException e) {
        throw (IOException)new IOException().initCause(e);
      } catch (InstantiationException e) {
        throw (IOException)new IOException().initCause(e);
      } catch (InvocationTargetException e) {
        throw (IOException)new IOException().initCause(e);
      }

View on GitHub (pinned to 2add963021)

Solutions

1. Use CompositeInputFormat end-to-end so splits passed to the node come from the same node's getSplits
2. Pass child splits only to child nodes (spl.get(i)) and CompositeInputSplit only to CNode
3. Type-check split instanceof CompositeInputSplit before calling getRecordReader in custom drivers

Example fix

// before
if (split instanceof FileSplit) {
  cnode.getRecordReader(split, job, reporter); // Invalid split type: FileSplit
}

// after
if (split instanceof CompositeInputSplit) {
  cnode.getRecordReader(split, job, reporter);
} else {
  throw new IOException("expected CompositeInputSplit, got " + split.getClass());
}

if (!(split instanceof CompositeInputSplit)) {
  throw new IOException("expected CompositeInputSplit from CompositeInputFormat.getSplits, got "
      + split.getClass().getName());
}

static boolean isCompositeSplit(InputSplit s) {
  return s instanceof CompositeInputSplit;
}

try {
  return cnode.getRecordReader(split, job, reporter);
} catch (IOException e) {
  throw new IOException("splits must come from the same CompositeInputFormat "
      + "that provides the record reader", e);
}

When it happens

Trigger: Custom code calling a composite node's getRecordReader with splits from a different InputFormat; a job whose InputFormat is not CompositeInputFormat but whose reader path routes into the join framework; feeding child splits directly to a composite node instead of its own zipped splits.

Common situations: Framework extensions and tests that mix splits from different formats; misconfigured jobs where the split producer and reader consumer disagree.

Related errors

• Too many splits
• Child key classes fail to agree
• Child value classes fail to agree
• Inconsistent split cardinality from child {} ({}/{})
• Invalid split type:{}

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/efa0e8858363c44b. Report an issue: GitHub.