apache/hadoop · error · IOException

Invalid split type:{}

Error message

Invalid split type:{}

What it means

Parser.CNode.createRecordReader (Parser.java:466) requires the InputSplit passed in to be a CompositeInputSplit — one assembled by CompositeInputFormat.getSplits, holding one child split per join input. Receiving any other split type (e.g. a plain FileSplit) is a programming/framework error: the composite record reader cannot extract its per-child splits.

Source

Thrown at hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/join/Parser.java:466

      }
      final int size = splits.get(0).size();
      List<InputSplit> ret = new ArrayList<InputSplit>();
      for (int i = 0; i < size; ++i) {
        CompositeInputSplit split = new CompositeInputSplit(splits.size());
        for (int j = 0; j < splits.size(); ++j) {
          split.add(splits.get(j).get(i));
        }
        ret.add(split);
      }
      return ret;
    }

    @SuppressWarnings("unchecked") // child types unknowable
    public ComposableRecordReader 
        createRecordReader(InputSplit split, TaskAttemptContext taskContext) 
        throws IOException, InterruptedException {
      if (!(split instanceof CompositeInputSplit)) {
        throw new IOException("Invalid split type:" +
                              split.getClass().getName());
      }
      final CompositeInputSplit spl = (CompositeInputSplit)split;
      final int capacity = kids.size();
      CompositeRecordReader ret = null;
      try {
        if (!rrCstrMap.containsKey(ident)) {
          throw new IOException("No RecordReader for " + ident);
        }
        ret = (CompositeRecordReader)rrCstrMap.get(ident).
          newInstance(id, taskContext.getConfiguration(), capacity, cmpcl);
      } catch (IllegalAccessException e) {
        throw new IOException(e);
      } catch (InstantiationException e) {
        throw new IOException(e);
      } catch (InvocationTargetException e) {
        throw new IOException(e);
      }

View on GitHub (pinned to 2add963021)

Solutions

1. Always obtain splits via CompositeInputFormat.getSplits(context) and pass the resulting CompositeInputSplit objects to createRecordReader
2. In delegating InputFormats, build a CompositeInputSplit(capacity) and add(childSplits) before creating the reader
3. In tests, construct splits via getSplits on test input rather than new FileSplit(...)
4. Fail fast upstream: check split instanceof CompositeInputSplit in wrapper code before delegation

Example fix

// before (test / wrapper code)
InputSplit split = fileSplits.get(0); // FileSplit!
RecordReader<Text, TupleWritable> rr = compositeIF.createRecordReader(split, ctx);

// after
List<InputSplit> splits = compositeIF.getSplits(job); // CompositeInputSplits
RecordReader<Text, TupleWritable> rr = compositeIF.createRecordReader(splits.get(0), ctx);

if (!(split instanceof CompositeInputSplit)) throw new IllegalArgumentException("Expected CompositeInputSplit, got " + split.getClass().getName());

static boolean isCompositeSplit(InputSplit s) { return s instanceof org.apache.hadoop.mapreduce.lib.join.CompositeInputSplit; }

try { return cnode.createRecordReader(split, ctx); } catch (IOException e) { if (e.getMessage().startsWith("Invalid split type")) throw new IllegalStateException("Splits must come from CompositeInputFormat.getSplits", e); throw e; }

When it happens

Trigger: Calling CompositeInputFormat (or a composite node's) createRecordReader with a FileSplit or custom InputSplit instead of a CompositeInputSplit; a custom InputFormat delegating to the join framework but forwarding its own raw splits; unit tests constructing record readers directly with arbitrary splits; a scheduler/AM misrouting splits between InputFormats.

Common situations: Writing custom InputFormats that delegate to CompositeInputFormat; direct unit-testing of createRecordReader without building splits through getSplits(); frameworks (Hadoop Streaming, Crunch, Hive wrappers) that pass through splits of the wrong type when the join InputFormat is nested.

Related errors

• Uninitialized InputSplit
• Child key classes fail to agree
• Child value classes fail to agree
• Invalid split type:{}
• Error gathering splits from child RReader

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/6ffed1924556f2d3. Report an issue: GitHub.