apache/hadoop · error · ClassCastException

Child key classes fail to agree

Error message

Child key classes fail to agree

What it means

CompositeRecordReader.createKey() asks each child RecordReader for its key class and throws ClassCastException("Child key classes fail to agree") when they differ. A map-side join streams all inputs in lockstep keyed by one shared WritableComparable, so every source listed in mapred.join.expr must emit exactly the same key class. The check runs lazily, on the first createKey() call, i.e. at the first next() in the task.

Source

Thrown at hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/join/CompositeRecordReader.java:414

  /**
   * Implement Comparable contract (compare key of join or head of heap
   * with that of another).
   */
  public int compareTo(ComposableRecordReader<K,?> other) {
    return cmp.compare(key(), other.key());
  }

  /**
   * Create a new key value common to all child RRs.
   * @throws ClassCastException if key classes differ.
   */
  @SuppressWarnings("unchecked") // Explicit check for key class agreement
  public K createKey() {
    if (null == keyclass) {
      final Class<?> cls = kids[0].createKey().getClass();
      for (RecordReader<K,? extends Writable> rr : kids) {
        if (!cls.equals(rr.createKey().getClass())) {
          throw new ClassCastException("Child key classes fail to agree");
        }
      }
      keyclass = cls.asSubclass(WritableComparable.class);
    }
    return (K) ReflectionUtils.newInstance(keyclass, getConf());
  }

  /**
   * Create a value to be used internally for joins.
   */
  protected TupleWritable createInternalValue() {
    Writable[] vals = new Writable[kids.length];
    for (int i = 0; i < vals.length; ++i) {
      vals[i] = kids[i].createValue();
    }
    return new TupleWritable(vals);
  }

View on GitHub (pinned to 2add963021)

Solutions

1. Pre-process the offending source with a map-only job that rewrites it using the same key class as the other sources
2. Use the same InputFormat class (hence the same key class) for every tbl(...) source in the expression
3. For SequenceFile sources, verify with SequenceFile.Reader.getKeyClass() on each path and re-generate the mismatches
4. Wrap the odd source in a custom ComposableRecordReader that projects its key onto the common key class

Example fix

// before: IntWritable keys joined against Text keys
job.set("mapred.join.expr",
    "inner(tbl(org.apache.hadoop.mapred.SequenceFileInputFormat,/ints)," +
    "tbl(org.apache.hadoop.mapred.KeyValueTextInputFormat,/text))");

// after: rewrite /text first so both sources are SequenceFiles with the same key class
job.set("mapred.join.expr",
    CompositeInputFormat.compose("inner", SequenceFileInputFormat.class,
        "/join/left", "/join/right"));

FileSystem fs = FileSystem.get(conf);
Set<Class<?>> keyClasses = new HashSet<Class<?>>();
for (Path p : joinSources) {
  SequenceFile.Reader r = new SequenceFile.Reader(fs, p, conf);
  try {
    keyClasses.add(r.getKeyClass());
  } finally {
    r.close();
  }
}
if (keyClasses.size() > 1) {
  throw new IOException("join sources disagree on key class: " + keyClasses);
}

static boolean keysAgree(ComposableRecordReader<? extends WritableComparable<?>> a,
    ComposableRecordReader<? extends WritableComparable<?>> b) {
  return a.createKey().getClass().equals(b.createKey().getClass());
}

try {
  K key = crr.createKey();
} catch (ClassCastException e) {
  throw new IOException("join children must share one key class; check every "
      + "tbl(...) input format in mapred.join.expr", e);
}

When it happens

Trigger: Joining sources with different key classes, e.g. inner(tbl(SequenceFileInputFormat,...) whose keys are IntWritable against tbl(KeyValueTextInputFormat,...) whose keys are Text; a custom ComposableRecordReader/WrappedRecordReader whose createKey() returns a class different from its siblings.

Common situations: Mixing input formats in one join expression without normalizing keys first; regenerating one source after a schema/serialization change so its key class no longer matches; a typo in the InputFormat class name so the wrong reader wraps a source.

Related errors

• Child value classes fail to agree
• Invalid split type:{}
• Invalid split type:{}
• Type mismatch in key from map: expected {keyClassName}, rece
• Type mismatch in value from map: expected {valueClassName},

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/dac479a5a6c5484b. Report an issue: GitHub.