apache/hadoop · error · HadoopIllegalArgumentException

Input files cannot be merged as they have different Key clas

Error message

Input files cannot be merged as they have different Key class compared to specified comparator

What it means

Thrown by MapFile.Merger.open(Path[], Path) when an explicit comparator was supplied whose key class does not identically match the key class of the input files (comparator.getKeyClass() != keyClass). The merger needs the comparator both to select the least key across inputs and to verify output ordering; a comparator bound to a different key type would compare apples to oranges and corrupt the merged map.

Source

Thrown at hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/MapFile.java:1081

        Reader reader = new Reader(inMapFiles[i], conf);
        if (keyClass == null || valueClass == null) {
          keyClass = (Class<WritableComparable>) reader.getKeyClass();
          valueClass = (Class<Writable>) reader.getValueClass();
        } else if (keyClass != reader.getKeyClass()
            || valueClass != reader.getValueClass()) {
          throw new HadoopIllegalArgumentException(
              "Input files cannot be merged as they"
                  + " have different Key and Value classes");
        }
        inReaders[i] = reader;
      }

      if (comparator == null) {
        Class<? extends WritableComparable> cls;
        cls = keyClass.asSubclass(WritableComparable.class);
        this.comparator = WritableComparator.get(cls, conf);
      } else if (comparator.getKeyClass() != keyClass) {
        throw new HadoopIllegalArgumentException(
            "Input files cannot be merged as they"
                + " have different Key class compared to"
                + " specified comparator");
      }

      outWriter = new MapFile.Writer(conf, outMapFile,
          MapFile.Writer.keyClass(keyClass),
          MapFile.Writer.valueClass(valueClass));
    }

    /**
     * Merge all input files to output map file.<br>
     * 1. Read first key/value from all input files to keys/values array. <br>
     * 2. Select the least key and corresponding value. <br>
     * 3. Write the selected key and value to output file. <br>
     * 4. Replace the already written key/value in keys/values arrays with the
     * next key/value from the selected input <br>
     * 5. Repeat step 2-4 till all keys are read. <br>

View on GitHub (pinned to 2add963021)

Solutions

Pass null as the comparator — Merger then derives WritableComparator.get(keyClass) from the files themselves.
Or supply a comparator created via WritableComparator.get(actualKeyClass, conf) so its key class matches by construction.
If using a custom comparator, make sure its getKeyClass() returns the exact class stored in the inputs.

Example fix

// before: comparator bound to Text but inputs hold LongWritable
WritableComparator cmp = WritableComparator.get(Text.class, conf);
new MapFile.Merger(conf, cmp).merge(inputs, false, out); // throws

// after: derive the comparator from the files' actual key class
Class<? extends WritableComparable> kc =
    new MapFile.Reader(inputs[0], conf).getKeyClass().asSubclass(WritableComparable.class);
new MapFile.Merger(conf, WritableComparator.get(kc, conf)).merge(inputs, false, out);

Defensive patterns

Strategy: validation

Validate before calling

if (comparator != null && comparator.getKeyClass() != keyClass) {
  comparator = WritableComparator.get(keyClass.asSubclass(WritableComparable.class), conf);
}
new MapFile.Merger(conf, comparator).merge(inputs, false, out);

Try / catch

try {
  new MapFile.Merger(conf, comparator).merge(inputs, deleteInputs, out);
} catch (HadoopIllegalArgumentException e) {
  if (e.getMessage().contains("specified comparator")) {
    // simplest recovery: pass null so Merger derives the comparator from inputs
    new MapFile.Merger(conf, null).merge(inputs, deleteInputs, out);
  } else { throw e; }
}

Prevention

Default to passing null comparator — MapFile.Merger derives the right one from the input files' key class.
If a custom comparator is required, construct it via WritableComparator.get(actualKeyClass, conf) so key classes match by construction.
When key types are refactored, grep for every Comparator handoff into Merger/Writer/Reader and update in the same commit.

When it happens

Trigger: Constructing MapFile.Merger(conf, comparator) where comparator is, say, a Text comparator while all inputs have LongWritable keys; reusing a comparator captured from another job's configuration; comparing custom writable subclasses where the comparator's getKeyClass() is the superclass while files store the subclass (identity == check fails).

Common situations: Refactoring key types without updating the comparator passed to the Merger; copy-pasted merge utility code with a hard-coded comparator; classloader subtleties where the same class name resolves to different Class instances.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/af639fca5b37a3fc. Report an issue: GitHub.