apache/hadoop · error · HadoopIllegalArgumentException

Input files cannot be merged as they have different Key and

Error message

Input files cannot be merged as they have different Key and Value classes

What it means

Thrown by MapFile.Merger.open(Path[], Path) when any input MapFile's key or value class differs from the first input's. The merge algorithm assumes a homogeneous set — it reads records from every input into shared arrays and re-emits them through one output writer, which is only type-safe if every file carries the same keyClass/valueClass. The check is exact class identity (==), not assignability.

Source

Thrown at hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/MapFile.java:1069

        }
      }
    }

    /*
     * Open all input files for reading and verify the key and value types. And
     * open Output file for writing
     */
    @SuppressWarnings("unchecked")
    private void open(Path[] inMapFiles, Path outMapFile) throws IOException {
      inReaders = new Reader[inMapFiles.length];
      for (int i = 0; i < inMapFiles.length; i++) {
        Reader reader = new Reader(inMapFiles[i], conf);
        if (keyClass == null || valueClass == null) {
          keyClass = (Class<WritableComparable>) reader.getKeyClass();
          valueClass = (Class<Writable>) reader.getValueClass();
        } else if (keyClass != reader.getKeyClass()
            || valueClass != reader.getValueClass()) {
          throw new HadoopIllegalArgumentException(
              "Input files cannot be merged as they"
                  + " have different Key and Value classes");
        }
        inReaders[i] = reader;
      }

      if (comparator == null) {
        Class<? extends WritableComparable> cls;
        cls = keyClass.asSubclass(WritableComparable.class);
        this.comparator = WritableComparator.get(cls, conf);
      } else if (comparator.getKeyClass() != keyClass) {
        throw new HadoopIllegalArgumentException(
            "Input files cannot be merged as they"
                + " have different Key class compared to"
                + " specified comparator");
      }

      outWriter = new MapFile.Writer(conf, outMapFile,

View on GitHub (pinned to 2add963021)

Solutions

1. Partition inputs by (keyClass, valueClass) and run one merge per group.
2. Find the offending file: open each input with new MapFile.Reader(path, conf) and print getKeyClass()/getValueClass() — the mismatching one is the outlier to remove or rewrite.
3. If types must converge, convert the outlier file with a small read/write job into the target classes before merging.

Example fix

// before: blind merge over a glob throws on the first mismatch
new MapFile.Merger(conf, null).merge(inputPaths, false, outPath);

// after: group inputs by their declared classes before merging
Map<String, List<Path>> groups = new HashMap<>();
for (Path p : inputPaths) {
  try (MapFile.Reader r = new MapFile.Reader(p, conf)) {
    groups.computeIfAbsent(r.getKeyClass() + "," + r.getValueClass(),
        k -> new ArrayList<>()).add(p);
  }
}
for (List<Path> group : groups.values()) {
  new MapFile.Merger(conf, null).merge(group.toArray(new Path[0]), false, outFor(group));
}

Map<String, List<Path>> groups = new HashMap<>();
for (Path p : inputs) {
  try (MapFile.Reader r = new MapFile.Reader(p, conf)) {
    groups.computeIfAbsent(r.getKeyClass().getName() + ":" + r.getValueClass().getName(),
        k -> new ArrayList<>()).add(p);
  }
}
// merge each homogeneous group separately
for (List<Path> g : groups.values()) {
  new MapFile.Merger(conf, null).merge(g.toArray(new Path[0]), false, outFor(g));
}

try {
  new MapFile.Merger(conf, cmp).merge(inputs, deleteInputs, out);
} catch (HadoopIllegalArgumentException e) {
  if (e.getMessage().contains("different Key and Value classes")) {
    // find and exclude/rewrite the outlier file whose classes differ
    identifyOutlier(inputs, conf);
  }
  throw e;
}

When it happens

Trigger: Calling new MapFile.Merger(conf, comparator).merge(...) or the merge path over a list where one MapFile was written with Text keys and another with BytesWritable (or LongWritable values vs IntWritable values); a directory glob that accidentally picks up a differently-typed map.

Common situations: Merging outputs of jobs that changed writable types between runs; globs (part-*) sweeping in an old-format file from a previous version; hand-authored MapFiles mixed with generated ones.

Related errors

• Input files cannot be merged as they have different Key clas
• FileSystem '{}'is not a ViewFileSystem.
• key class or comparator option must be set
• Mkdirs failed to create directory {dirName}
• key out of order: {key} after {lastKey}

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/521a980866551332. Report an issue: GitHub.