{"record":{"id":"521a980866551332","repo":"apache/hadoop","slug":"input-files-cannot-be-merged-as-they-have-differen","errorCode":null,"errorMessage":"Input files cannot be merged as they have different Key and Value classes","messagePattern":"Input files cannot be merged as they have different Key and Value classes","errorType":"validation","errorClass":"HadoopIllegalArgumentException","httpStatus":null,"severity":"error","filePath":"hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/MapFile.java","lineNumber":1069,"sourceCode":"        }\n      }\n    }\n\n    /*\n     * Open all input files for reading and verify the key and value types. And\n     * open Output file for writing\n     */\n    @SuppressWarnings(\"unchecked\")\n    private void open(Path[] inMapFiles, Path outMapFile) throws IOException {\n      inReaders = new Reader[inMapFiles.length];\n      for (int i = 0; i < inMapFiles.length; i++) {\n        Reader reader = new Reader(inMapFiles[i], conf);\n        if (keyClass == null || valueClass == null) {\n          keyClass = (Class<WritableComparable>) reader.getKeyClass();\n          valueClass = (Class<Writable>) reader.getValueClass();\n        } else if (keyClass != reader.getKeyClass()\n            || valueClass != reader.getValueClass()) {\n          throw new HadoopIllegalArgumentException(\n              \"Input files cannot be merged as they\"\n                  + \" have different Key and Value classes\");\n        }\n        inReaders[i] = reader;\n      }\n\n      if (comparator == null) {\n        Class<? extends WritableComparable> cls;\n        cls = keyClass.asSubclass(WritableComparable.class);\n        this.comparator = WritableComparator.get(cls, conf);\n      } else if (comparator.getKeyClass() != keyClass) {\n        throw new HadoopIllegalArgumentException(\n            \"Input files cannot be merged as they\"\n                + \" have different Key class compared to\"\n                + \" specified comparator\");\n      }\n\n      outWriter = new MapFile.Writer(conf, outMapFile,","sourceCodeStart":1051,"sourceCodeEnd":1087,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/MapFile.java#L1051-L1087","documentation":"Thrown by MapFile.Merger.open(Path[], Path) when any input MapFile's key or value class differs from the first input's. The merge algorithm assumes a homogeneous set — it reads records from every input into shared arrays and re-emits them through one output writer, which is only type-safe if every file carries the same keyClass/valueClass. The check is exact class identity (==), not assignability.","triggerScenarios":"Calling new MapFile.Merger(conf, comparator).merge(...) or the merge path over a list where one MapFile was written with Text keys and another with BytesWritable (or LongWritable values vs IntWritable values); a directory glob that accidentally picks up a differently-typed map.","commonSituations":"Merging outputs of jobs that changed writable types between runs; globs (part-*) sweeping in an old-format file from a previous version; hand-authored MapFiles mixed with generated ones.","solutions":["Partition inputs by (keyClass, valueClass) and run one merge per group.","Find the offending file: open each input with new MapFile.Reader(path, conf) and print getKeyClass()/getValueClass() — the mismatching one is the outlier to remove or rewrite.","If types must converge, convert the outlier file with a small read/write job into the target classes before merging."],"exampleFix":"// before: blind merge over a glob throws on the first mismatch\nnew MapFile.Merger(conf, null).merge(inputPaths, false, outPath);\n\n// after: group inputs by their declared classes before merging\nMap<String, List<Path>> groups = new HashMap<>();\nfor (Path p : inputPaths) {\n  try (MapFile.Reader r = new MapFile.Reader(p, conf)) {\n    groups.computeIfAbsent(r.getKeyClass() + \",\" + r.getValueClass(),\n        k -> new ArrayList<>()).add(p);\n  }\n}\nfor (List<Path> group : groups.values()) {\n  new MapFile.Merger(conf, null).merge(group.toArray(new Path[0]), false, outFor(group));\n}","handlingStrategy":"validation","validationCode":"Map<String, List<Path>> groups = new HashMap<>();\nfor (Path p : inputs) {\n  try (MapFile.Reader r = new MapFile.Reader(p, conf)) {\n    groups.computeIfAbsent(r.getKeyClass().getName() + \":\" + r.getValueClass().getName(),\n        k -> new ArrayList<>()).add(p);\n  }\n}\n// merge each homogeneous group separately\nfor (List<Path> g : groups.values()) {\n  new MapFile.Merger(conf, null).merge(g.toArray(new Path[0]), false, outFor(g));\n}","typeGuard":null,"tryCatchPattern":"try {\n  new MapFile.Merger(conf, cmp).merge(inputs, deleteInputs, out);\n} catch (HadoopIllegalArgumentException e) {\n  if (e.getMessage().contains(\"different Key and Value classes\")) {\n    // find and exclude/rewrite the outlier file whose classes differ\n    identifyOutlier(inputs, conf);\n  }\n  throw e;\n}","preventionTips":["Pre-open each input and record getKeyClass()/getValueClass() before any merge.","Constrain input globs to outputs of a single job/version so type drift cannot mix in.","Run one merge per (keyClass, valueClass) pair rather than one merge over everything."],"tags":["mapfile","merge","type-mismatch","hadoop-common"],"backgroundTag":"incompatible-input-types","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}