{"record":{"id":"c8e559c124b3b781","repo":"apache/hadoop","slug":"key-out-of-order-key-after-lastkey","errorCode":null,"errorMessage":"key out of order: {key} after {lastKey}","messagePattern":"key out of order: (.+?) after (.+?)","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/MapFile.java","lineNumber":421,"sourceCode":"\n      long pos = data.getLength();      \n      // Only write an index if we've changed positions. In a block compressed\n      // file, this means we write an entry at the start of each block      \n      if (size >= lastIndexKeyCount + indexInterval && pos > lastIndexPos) {\n        position.set(pos);                        // point to current eof\n        index.append(key, position);\n        lastIndexPos = pos;\n        lastIndexKeyCount = size;\n      }\n\n      data.append(key, val);                      // append key/value to data\n      size++;\n    }\n\n    private void checkKey(WritableComparable key) throws IOException {\n      // check that keys are well-ordered\n      if (size != 0 && comparator.compare(lastKey, key) > 0)\n        throw new IOException(\"key out of order: \"+key+\" after \"+lastKey);\n          \n      // update lastKey with a copy of key by writing and reading\n      outBuf.reset();\n      key.write(outBuf);                          // write new key\n\n      inBuf.reset(outBuf.getData(), outBuf.getLength());\n      lastKey.readFields(inBuf);                  // read into lastKey\n    }\n\n  }\n  \n  /** Provide access to an existing map. */\n  public static class Reader implements java.io.Closeable {\n      \n    /** Number of index entries to skip between each entry.  Zero by default.\n     * Setting this to values larger than zero can facilitate opening large map\n     * files using less memory. */\n    private int INDEX_SKIP = 0;","sourceCodeStart":403,"sourceCodeEnd":439,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/MapFile.java#L403-L439","documentation":"Thrown by MapFile.Writer.append (via checkKey) when the key being appended compares LESS than the previous key using this writer's comparator (comparator.compare(lastKey, key) > 0). A MapFile is a sorted archive: every append must be non-decreasing. The message names both the offending key and the previous lastKey, so the sort break is visible immediately.","triggerScenarios":"Appending keys in any order other than ascending per the comparator in effect — e.g. emitting Text keys out of lexicographic order, or appending IntWritable 5 after 10; using a custom WritableComparator whose compare() disagrees with the order the producer sorted by; a downstream job's reduce output assumption broken because the comparator differs from the mapper's partition/sort comparator.","commonSituations":"Feeding a MapFile.Writer from an unsorted map output instead of reduce-sorted output; changing the key class's serialization (e.g. Text encoding change) without regenerating files; custom comparators where compare() is not consistent with the natural order the data was produced in.","solutions":["Sort all entries with the SAME comparator the writer uses before appending (see example: sort with WritableComparator.get(keyClass)).","If input is a Hadoop job, run it through a reduce phase (or use sort job) so keys arrive sorted.","Verify a custom comparator's compare() is a valid total order and matches how the data was produced; unit-test it against sample pairs.","If you truly need unsorted data, use SequenceFile.Writer instead — MapFile's index requires sorted keys."],"exampleFix":"// before: appends unsorted keys, throws on second append\ntry (MapFile.Writer w = new MapFile.Writer(conf, dir,\n         MapFile.Writer.keyClass(Text.class))) {\n  for (Map.Entry<Text, Text> e : hashmap.entrySet()) w.append(e.getKey(), e.getValue());\n}\n\n// after: sort entries with the writer's comparator first\nWritableComparator cmp = WritableComparator.get(Text.class, conf);\nList<Map.Entry<Text, Text>> es = new ArrayList<>(hashmap.entrySet());\nes.sort((a, b) -> cmp.compare(a.getKey(), b.getKey()));\ntry (MapFile.Writer w = new MapFile.Writer(conf, dir,\n         MapFile.Writer.keyClass(Text.class))) {\n  for (Map.Entry<Text, Text> e : es) w.append(e.getKey(), e.getValue());\n}","handlingStrategy":"validation","validationCode":"WritableComparator cmp = WritableComparator.get(Text.class, conf);\nif (lastKey != null && cmp.compare(lastKey, nextKey) > 0) {\n  throw new IllegalArgumentException(\"Input not sorted at key \" + nextKey);\n}\nwriter.append(nextKey, value);","typeGuard":null,"tryCatchPattern":"try {\n  writer.append(key, value);\n} catch (IOException e) {\n  if (e.getMessage() != null && e.getMessage().startsWith(\"key out of order\")) {\n    failJobWithHint(\"Feed sorted input or use SequenceFile for unsorted data\");\n  } else { throw e; }\n}","preventionTips":["Sort with the SAME comparator instance/class the Writer will use — deriving it via WritableComparator.get(keyClass, conf) guarantees consistency.","Drive MapFile.Writer from reduce output or an explicit sort pass, never from a HashMap iteration.","Unit-test custom WritableComparators for total-order consistency before using them to write MapFiles."],"tags":["mapfile","sorted-keys","comparator","hadoop-common"],"backgroundTag":"unsorted-mapfile-keys","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}