{"record":{"id":"c6271d238269fb4b","repo":"apache/hadoop","slug":"key-out-of-order-k-after-lastkey","errorCode":null,"errorMessage":"key out of order: {k} after {lastKey}","messagePattern":"key out of order: (.+?) after (.+?)","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/MapFile.java","lineNumber":598,"sourceCode":"        return;\n      this.count = 0;\n      this.positions = new long[1024];\n\n      try {\n        int skip = INDEX_SKIP;\n        LongWritable position = new LongWritable();\n        WritableComparable lastKey = null;\n        long lastIndex = -1;\n        ArrayList<WritableComparable> keyBuilder = new ArrayList<WritableComparable>(1024);\n        while (true) {\n          WritableComparable k = comparator.newKey();\n\n          if (!index.next(k, position))\n            break;\n\n          // check order to make sure comparator is compatible\n          if (lastKey != null && comparator.compare(lastKey, k) > 0)\n            throw new IOException(\"key out of order: \"+k+\" after \"+lastKey);\n          lastKey = k;\n          if (skip > 0) {\n            skip--;\n            continue;                             // skip this entry\n          } else {\n            skip = INDEX_SKIP;                    // reset skip\n          }\n\n\t  // don't read an index that is the same as the previous one. Block\n\t  // compressed map files used to do this (multiple entries would point\n\t  // at the same block)\n\t  if (position.get() == lastIndex)\n\t    continue;\n\n          if (count == positions.length) {\n\t    positions = Arrays.copyOf(positions, positions.length * 2);\n          }\n","sourceCodeStart":580,"sourceCodeEnd":616,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/MapFile.java#L580-L616","documentation":"Thrown by MapFile.Reader's private readIndex() while it loads the index into memory: consecutive index keys must be ascending under the comparator in use, and comparator.compare(lastKey, k) > 0 means they are not. Per the code comment, this check exists specifically to detect an incompatible comparator — the file's index was written under a different ordering than the reader is applying. It surfaces lazily, on the first seek/get that forces index loading.","triggerScenarios":"Opening a MapFile with SequenceFile.Reader.comparator(cmp) where cmp orders keys differently than the comparator that wrote the file — e.g. a custom WritableComparator with different byte semantics, or a reversed/decreasing comparator; also a key class whose serialization changed between writing and reading so raw byte order no longer matches.","commonSituations":"Passing LongWritable.DecreasingComparator (or any descending comparator) to read an ascending map; upgrading a custom Writable whose compare() logic changed; reading a MapFile written by an old Hadoop version with modified key serialization; reader constructed with a comparator whose getKeyClass() differs in bytes-for-bytes layout from the writer's.","solutions":["Open the reader WITHOUT an explicit comparator so it derives the default one from the file's key class (data.getKeyClass()).","If you must pass a comparator, use the exact comparator class that wrote the file — check how the file was produced.","If key serialization changed across versions, regenerate the MapFile with the current code before reading.","Catch IOException around the first seek/get (index loading is lazy) to fail fast with your own context."],"exampleFix":"// before: custom/decreasing comparator contradicts the file's ordering\nMapFile.Reader r = new MapFile.Reader(dir, conf,\n    SequenceFile.Reader.comparator(new LongWritable.DecreasingComparator()));\nr.get(new LongWritable(42), value); // throws on first index load\n\n// after: let the reader derive the comparator from the file's own key class\nMapFile.Reader r = new MapFile.Reader(dir, conf);\nr.get(new LongWritable(42), value);","handlingStrategy":"validation","validationCode":"// derive the comparator from the file itself instead of guessing one\ntry (MapFile.Reader probe = new MapFile.Reader(dir, conf)) {\n  Class<?> fileKeyClass = probe.getDataKeyClassForValidation(); // or open data via SequenceFile\n}\nnew MapFile.Reader(dir, conf); // no comparator option = default comparator from key class","typeGuard":null,"tryCatchPattern":"try {\n  reader.seek(newKey); // first seek/get triggers readIndex()\n} catch (IOException e) {\n  if (e.getMessage() != null && e.getMessage().startsWith(\"key out of order\")) {\n    // reader comparator incompatible with file — reopen without explicit comparator\n    reader = new MapFile.Reader(dir, conf);\n  } else { throw e; }\n}","preventionTips":["Don't pass SequenceFile.Reader.comparator() to MapFile.Reader unless it is verbatim the comparator that wrote the map.","Remember index loading is lazy — wrap the FIRST access (seek/get), not just the constructor, in error handling.","After changing a key class's serialization or a comparator's logic, regenerate MapFiles; do not read across the change."],"tags":["mapfile","comparator","sorted-keys","hadoop-common"],"backgroundTag":"incompatible-comparator","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}