{"record":{"id":"c86f7638c5511671","repo":"apache/hadoop","slug":"keys-are-not-added-in-sorted-order","errorCode":null,"errorMessage":"Keys are not added in sorted order","messagePattern":"Keys are not added in sorted order","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/file/tfile/TFile.java","lineNumber":474,"sourceCode":"           * verify length.\n           */\n          if (expectedLength >= 0 && expectedLength != len) {\n            throw new IOException(\"Incorrect key length: expected=\"\n                + expectedLength + \" actual=\" + len);\n          }\n\n          Utils.writeVInt(blkAppender, len);\n          blkAppender.write(key, 0, len);\n          if (tfileIndex.getFirstKey() == null) {\n            tfileIndex.setFirstKey(key, 0, len);\n          }\n\n          if (tfileMeta.isSorted() && tfileMeta.getRecordCount()>0) {\n            byte[] lastKey = lastKeyBufferOS.getBuffer();\n            int lastLen = lastKeyBufferOS.size();\n            if (tfileMeta.getComparator().compare(key, 0, len, lastKey, 0,\n                lastLen) < 0) {\n              throw new IOException(\"Keys are not added in sorted order\");\n            }\n          }\n\n          BoundedByteArrayOutputStream tmp = currentKeyBufferOS;\n          currentKeyBufferOS = lastKeyBufferOS;\n          lastKeyBufferOS = tmp;\n          --errorCount;\n        } finally {\n          closed = true;\n          state = State.END_KEY;\n        }\n      }\n    }\n\n    /**\n     * Helper class to register value after close call on value append stream.\n     */\n    private class ValueRegister extends DataOutputStream {","sourceCodeStart":456,"sourceCodeEnd":492,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/file/tfile/TFile.java#L456-L492","documentation":"Thrown while closing a key append stream on a sorted TFile (writer created with a comparator name) when the new key compares less than the previously appended key under the file's comparator. Sorted TFiles require keys in non-decreasing order because the block index and binary-search-based scanners depend on it. Once thrown, the writer is inconsistent and only close() is valid.","triggerScenarios":"writer.append(...) or prepareAppendKey(...)+write where the key is smaller than the last key, judged by the comparator passed to the TFile.Writer constructor (e.g. \"memcmp\", a BytesComparator, or a custom RawComparator class name).","commonSituations":"Feeding unsorted input into a sorted writer; a custom RawComparator whose compare() disagrees with the order keys were produced (sign inversion, comparing only a prefix, unsigned vs signed byte handling); merging sources each sorted with a different comparator; Hadoop version changes altering comparator semantics.","solutions":["Sort the input records with the exact same comparator before appending","If the data is not sorted, create the writer with a null comparator name to get an unsorted TFile","For a custom comparator, unit-test that compare(a,b) < 0 exactly when a should precede b, including unsigned bytes, prefixes, and equal keys"],"exampleFix":"// before\nwriter = new TFile.Writer(out, blockSize, \"none\", \"com.example.MyComparator\");\nfor (Record r : unsortedRecords) { writer.append(r.key(), r.value()); } // may throw\n\n// after\nunsortedRecords.sort(myComparator); // same ordering as MyComparator\nwriter = new TFile.Writer(out, blockSize, \"none\", \"com.example.MyComparator\");\nfor (Record r : unsortedRecords) { writer.append(r.key(), r.value()); }","handlingStrategy":"validation","validationCode":"// Mirror the file comparator on the last appended key before writing\nRawComparator<byte[]> cmp = fileComparator; // same comparator passed to TFile.Writer\nif (lastKey != null && cmp.compare(key, 0, key.length, lastKey, 0, lastKey.length) < 0) {\n  throw new IllegalArgumentException(\"Input not sorted at key: \" + Arrays.toString(key));\n}\nwriter.append(key, value);\nlastKey = key.clone();","typeGuard":null,"tryCatchPattern":"catch (IOException e) {\n  if (\"Keys are not added in sorted order\".equals(e.getMessage())) {\n    // writer is inconsistent: close it, discard the partial file, fix ordering upstream\n  }\n}","preventionTips":["Sort inputs with the identical comparator instance/class used for the writer","Unit-test custom RawComparators for antisymmetry and unsigned-byte semantics before deploying","If input order cannot be guaranteed, write unsorted TFiles (null comparator) instead"],"tags":["hadoop","tfile","sorted-order","comparator","writer"],"backgroundTag":"sort-order-violation","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-23T01:17:44.959Z"}