{"record":{"id":"0d7750ab9fa06ac0","repo":"apache/hadoop","slug":"cannot-search-in-unsorted-tfile","errorCode":null,"errorMessage":"Cannot search in unsorted TFile","messagePattern":"Cannot search in unsorted TFile","errorType":"exception","errorClass":"RuntimeException","httpStatus":null,"severity":"error","filePath":"hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/file/tfile/TFile.java","lineNumber":2232,"sourceCode":"          recordNumIndex.add(sum);\n        }\n      } else {\n        if (entryCount != 0) {\n          throw new RuntimeException(\"Internal error\");\n        }\n      }\n      this.comparator = comparator;\n    }\n\n    /**\n     * @param key\n     *          input key.\n     * @return the ID of the first block that contains key >= input key. Or -1\n     *         if no such block exists.\n     */\n    public int lowerBound(RawComparable key) {\n      if (comparator == null) {\n        throw new RuntimeException(\"Cannot search in unsorted TFile\");\n      }\n\n      if (firstKey == null) {\n        return -1; // not found\n      }\n\n      int ret = Utils.lowerBound(index, key, comparator);\n      if (ret == index.size()) {\n        return -1;\n      }\n      return ret;\n    }\n\n    /**\n     * @param key\n     *          input key.\n     * @return the ID of the first block that contains key > input key. Or -1\n     *         if no such block exists.","sourceCodeStart":2214,"sourceCodeEnd":2250,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/file/tfile/TFile.java#L2214-L2250","documentation":"RuntimeException from TFileIndex.lowerBound(RawComparable): the file's comparator is null, which happens exactly when the comparator name stored at write time was empty — an unsorted TFile. lowerBound binary-searches the block index for the first key >= the probe, which is meaningless without a sort order, so the API refuses.","triggerScenarios":"Calling reader.createScanner(key, ...) or any seek that routes to lowerBound on a file written with new TFile.Writer(out, blockSize, null, conf) or an empty comparator string. Unsorted files can only be scanned sequentially.","commonSituations":"A pipeline originally writing sorted (memcmp) files silently switches to unsorted writes, or a utility assumes all TFiles are sorted and hits an old unsorted archive.","solutions":["Check reader.getComparatorName() first: an empty name means unsorted; fall back to a full sequential scan.","If sorted access is required, rewrite the data sorted (Writer with COMPARATOR_MEMCMP and keys appended in order).","Fix the writer side: pass TFile.COMPARATOR_MEMCMP (and write keys in order) when creating files that will be searched."],"exampleFix":"// before\nTFile.Reader.Scanner s = reader.createScanner(probeKey, probeKey); // throws\n// after\nif (reader.getComparatorName() == null || reader.getComparatorName().isEmpty()) {\n  // unsorted: sequential scan\n  TFile.Reader.Scanner s = reader.createScanner();\n} else {\n  TFile.Reader.Scanner s = reader.createScanner(probeKey, probeKey);\n}","handlingStrategy":"validation","validationCode":"String cmp = reader.getComparatorName();\nboolean sorted = cmp != null && !cmp.isEmpty();\nTFile.Reader.Scanner s = sorted ? reader.createScanner(probe, probe) : reader.createScanner(); // full scan","typeGuard":null,"tryCatchPattern":"try {\n  int block = reader.lowerBound(...); // via scanner seek\n} catch (RuntimeException e) {\n  if (\"Cannot search in unsorted TFile\".equals(e.getMessage())) { /* fall back to sequential scan */ }\n  else throw e;\n}","preventionTips":["Check getComparatorName() before any bounded scanner creation.","Decide sortedness at write time based on read patterns and record it in your catalog.","Write keys in order with COMPARATOR_MEMCMP when any search is expected."],"tags":["tfile","hadoop-common","unsorted","binary-search","precondition"],"backgroundTag":"search-on-unsorted-data","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-23T01:17:44.959Z"}