apache/hadoop · error · RuntimeException

Cannot search in unsorted TFile

Error message

Cannot search in unsorted TFile

What it means

RuntimeException from TFileIndex.lowerBound(RawComparable): the file's comparator is null, which happens exactly when the comparator name stored at write time was empty — an unsorted TFile. lowerBound binary-searches the block index for the first key >= the probe, which is meaningless without a sort order, so the API refuses.

Source

Thrown at hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/file/tfile/TFile.java:2232

          recordNumIndex.add(sum);
        }
      } else {
        if (entryCount != 0) {
          throw new RuntimeException("Internal error");
        }
      }
      this.comparator = comparator;
    }

    /**
     * @param key
     *          input key.
     * @return the ID of the first block that contains key >= input key. Or -1
     *         if no such block exists.
     */
    public int lowerBound(RawComparable key) {
      if (comparator == null) {
        throw new RuntimeException("Cannot search in unsorted TFile");
      }

      if (firstKey == null) {
        return -1; // not found
      }

      int ret = Utils.lowerBound(index, key, comparator);
      if (ret == index.size()) {
        return -1;
      }
      return ret;
    }

    /**
     * @param key
     *          input key.
     * @return the ID of the first block that contains key > input key. Or -1
     *         if no such block exists.

View on GitHub (pinned to 2add963021)

Solutions

1. Check reader.getComparatorName() first: an empty name means unsorted; fall back to a full sequential scan.
2. If sorted access is required, rewrite the data sorted (Writer with COMPARATOR_MEMCMP and keys appended in order).
3. Fix the writer side: pass TFile.COMPARATOR_MEMCMP (and write keys in order) when creating files that will be searched.

Example fix

// before
TFile.Reader.Scanner s = reader.createScanner(probeKey, probeKey); // throws
// after
if (reader.getComparatorName() == null || reader.getComparatorName().isEmpty()) {
  // unsorted: sequential scan
  TFile.Reader.Scanner s = reader.createScanner();
} else {
  TFile.Reader.Scanner s = reader.createScanner(probeKey, probeKey);
}

String cmp = reader.getComparatorName();
boolean sorted = cmp != null && !cmp.isEmpty();
TFile.Reader.Scanner s = sorted ? reader.createScanner(probe, probe) : reader.createScanner(); // full scan

try {
  int block = reader.lowerBound(...); // via scanner seek
} catch (RuntimeException e) {
  if ("Cannot search in unsorted TFile".equals(e.getMessage())) { /* fall back to sequential scan */ }
  else throw e;
}

When it happens

Trigger: Calling reader.createScanner(key, ...) or any seek that routes to lowerBound on a file written with new TFile.Writer(out, blockSize, null, conf) or an empty comparator string. Unsorted files can only be scanned sequentially.

Common situations: A pipeline originally writing sorted (memcmp) files silently switches to unsorted writes, or a utility assumes all TFiles are sorted and hits an old unsorted archive.

Related errors

• Value length unknown.
• Entries are not comparable for unsorted TFiles
• Seeking in unsorted TFile
• Cannot compare keys for unsorted TFiles.
• Key length out of range: {klen}

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/0d7750ab9fa06ac0. Report an issue: GitHub.