apache/hadoop · error · RuntimeException

Cannot compare keys for unsorted TFiles.

Error message

Cannot compare keys for unsorted TFiles.

What it means

Thrown by TFile.Reader.compareKeys(byte[] a, int o1, int l1, byte[] b, int o2, int l2) on an unsorted TFile. Key comparison via the reader delegates to the comparator embedded at write time; unsorted files (writer created with a null comparator name) have no comparator, so the call is rejected with RuntimeException rather than returning a meaningless ordering.

Source

Thrown at hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/file/tfile/TFile.java:1019

      int blkIndex =
          (greater) ? tfileIndex.upperBound(key) : tfileIndex.lowerBound(key);
      if (blkIndex < 0) return end;
      return new Location(blkIndex, 0);
    }

    Location getLocationByRecordNum(long recNum) throws IOException {
      checkTFileDataIndex();
      return tfileIndex.getLocationByRecordNum(recNum);
    }

    long getRecordNumByLocation(Location location) throws IOException {
      checkTFileDataIndex();
      return tfileIndex.getRecordNumByLocation(location);      
    }
    
    int compareKeys(byte[] a, int o1, int l1, byte[] b, int o2, int l2) {
      if (!isSorted()) {
        throw new RuntimeException("Cannot compare keys for unsorted TFiles.");
      }
      return comparator.compare(a, o1, l1, b, o2, l2);
    }

    int compareKeys(RawComparable a, RawComparable b) {
      if (!isSorted()) {
        throw new RuntimeException("Cannot compare keys for unsorted TFiles.");
      }
      return comparator.compare(a, b);
    }

    /**
     * Get the location pointing to the beginning of the first key-value pair in
     * a compressed block whose byte offset in the TFile is greater than or
     * equal to the specified offset.
     * 
     * @param offset
     *          the user supplied offset.

View on GitHub (pinned to 2add963021)

Solutions

1. Write input TFiles in sorted mode with an explicit comparator so compareKeys works
2. For unsorted files, compare raw key bytes yourself (e.g. with Bytes.compareTo or WritableComparator) instead of reader.compareKeys
3. Branch on reader.isSorted() and use a fallback comparator in the unsorted path

Example fix

// before
int c = reader.compareKeys(a, 0, a.length, b, 0, b.length); // RuntimeException if unsorted

// after
int c = reader.isSorted()
    ? reader.compareKeys(a, 0, a.length, b, 0, b.length)
    : Bytes.compareTo(a, 0, a.length, b, 0, b.length);

int compareKeys(TFile.Reader reader, byte[] a, byte[] b) {
  return reader.isSorted()
      ? reader.compareKeys(a, 0, a.length, b, 0, b.length)
      : Bytes.compareTo(a, 0, a.length, b, 0, b.length);
}

static boolean hasFileComparator(TFile.Reader reader) {
  return reader.isSorted();
}

catch (RuntimeException e) {
  if (e.getMessage() != null && e.getMessage().contains("unsorted TFiles")) {
    // compare raw bytes with your own comparator instead
  }
}

When it happens

Trigger: Calling reader.compareKeys(...) to order keys pulled from a TFile that was written without a comparator; common in merge utilities that compare current keys of several readers.

Common situations: Merging or diffing TFiles where some inputs were written unsorted; consumers assuming every TFile embeds a comparator; mixed-version pipelines where older writers omitted the comparator.

Related errors

• Entries are not comparable for unsorted TFiles
• Seeking in unsorted TFile
• Keys are not added in sorted order
• LZO codec %s=%s could not be loaded
• LZO codec class not specified. Did you forget to set propert

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/1316985400e0c382. Report an issue: GitHub.