What causes "Entries are not comparable for unsorted TFiles" in apache/hadoop?

Write the TFile in sorted mode by passing a comparator name (e.g. "memcmp" or a custom RawComparator class name) to the TFile.Writer constructor

apache/hadoop · error · RuntimeException

Entries are not comparable for unsorted TFiles

Error message

Entries are not comparable for unsorted TFiles

What it means

Thrown by TFile.Reader.getEntryComparator() when the TFile was written without a comparator (unsorted mode, writer created with a null comparator name). An entry comparator only exists for sorted TFiles, because comparing entries means comparing keys with the file's embedded comparator. Unsorted files carry none, so the request is rejected with RuntimeException.

Source

Thrown at hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/file/tfile/TFile.java:937

     * @return The last key in the TFile.
     * @throws IOException raised on errors performing I/O.
     */
    public RawComparable getLastKey() throws IOException {
      checkTFileDataIndex();
      return tfileIndex.getLastKey();
    }

    /**
     * Get a Comparator object to compare Entries. It is useful when you want
     * stores the entries in a collection (such as PriorityQueue) and perform
     * sorting or comparison among entries based on the keys without copying out
     * the key.
     * 
     * @return An Entry Comparator..
     */
    public Comparator<Scanner.Entry> getEntryComparator() {
      if (!isSorted()) {
        throw new RuntimeException(
            "Entries are not comparable for unsorted TFiles");
      }

      return new Comparator<Scanner.Entry>() {
        /**
         * Provide a customized comparator for Entries. This is useful if we
         * have a collection of Entry objects. However, if the Entry objects
         * come from different TFiles, users must ensure that those TFiles share
         * the same RawComparator.
         */
        @Override
        public int compare(Scanner.Entry o1, Scanner.Entry o2) {
          return comparator.compare(o1.getKeyBuffer(), 0, o1.getKeyLength(), o2
              .getKeyBuffer(), 0, o2.getKeyLength());
        }
      };
    }

View on GitHub (pinned to 2add963021)

Solutions

Write the TFile in sorted mode by passing a comparator name (e.g. "memcmp" or a custom RawComparator class name) to the TFile.Writer constructor
If the file must stay unsorted, sort entries externally with your own comparator over the raw key bytes (entry.compareTo other entries is not defined)
Guard with reader.isSorted() before calling getEntryComparator() and fail with a clear domain error otherwise

Example fix

// before
Comparator<Scanner.Entry> cmp = reader.getEntryComparator(); // RuntimeException on unsorted file

// after
if (!reader.isSorted()) {
  throw new IOException("Cannot merge: input TFile is not sorted: " + path);
}
Comparator<Scanner.Entry> cmp = reader.getEntryComparator();

Defensive patterns

Strategy: validation

Validate before calling

if (!reader.isSorted()) {
  throw new IOException("getEntryComparator requires a sorted TFile: " + path);
}
Comparator<Scanner.Entry> entryCmp = reader.getEntryComparator();

Type guard

static boolean supportsEntryComparison(TFile.Reader reader) {
  return reader.isSorted();
}

Try / catch

catch (RuntimeException e) {
  if ("Entries are not comparable for unsorted TFiles".equals(e.getMessage())) {
    // fall back to external sorting with an application comparator over raw key bytes
  }
}

Prevention

Standardize producer writers on sorted mode (explicit comparator name) when consumers merge entries
Check reader.isSorted() before any comparator-dependent API
Document per dataset whether files are sorted and with which comparator

When it happens

Trigger: Calling reader.getEntryComparator() on a TFile written with new TFile.Writer(out, size, compressName, null); typical when putting Scanner.Entry objects into a PriorityQueue or TreeSet to merge or top-N entries.

Common situations: Generic merge code that assumes all TFiles are sorted; switching a producer from sorted to unsorted output while the consumer still collects entries into ordered collections; comparing entries across files that were written with different configurations.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/dd084077974e73a7. Report an issue: GitHub.