apache/hadoop · error · InvalidJobConfException

Native output collector doesn't support customized java comp

Error message

Native output collector doesn't support customized java comparator {}

What it means

When a custom key comparator is configured (mapreduce.job.output.key.comparator.class / KEY_COMPARATOR), the native collector would have to run comparisons in C++; it cannot honor arbitrary Java RawComparator logic. init checks Platforms.define(comparatorClass) and, if no platform declares that comparator, logs and throws InvalidJobConfException naming the unsupported comparator class.

Source

Thrown at hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/java/org/apache/hadoop/mapred/nativetask/NativeMapOutputCollectorDelegator.java:92

  public void init(Context context) throws IOException, ClassNotFoundException {
    this.context = context;
    this.job = context.getJobConf();

    Platforms.init(job);

    if (job.getNumReduceTasks() == 0) {
      String message = "There is no reducer, no need to use native output collector";
      LOG.error(message);
      throw new InvalidJobConfException(message);
    }

    Class<?> comparatorClass = job.getClass(MRJobConfig.KEY_COMPARATOR, null,
        RawComparator.class);
    if (comparatorClass != null && !Platforms.define(comparatorClass)) {
      String message = "Native output collector doesn't support customized java comparator "
        + job.get(MRJobConfig.KEY_COMPARATOR);
      LOG.error(message);
      throw new InvalidJobConfException(message);
    }



    if (!QuickSort.class.getName().equals(job.get(Constants.MAP_SORT_CLASS))) {
      String message = "Native-Task doesn't support sort class " +
        job.get(Constants.MAP_SORT_CLASS);
      LOG.error(message);
      throw new InvalidJobConfException(message);
    }

    if (job.getBoolean(MRConfig.SHUFFLE_SSL_ENABLED_KEY, false) == true) {
      String message = "Native-Task doesn't support secure shuffle";
      LOG.error(message);
      throw new InvalidJobConfException(message);
    }

    final Class<?> keyCls = job.getMapOutputKeyClass();

View on GitHub (pinned to 2add963021)

Solutions

1. Remove the custom comparator and rely on the key type's natural WritableComparator ordering (adjust the key type instead, e.g. composite key with correct byte order)
2. Implement/enable a native platform that defines this comparator (Platforms.define returns true) and register it
3. Disable the native collector for this job: unset mapreduce.job.map.output.collector.class so the Java collector runs the custom comparator

Example fix

// before
job.setMapOutputKeyComparatorClass(MyCustomComparator.class);
conf.set("mapreduce.job.map.output.collector.class",
    "org.apache.hadoop.mapred.nativetask.NativeMapOutputCollectorDelegator");

// after: encode ordering into the key itself
public class MyKey extends BinaryComparable implements WritableComparable<MyKey> { /* fields ordered for default byte comparison */ }
// no setMapOutputKeyComparatorClass call

Class<?> cmp = job.getClass(MRJobConfig.KEY_COMPARATOR, null, RawComparator.class);
if (cmp != null && !Platforms.define(cmp)) {
  jobConf.unset("mapreduce.job.map.output.collector.class"); // custom comparator needs the Java collector
}

try {
  collector.init(context);
} catch (InvalidJobConfException e) {
  if (e.getMessage() != null && e.getMessage().contains("customized java comparator")) {
    // reschedule without native collector or with a natively supported comparator
  }
  throw e;
}

When it happens

Trigger: job.setMapOutputKeyComparatorClass(MyRawComparator.class) (or the equivalent conf key) combined with mapreduce.job.map.output.collector.class = NativeMapOutputCollectorDelegator, where MyRawComparator is not one of the comparator classes the configured native platform implements.

Common situations: Secondary-sort and custom-ordering jobs moved onto a cluster where the site config enables nativetask; comparators extending WritableComparator with custom compare() overrides that were never ported to a native platform.

Related errors

• Native-Task doesn't support sort class {}
• Native output collector doesn't support this key, this key i
• There is no reducer, no need to use native output collector
• Native-Task doesn't support secure shuffle
• Key type not supported. Cannot find serializer for {}

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/8952d134553e3991. Report an issue: GitHub.