apache/hadoop · error · InvalidJobConfException

Native output collector doesn't support this key, this key i

Error message

Native output collector doesn't support this key, this key is not comparable in native: {}

What it means

Even when a serializer exists for the key class, the native collector must also compare keys in C++ to sort and spill. init calls Platforms.support(keyClassName, serializer, job); a false return means the key is serializable but not comparable by the native platform, and init throws InvalidJobConfException('...not comparable in native: <class>'). This separates 'cannot serialize' from 'cannot compare' failures.

Source

Thrown at hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/java/org/apache/hadoop/mapred/nativetask/NativeMapOutputCollectorDelegator.java:122

    if (job.getBoolean(MRConfig.SHUFFLE_SSL_ENABLED_KEY, false) == true) {
      String message = "Native-Task doesn't support secure shuffle";
      LOG.error(message);
      throw new InvalidJobConfException(message);
    }

    final Class<?> keyCls = job.getMapOutputKeyClass();
    try {
      @SuppressWarnings("rawtypes")
      final INativeSerializer serializer = NativeSerialization.getInstance().getSerializer(keyCls);
      if (null == serializer) {
        String message = "Key type not supported. Cannot find serializer for " + keyCls.getName();
        LOG.error(message);
        throw new InvalidJobConfException(message);
      } else if (!Platforms.support(keyCls.getName(), serializer, job)) {
        String message = "Native output collector doesn't support this key, " +
          "this key is not comparable in native: " + keyCls.getName();
        LOG.error(message);
        throw new InvalidJobConfException(message);
      }
    } catch (final IOException e) {
      String message = "Cannot find serializer for " + keyCls.getName();
      LOG.error(message);
      throw new IOException(message);
    }

    final boolean ret = NativeRuntime.isNativeLibraryLoaded();
    if (ret) {
      if (job.getBoolean(MRJobConfig.MAP_OUTPUT_COMPRESS, false)) {
        String codec = job.get(MRJobConfig.MAP_OUTPUT_COMPRESS_CODEC);
        if (!NativeRuntime.supportsCompressionCodec(codec.getBytes(StandardCharsets.UTF_8))) {
          String message = "Native output collector doesn't support compression codec " + codec;
          LOG.error(message);
          throw new InvalidJobConfException(message);
        }
      }
      NativeRuntime.configure(job);

View on GitHub (pinned to 2add963021)

Solutions

1. Switch the map output key to a fully supported type (Text, IntWritable, LongWritable, FloatWritable, DoubleWritable, BytesWritable) whose comparison the platform implements natively
2. Complete the custom platform: implement the native comparator for the key class so Platforms.support returns true
3. Disable the native collector for this job (unset mapreduce.job.map.output.collector.class)

INativeSerializer serializer = NativeSerialization.getInstance().getSerializer(keyCls);
if (serializer != null && !Platforms.support(keyCls.getName(), serializer, job)) {
  jobConf.unset("mapreduce.job.map.output.collector.class"); // key not comparable natively
}

try {
  collector.init(context);
} catch (InvalidJobConfException e) {
  if (e.getMessage() != null && e.getMessage().contains("not comparable in native")) {
    // pick a fully supported key type or fall back to the Java collector
  }
  throw e;
}

When it happens

Trigger: A map output key class has a registered native serializer but the active native platform has no comparator for it - e.g. a custom WritableComparable handled by a serialization-only platform - with the native collector enabled.

Common situations: Partially ported custom platforms that add a serializer but forget the native comparator; composite/custom WritableComparable keys on native-enabled clusters where only standard types compare natively.

Related errors

• Native output collector doesn't support customized java comp
• Native-Task doesn't support sort class {}
• Key type not supported. Cannot find serializer for {}
• Cannot find serializer for {}
• There is no reducer, no need to use native output collector

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/ce5706395f21b2a2. Report an issue: GitHub.