apache/hadoop · error · InvalidJobConfException

Native-Task doesn't support secure shuffle

Error message

Native-Task doesn't support secure shuffle

What it means

HTTPS (SSL) shuffle encrypts map-output transfer between nodes, a path the native collector's buffer handoff does not implement. init checks mapreduce.shuffle.ssl.enabled (MRConfig.SHUFFLE_SSL_ENABLED_KEY); when it is true, it throws InvalidJobConfException because the native task cannot serve encrypted shuffle traffic.

Source

Thrown at hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/java/org/apache/hadoop/mapred/nativetask/NativeMapOutputCollectorDelegator.java:107

      String message = "Native output collector doesn't support customized java comparator "
        + job.get(MRJobConfig.KEY_COMPARATOR);
      LOG.error(message);
      throw new InvalidJobConfException(message);
    }



    if (!QuickSort.class.getName().equals(job.get(Constants.MAP_SORT_CLASS))) {
      String message = "Native-Task doesn't support sort class " +
        job.get(Constants.MAP_SORT_CLASS);
      LOG.error(message);
      throw new InvalidJobConfException(message);
    }

    if (job.getBoolean(MRConfig.SHUFFLE_SSL_ENABLED_KEY, false) == true) {
      String message = "Native-Task doesn't support secure shuffle";
      LOG.error(message);
      throw new InvalidJobConfException(message);
    }

    final Class<?> keyCls = job.getMapOutputKeyClass();
    try {
      @SuppressWarnings("rawtypes")
      final INativeSerializer serializer = NativeSerialization.getInstance().getSerializer(keyCls);
      if (null == serializer) {
        String message = "Key type not supported. Cannot find serializer for " + keyCls.getName();
        LOG.error(message);
        throw new InvalidJobConfException(message);
      } else if (!Platforms.support(keyCls.getName(), serializer, job)) {
        String message = "Native output collector doesn't support this key, " +
          "this key is not comparable in native: " + keyCls.getName();
        LOG.error(message);
        throw new InvalidJobConfException(message);
      }
    } catch (final IOException e) {
      String message = "Cannot find serializer for " + keyCls.getName();

View on GitHub (pinned to 2add963021)

Solutions

1. Disable the native collector on this job (unset mapreduce.job.map.output.collector.class) so the Java collector supports the SSL shuffle path
2. If shuffle encryption is not mandatory, set mapreduce.shuffle.ssl.enabled back to false on the affected jobs/cluster
3. Prefer TLS on the shuffle HTTP layer via http.policy / ssl-enabled shuffle settings supported by the Java collector instead of mixing nativetask with encrypted shuffle

Example fix

# before
<property><name>mapreduce.shuffle.ssl.enabled</name><value>true</value></property>
<property><name>mapreduce.job.map.output.collector.class</name><value>org.apache.hadoop.mapred.nativetask.NativeMapOutputCollectorDelegator</value></property>

# after
<property><name>mapreduce.job.map.output.collector.class</name><value>org.apache.hadoop.mapred.MapTask$MapOutputBuffer</value></property>

boolean nativeCollector = "org.apache.hadoop.mapred.nativetask.NativeMapOutputCollectorDelegator"
    .equals(jobConf.get("mapreduce.job.map.output.collector.class"));
if (nativeCollector && jobConf.getBoolean("mapreduce.shuffle.ssl.enabled", false)) {
  jobConf.unset("mapreduce.job.map.output.collector.class"); // SSL shuffle requires the Java collector
}

When it happens

Trigger: mapreduce.shuffle.ssl.enabled = true in mapred-site.xml (or on the job) together with mapreduce.job.map.output.collector.class = NativeMapOutputCollectorDelegator; init of the delegator on a map task fails immediately.

Common situations: Security-hardened clusters where shuffle TLS is enforced globally, and a user opts a job into nativetask for throughput; enabling shuffle SSL site-wide after jobs were already configured for the native collector.

Related errors

• Verification of the hashReply failed
• security validation of TT Map output failed
• There is no reducer, no need to use native output collector
• Native output collector doesn't support customized java comp
• Native-Task doesn't support sort class {}

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/23528d38cca80014. Report an issue: GitHub.