{"record":{"id":"23528d38cca80014","repo":"apache/hadoop","slug":"native-task-doesn-t-support-secure-shuffle","errorCode":null,"errorMessage":"Native-Task doesn't support secure shuffle","messagePattern":"Native-Task doesn't support secure shuffle","errorType":"exception","errorClass":"InvalidJobConfException","httpStatus":null,"severity":"error","filePath":"hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/java/org/apache/hadoop/mapred/nativetask/NativeMapOutputCollectorDelegator.java","lineNumber":107,"sourceCode":"      String message = \"Native output collector doesn't support customized java comparator \"\n        + job.get(MRJobConfig.KEY_COMPARATOR);\n      LOG.error(message);\n      throw new InvalidJobConfException(message);\n    }\n\n\n\n    if (!QuickSort.class.getName().equals(job.get(Constants.MAP_SORT_CLASS))) {\n      String message = \"Native-Task doesn't support sort class \" +\n        job.get(Constants.MAP_SORT_CLASS);\n      LOG.error(message);\n      throw new InvalidJobConfException(message);\n    }\n\n    if (job.getBoolean(MRConfig.SHUFFLE_SSL_ENABLED_KEY, false) == true) {\n      String message = \"Native-Task doesn't support secure shuffle\";\n      LOG.error(message);\n      throw new InvalidJobConfException(message);\n    }\n\n    final Class<?> keyCls = job.getMapOutputKeyClass();\n    try {\n      @SuppressWarnings(\"rawtypes\")\n      final INativeSerializer serializer = NativeSerialization.getInstance().getSerializer(keyCls);\n      if (null == serializer) {\n        String message = \"Key type not supported. Cannot find serializer for \" + keyCls.getName();\n        LOG.error(message);\n        throw new InvalidJobConfException(message);\n      } else if (!Platforms.support(keyCls.getName(), serializer, job)) {\n        String message = \"Native output collector doesn't support this key, \" +\n          \"this key is not comparable in native: \" + keyCls.getName();\n        LOG.error(message);\n        throw new InvalidJobConfException(message);\n      }\n    } catch (final IOException e) {\n      String message = \"Cannot find serializer for \" + keyCls.getName();","sourceCodeStart":89,"sourceCodeEnd":125,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/java/org/apache/hadoop/mapred/nativetask/NativeMapOutputCollectorDelegator.java#L89-L125","documentation":"HTTPS (SSL) shuffle encrypts map-output transfer between nodes, a path the native collector's buffer handoff does not implement. init checks mapreduce.shuffle.ssl.enabled (MRConfig.SHUFFLE_SSL_ENABLED_KEY); when it is true, it throws InvalidJobConfException because the native task cannot serve encrypted shuffle traffic.","triggerScenarios":"mapreduce.shuffle.ssl.enabled = true in mapred-site.xml (or on the job) together with mapreduce.job.map.output.collector.class = NativeMapOutputCollectorDelegator; init of the delegator on a map task fails immediately.","commonSituations":"Security-hardened clusters where shuffle TLS is enforced globally, and a user opts a job into nativetask for throughput; enabling shuffle SSL site-wide after jobs were already configured for the native collector.","solutions":["Disable the native collector on this job (unset mapreduce.job.map.output.collector.class) so the Java collector supports the SSL shuffle path","If shuffle encryption is not mandatory, set mapreduce.shuffle.ssl.enabled back to false on the affected jobs/cluster","Prefer TLS on the shuffle HTTP layer via http.policy / ssl-enabled shuffle settings supported by the Java collector instead of mixing nativetask with encrypted shuffle"],"exampleFix":"# before\n<property><name>mapreduce.shuffle.ssl.enabled</name><value>true</value></property>\n<property><name>mapreduce.job.map.output.collector.class</name><value>org.apache.hadoop.mapred.nativetask.NativeMapOutputCollectorDelegator</value></property>\n\n# after\n<property><name>mapreduce.job.map.output.collector.class</name><value>org.apache.hadoop.mapred.MapTask$MapOutputBuffer</value></property>","handlingStrategy":"validation","validationCode":"boolean nativeCollector = \"org.apache.hadoop.mapred.nativetask.NativeMapOutputCollectorDelegator\"\n    .equals(jobConf.get(\"mapreduce.job.map.output.collector.class\"));\nif (nativeCollector && jobConf.getBoolean(\"mapreduce.shuffle.ssl.enabled\", false)) {\n  jobConf.unset(\"mapreduce.job.map.output.collector.class\"); // SSL shuffle requires the Java collector\n}","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Treat nativetask and encrypted shuffle as mutually exclusive in cluster policy checks","When enabling shuffle SSL cluster-wide, sweep job configs for native-collector settings"],"tags":["nativetask","security","shuffle","ssl","job-configuration"],"backgroundTag":"native-task-unsupported-job-config","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}