{"record":{"id":"8952d134553e3991","repo":"apache/hadoop","slug":"native-output-collector-doesn-t-support-customized","errorCode":null,"errorMessage":"Native output collector doesn't support customized java comparator {}","messagePattern":"Native output collector doesn't support customized java comparator (.+?)","errorType":"exception","errorClass":"InvalidJobConfException","httpStatus":null,"severity":"error","filePath":"hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/java/org/apache/hadoop/mapred/nativetask/NativeMapOutputCollectorDelegator.java","lineNumber":92,"sourceCode":"  public void init(Context context) throws IOException, ClassNotFoundException {\n    this.context = context;\n    this.job = context.getJobConf();\n\n    Platforms.init(job);\n\n    if (job.getNumReduceTasks() == 0) {\n      String message = \"There is no reducer, no need to use native output collector\";\n      LOG.error(message);\n      throw new InvalidJobConfException(message);\n    }\n\n    Class<?> comparatorClass = job.getClass(MRJobConfig.KEY_COMPARATOR, null,\n        RawComparator.class);\n    if (comparatorClass != null && !Platforms.define(comparatorClass)) {\n      String message = \"Native output collector doesn't support customized java comparator \"\n        + job.get(MRJobConfig.KEY_COMPARATOR);\n      LOG.error(message);\n      throw new InvalidJobConfException(message);\n    }\n\n\n\n    if (!QuickSort.class.getName().equals(job.get(Constants.MAP_SORT_CLASS))) {\n      String message = \"Native-Task doesn't support sort class \" +\n        job.get(Constants.MAP_SORT_CLASS);\n      LOG.error(message);\n      throw new InvalidJobConfException(message);\n    }\n\n    if (job.getBoolean(MRConfig.SHUFFLE_SSL_ENABLED_KEY, false) == true) {\n      String message = \"Native-Task doesn't support secure shuffle\";\n      LOG.error(message);\n      throw new InvalidJobConfException(message);\n    }\n\n    final Class<?> keyCls = job.getMapOutputKeyClass();","sourceCodeStart":74,"sourceCodeEnd":110,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/java/org/apache/hadoop/mapred/nativetask/NativeMapOutputCollectorDelegator.java#L74-L110","documentation":"When a custom key comparator is configured (mapreduce.job.output.key.comparator.class / KEY_COMPARATOR), the native collector would have to run comparisons in C++; it cannot honor arbitrary Java RawComparator logic. init checks Platforms.define(comparatorClass) and, if no platform declares that comparator, logs and throws InvalidJobConfException naming the unsupported comparator class.","triggerScenarios":"job.setMapOutputKeyComparatorClass(MyRawComparator.class) (or the equivalent conf key) combined with mapreduce.job.map.output.collector.class = NativeMapOutputCollectorDelegator, where MyRawComparator is not one of the comparator classes the configured native platform implements.","commonSituations":"Secondary-sort and custom-ordering jobs moved onto a cluster where the site config enables nativetask; comparators extending WritableComparator with custom compare() overrides that were never ported to a native platform.","solutions":["Remove the custom comparator and rely on the key type's natural WritableComparator ordering (adjust the key type instead, e.g. composite key with correct byte order)","Implement/enable a native platform that defines this comparator (Platforms.define returns true) and register it","Disable the native collector for this job: unset mapreduce.job.map.output.collector.class so the Java collector runs the custom comparator"],"exampleFix":"// before\njob.setMapOutputKeyComparatorClass(MyCustomComparator.class);\nconf.set(\"mapreduce.job.map.output.collector.class\",\n    \"org.apache.hadoop.mapred.nativetask.NativeMapOutputCollectorDelegator\");\n\n// after: encode ordering into the key itself\npublic class MyKey extends BinaryComparable implements WritableComparable<MyKey> { /* fields ordered for default byte comparison */ }\n// no setMapOutputKeyComparatorClass call","handlingStrategy":"validation","validationCode":"Class<?> cmp = job.getClass(MRJobConfig.KEY_COMPARATOR, null, RawComparator.class);\nif (cmp != null && !Platforms.define(cmp)) {\n  jobConf.unset(\"mapreduce.job.map.output.collector.class\"); // custom comparator needs the Java collector\n}","typeGuard":null,"tryCatchPattern":"try {\n  collector.init(context);\n} catch (InvalidJobConfException e) {\n  if (e.getMessage() != null && e.getMessage().contains(\"customized java comparator\")) {\n    // reschedule without native collector or with a natively supported comparator\n  }\n  throw e;\n}","preventionTips":["Prefer encoding custom orderings into the key's byte layout over custom RawComparators when nativetask is in play","Check Platforms.define(comparatorClass) in a job-config validator before submission","Keep a list of natively supported comparators next to cluster onboarding docs"],"tags":["nativetask","comparator","sorting","job-configuration"],"backgroundTag":"custom-comparator-unsupported","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}