{"record":{"id":"ef251099dbfcea62","repo":"apache/hadoop","slug":"there-is-no-reducer-no-need-to-use-native-output","errorCode":null,"errorMessage":"There is no reducer, no need to use native output collector","messagePattern":"There is no reducer, no need to use native output collector","errorType":"exception","errorClass":"InvalidJobConfException","httpStatus":null,"severity":"error","filePath":"hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/java/org/apache/hadoop/mapred/nativetask/NativeMapOutputCollectorDelegator.java","lineNumber":83,"sourceCode":"  }\n\n  @Override\n  public void flush() throws IOException, InterruptedException, ClassNotFoundException {\n    handler.flush();\n  }\n\n  @SuppressWarnings(\"unchecked\")\n  @Override\n  public void init(Context context) throws IOException, ClassNotFoundException {\n    this.context = context;\n    this.job = context.getJobConf();\n\n    Platforms.init(job);\n\n    if (job.getNumReduceTasks() == 0) {\n      String message = \"There is no reducer, no need to use native output collector\";\n      LOG.error(message);\n      throw new InvalidJobConfException(message);\n    }\n\n    Class<?> comparatorClass = job.getClass(MRJobConfig.KEY_COMPARATOR, null,\n        RawComparator.class);\n    if (comparatorClass != null && !Platforms.define(comparatorClass)) {\n      String message = \"Native output collector doesn't support customized java comparator \"\n        + job.get(MRJobConfig.KEY_COMPARATOR);\n      LOG.error(message);\n      throw new InvalidJobConfException(message);\n    }\n\n\n\n    if (!QuickSort.class.getName().equals(job.get(Constants.MAP_SORT_CLASS))) {\n      String message = \"Native-Task doesn't support sort class \" +\n        job.get(Constants.MAP_SORT_CLASS);\n      LOG.error(message);\n      throw new InvalidJobConfException(message);","sourceCodeStart":65,"sourceCodeEnd":101,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/java/org/apache/hadoop/mapred/nativetask/NativeMapOutputCollectorDelegator.java#L65-L101","documentation":"NativeMapOutputCollectorDelegator.init replaces the Java map-output collector with a native one that exists to sort/collect intermediate data for the shuffle. With job.getNumReduceTasks() == 0 there is no reduce phase and no shuffle to feed, so the delegator throws InvalidJobConfException telling you the native collector is unnecessary - it refuses to run rather than silently doing nothing.","triggerScenarios":"Enabling the native collector via mapreduce.job.map.output.collector.class = org.apache.hadoop.mapred.nativetask.NativeMapOutputCollectorDelegator while the job is map-only (numReduceTasks = 0, e.g. job.setNumReduceTasks(0) or a default of 0).","commonSituations":"A shared site-level config turns on the native collector for all jobs, and someone submits a map-only job (pure transformations, bulk loads, DistCp-like workloads); jobs where a reducer was removed late in development but the collector setting stayed.","solutions":["Set the number of reduce tasks to at least 1 if a shuffle actually exists (job.setNumReduceTasks(1) or more)","If the job is genuinely map-only, unset mapreduce.job.map.output.collector.class (or set it back to org.apache.hadoop.mapred.MapTask$MapOutputBuffer) for this job","Fix job templates that blindly apply the native-collector setting regardless of job shape"],"exampleFix":"// before\nconf.set(\"mapreduce.job.map.output.collector.class\",\n    \"org.apache.hadoop.mapred.nativetask.NativeMapOutputCollectorDelegator\");\njob.setNumReduceTasks(0);\n\n// after (map-only job)\nconf.unset(\"mapreduce.job.map.output.collector.class\"); // default Java collector\njob.setNumReduceTasks(0);","handlingStrategy":"validation","validationCode":"if (jobConf.getInt(MRJobConfig.NUM_REDUCES, 1) == 0) {\n  jobConf.unset(\"mapreduce.job.map.output.collector.class\"); // native collector is pointless map-only\n}","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Apply the native-collector setting only in job templates that actually run reducers","Fail fast in job drivers: assert numReduceTasks > 0 before enabling NativeMapOutputCollectorDelegator","Remember the native collector only accelerates the map->reduce shuffle path"],"tags":["nativetask","mapreduce","job-configuration","map-only"],"backgroundTag":"invalid-job-configuration","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-23T01:17:44.959Z"}