apache/hadoop · error · InvalidJobConfException

There is no reducer, no need to use native output collector

Error message

There is no reducer, no need to use native output collector

What it means

NativeMapOutputCollectorDelegator.init replaces the Java map-output collector with a native one that exists to sort/collect intermediate data for the shuffle. With job.getNumReduceTasks() == 0 there is no reduce phase and no shuffle to feed, so the delegator throws InvalidJobConfException telling you the native collector is unnecessary - it refuses to run rather than silently doing nothing.

Source

Thrown at hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/java/org/apache/hadoop/mapred/nativetask/NativeMapOutputCollectorDelegator.java:83

  }

  @Override
  public void flush() throws IOException, InterruptedException, ClassNotFoundException {
    handler.flush();
  }

  @SuppressWarnings("unchecked")
  @Override
  public void init(Context context) throws IOException, ClassNotFoundException {
    this.context = context;
    this.job = context.getJobConf();

    Platforms.init(job);

    if (job.getNumReduceTasks() == 0) {
      String message = "There is no reducer, no need to use native output collector";
      LOG.error(message);
      throw new InvalidJobConfException(message);
    }

    Class<?> comparatorClass = job.getClass(MRJobConfig.KEY_COMPARATOR, null,
        RawComparator.class);
    if (comparatorClass != null && !Platforms.define(comparatorClass)) {
      String message = "Native output collector doesn't support customized java comparator "
        + job.get(MRJobConfig.KEY_COMPARATOR);
      LOG.error(message);
      throw new InvalidJobConfException(message);
    }



    if (!QuickSort.class.getName().equals(job.get(Constants.MAP_SORT_CLASS))) {
      String message = "Native-Task doesn't support sort class " +
        job.get(Constants.MAP_SORT_CLASS);
      LOG.error(message);
      throw new InvalidJobConfException(message);

View on GitHub (pinned to 2add963021)

Solutions

1. Set the number of reduce tasks to at least 1 if a shuffle actually exists (job.setNumReduceTasks(1) or more)
2. If the job is genuinely map-only, unset mapreduce.job.map.output.collector.class (or set it back to org.apache.hadoop.mapred.MapTask$MapOutputBuffer) for this job
3. Fix job templates that blindly apply the native-collector setting regardless of job shape

Example fix

// before
conf.set("mapreduce.job.map.output.collector.class",
    "org.apache.hadoop.mapred.nativetask.NativeMapOutputCollectorDelegator");
job.setNumReduceTasks(0);

// after (map-only job)
conf.unset("mapreduce.job.map.output.collector.class"); // default Java collector
job.setNumReduceTasks(0);

if (jobConf.getInt(MRJobConfig.NUM_REDUCES, 1) == 0) {
  jobConf.unset("mapreduce.job.map.output.collector.class"); // native collector is pointless map-only
}

When it happens

Trigger: Enabling the native collector via mapreduce.job.map.output.collector.class = org.apache.hadoop.mapred.nativetask.NativeMapOutputCollectorDelegator while the job is map-only (numReduceTasks = 0, e.g. job.setNumReduceTasks(0) or a default of 0).

Common situations: A shared site-level config turns on the native collector for all jobs, and someone submits a map-only job (pure transformations, bulk loads, DistCp-like workloads); jobs where a reducer was removed late in development but the collector setting stayed.

Related errors

• Output directory not set.
• SequenceFileAsBinaryOutputFormat doesn't support Record Comp
• Native output collector doesn't support customized java comp
• Native-Task doesn't support sort class {}
• Native-Task doesn't support secure shuffle

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/ef251099dbfcea62. Report an issue: GitHub.