apache/hadoop · error · UnsupportedOperationException

Incompatible with LocalRunner

Error message

Incompatible with LocalRunner

What it means

YarnOutputFiles is the MapOutputFile implementation that YarnChild installs for tasks running on YARN (via MRConfig.TASK_LOCAL_OUTPUT_CLASS). On YARN, reduce inputs arrive over shuffle HTTP, so getLocalMapOutput-style handoff is meaningless; getInputFile(int mapId) exists only for the LocalJobRunner's in-JVM shuffle and always throws UnsupportedOperationException('Incompatible with LocalRunner'). Hitting it means local-runner code path (e.g. ReduceTask's local branch at ReduceTask.java:198, or a test at TestMapRed.java:310) executed against the YARN implementation -- a wiring or framework mismatch, never a runtime IO condition.

Source

Thrown at hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapred/YarnOutputFiles.java:198

   * @return path
   * @throws IOException
   */
  public Path getSpillIndexFileForWrite(int spillNumber, long size)
      throws IOException {
    return lDirAlloc.getLocalPathForWrite(
        String.format(SPILL_INDEX_FILE_PATTERN,
            conf.get(JobContext.TASK_ATTEMPT_ID), spillNumber), size, conf);
  }

  /**
   * Return a local reduce input file created earlier
   * 
   * @param mapId a map task id
   * @return path
   * @throws IOException 
   */
  public Path getInputFile(int mapId) throws IOException {
    throw new UnsupportedOperationException("Incompatible with LocalRunner");
  }

  /**
   * Create a local reduce input file name.
   * 
   * @param mapId a map task id
   * @param size the size of the file
   * @return path
   * @throws IOException
   */
  public Path getInputFileForWrite(org.apache.hadoop.mapreduce.TaskID mapId,
      long size) throws IOException {
    return lDirAlloc.getLocalPathForWrite(String.format(
        REDUCE_INPUT_FILE_FORMAT_STRING,
        getAttemptOutputDir().toString(), mapId.getId()),
        size, conf);
  }

View on GitHub (pinned to 2add963021)

Solutions

1. Let each framework wire its own implementation: framework=local uses the local runner's MapOutputFile, framework=yarn uses YarnOutputFiles -- remove any manual setClass(MRConfig.TASK_LOCAL_OUTPUT_CLASS, ...)
2. In code, branch on instanceof before calling local-only methods, or select the implementation from JobConf: conf.getClass(MRConfig.TASK_LOCAL_OUTPUT_CLASS, ...)
3. Align hadoop-mapreduce-client-* jar versions on the classpath so LocalJobRunner finds its own output-files class

Example fix

// before: local-runner code path calling the YARN implementation
Path in = mapOutputFile.getInputFile(mapId); // throws under YarnOutputFiles
// after: guard by implementation before the local-shuffle handoff
if (mapOutputFile instanceof YarnOutputFiles) {
  throw new IllegalStateException("local shuffle handoff not available on YARN");
}
Path in = mapOutputFile.getInputFile(mapId);

// narrow before calling local-runner-only MapOutputFile methods
private static boolean supportsLocalShuffleHandoff(MapOutputFile mof) {
  return !(mof instanceof org.apache.hadoop.mapred.YarnOutputFiles);
}

// usage
if (supportsLocalShuffleHandoff(mapOutputFile)) {
  Path reduceIn = mapOutputFile.getInputFile(mapId);
}

Catch UnsupportedOperationException around getInputFile and translate it into a configuration error ('local shuffle handoff used with YARN task output class') instead of letting it kill the task.

When it happens

Trigger: Running with mapreduce.framework.name=local (LocalJobRunner) while MRConfig.TASK_LOCAL_OUTPUT_CLASS is pinned to YarnOutputFiles; unit tests instantiating YarnOutputFiles directly and calling getInputFile; mixed-version jars where the local runner's own output-files class is absent so YarnOutputFiles resolves.

Common situations: Test suites moved from MiniMRCluster to LocalJobRunner with stale configuration; jobs that hardcode task output classes; downstream code copying Hadoop's internal local-shuffle path.

Related errors

• ShuffleProvider Service: {} was NOT found in the list of aux
• Use Token.renew instead
• "Mkdirs failed to create " + reduceIn.getParent().toString()
• "Couldn't rename " + mapOut
• "Couldn't rename " + mapOutIndex

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/32b5fa0082331a20. Report an issue: GitHub.