apache/hadoop · error · UnsupportedOperationException

Unable to recover task %s, output: %s

Error message

Unable to recover task %s, output: %s

What it means

The mapreduce-API TOS staging committer (org.apache.hadoop.fs.tosfs.commit.Committer) deliberately does not implement task-output recovery: recoverTask() unconditionally throws UnsupportedOperationException. The trap is that the committer inherits the default OutputCommitter.isRecoverySupported() == true, so the MR framework believes recovery works and calls the method, which then kills the job.

Source

Thrown at hadoop-cloud-storage-project/hadoop-tos/src/main/java/org/apache/hadoop/fs/tosfs/commit/Committer.java:441

            } catch (FileNotFoundException e) {
              LOG.debug("Listed file already deleted: {}", f);
            } catch (IOException e) {
              throw new UncheckedIOException(e);
            } finally {
              final FileStatus pendingFile = f;
              CommonUtils.runQuietly(() -> destFs.delete(pendingFile.getPath(), false));
            }
          });
    } finally {
      CommonUtils.runQuietly(() -> destFs.delete(taskAttemptBasePath, true));
    }
  }

  @Override
  public void recoverTask(TaskAttemptContext context) {
    checkJobId(context);
    String taskId = context.getTaskAttemptID().toString();
    throw new UnsupportedOperationException(
        String.format("Unable to recover task %s, output: %s", taskId, outputPath));
  }

  private int commitThreads() {
    return conf.getInt(COMMITTER_THREADS, DEFAULT_COMMITTER_THREADS);
  }

  private void cleanup(ExecutorService pool, boolean suppress) {
    LOG.info("Cleanup the job by abort the multipart uploads and clean staging dir, suppress {}",
        suppress);
    try {
      Path jobOutput = getOutputPath();
      Iterable<MultipartUpload> pending = storage.listUploads(
          ObjectUtils.pathToKey(CommitUtils.magicJobPath(jobId, jobOutput), true));
      Tasks.foreach(pending)
          .executeWith(pool)
          .suppressFailureWhenFinished()
          .run(u -> storage.abortMultipartUpload(u.key(), u.uploadId()));

View on GitHub (pinned to 2add963021)

Solutions

1. Set yarn.app.mapreduce.am.job.recovery.enable=false in mapred-site.xml (or on the job configuration) so the framework never attempts task recovery with this committer
2. Subclass the TOS Committer, override isRecoverySupported() to return false, and register the subclass via the output-committer factory so the framework skips recovery cleanly
3. Re-run the job from scratch: abortJob will clean any pending uploads/staging state left behind, so a fresh run is safe
4. If recovery is a hard requirement, implement recoverTask in a custom committer that remaps the previous attempt's pending-set files instead of throwing

Example fix

// before: committer inherits isRecoverySupported()==true, framework calls recoverTask(), job dies
public class MyCommitter extends org.apache.hadoop.fs.tosfs.commit.Committer {
  // recoverTask() inherited -> throws UnsupportedOperationException on AM restart
}
// after: declare recovery unsupported so the MR framework never calls recoverTask
public class MyCommitter extends org.apache.hadoop.fs.tosfs.commit.Committer {
  @Override
  public boolean isRecoverySupported() {
    return false;
  }
}

// before submitting a job whose output goes through the TOS committer:
Configuration conf = job.getConfiguration();
String amRecovery = "yarn.app.mapreduce.am.job.recovery.enable"; // MRJobConfig.MR_AM_JOB_RECOVERY_ENABLE
if (conf.getBoolean(amRecovery, true)) {
  conf.setBoolean(amRecovery, false); // TOS committer cannot recover tasks
}

try {
  committer.recoverTask(context);
} catch (UnsupportedOperationException e) {
  throw new IOException("TOS committer does not support task recovery; "
      + "disable yarn.app.mapreduce.am.job.recovery.enable and re-run the job", e);
}

When it happens

Trigger: The MR ApplicationMaster restarts (YARN RM restart, AM preemption, node loss) while yarn.app.mapreduce.am.job.recovery.enable is true (the default). If a task attempt had already committed output before the restart, the restarted AM invokes committer.recoverTask(context) for the old attempt and this UnsupportedOperationException is thrown.

Common situations: Jobs writing to tos:// URIs through the TOS FileOutputCommitter experiencing an AM restart; clusters with YARN RM recovery enabled; users porting jobs from FileOutputCommitter (which does implement recoverTask) to the TOS committer and assuming the same recovery semantics.

Related errors

• Invalid state of the job for cleanup. State found " + jobRun
• Only 1 or 2 algorithm version is supported
• Failed to delete {}
• Failed to rename {} to {}
• Failed to create {clazz}:{e}

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/98d77be9a59dd7b2. Report an issue: GitHub.