apache/hadoop · error · PathCommitException

Unable to recover task %s

Error message

Unable to recover task %s

What it means

OutputCommitter.recoverTask is invoked by the MapReduce application master during job recovery to reuse a previously committed task's output. S3A committers never support this: at recovery time the task's data is still pending multipart uploads, not committed files, so AbstractS3ACommitter.recoverTask logs a warning and always throws PathCommitException to fail the recovery attempt rather than produce wrong results.

Source

Thrown at hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/commit/AbstractS3ACommitter.java:465

   * to be configured to support magic paths where the output isn't immediately
   * visible. If the committer returns true, then committer setup will
   * fail if the FS doesn't have the capability.
   * Base implementation returns false.
   * @return what the requirements of the committer are of the filesystem.
   */
  protected boolean requiresDelayedCommitOutputInFileSystem() {
    return false;
  }

  /**
   * Task recovery considered Unsupported: Warn and fail.
   * @param taskContext Context of the task whose output is being recovered
   * @throws IOException always.
   */
  @Override
  public void recoverTask(TaskAttemptContext taskContext) throws IOException {
    LOG.warn("Cannot recover task {}", taskContext.getTaskAttemptID());
    throw new PathCommitException(outputPath,
        String.format("Unable to recover task %s",
            taskContext.getTaskAttemptID()));
  }

  /**
   * if the job requires a success marker on a successful job,
   * create the file {@link CommitConstants#_SUCCESS}.
   *
   * While the classic committers create a 0-byte file, the S3A committers
   * PUT up a the contents of a {@link SuccessData} file.
   * @param commitContext commit context
   * @param pending the pending commits
   *
   * @return the success data, even if the marker wasn't created
   *
   * @throws IOException IO failure
   */
  protected SuccessData maybeCreateSuccessMarkerFromCommits(

View on GitHub (pinned to 2add963021)

Solutions

1. Rerun the job with AM recovery disabled for that job: set yarn.app.mapreduce.am.job.recovery.enable=false so tasks restart cleanly instead of recovering
2. Before the rerun, clean the output's _temporary (and staging) directories left by the failed attempt so stale pendingsets do not surface later
3. Reduce AM restart churn (memory/queue settings) if failovers are frequent

Example fix

# before
spark-submit --conf spark.hadoop.yarn.app.mapreduce.am.job.recovery.enable=true ...

# after
spark-submit --conf spark.hadoop.yarn.app.mapreduce.am.job.recovery.enable=false \
  --conf spark.hadoop.mapreduce.fileoutputcommitter.marksuccessfuljobs=true ...

// detect the recovery path before it hits the committer:
if (jobContext.getConfiguration().getBoolean(
        "yarn.app.mapreduce.am.job.recovery.enable", true)
    && fsUri.getScheme().equals("s3a")) {
  job.getConfiguration().setBoolean("yarn.app.mapreduce.am.job.recovery.enable", false);
}

try {
  committer.recoverTask(taskContext);
} catch (PathCommitException e) {
  // recovery is fundamentally unsupported on S3A: restart the job cleanly
  LOG.warn("S3A committer cannot recover tasks; rerunning job with recovery off", e);
  rerunJobWithRecoveryDisabled(outputPath);
}

When it happens

Trigger: The MR AM restarts (failure or preemption), yarn.app.mapreduce.am.job.recovery.enable is true (default), and recovery replays a task attempt whose committer already wrote commit metadata, calling recoverTask on the S3A committer.

Common situations: Long Spark-on-YARN or MR jobs over S3A surviving an AM failover; clusters with aggressive AM restarts; jobs using staging or directory committers where task outputs live under _temporary until job commit.

Related errors

• Multipart uploads are disabled for the FileSystem, the commi
• Task attempt {attemptID} has a self-generated job UUID
• Mismatch in Job ID (%s) and commit job ID (%s)
• E_NO_SPARK_UUID
• Filesystem not supported by this committer

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/18c2aadd3501c15c. Report an issue: GitHub.