apache/hadoop · error · PathCommitException

Task attempt {attemptID} has a self-generated job UUID

Error message

Task attempt {attemptID} has a self-generated job UUID

What it means

S3A committers correlate task output with the job through a UUID that must originate at job setup (spark.sql.sources.writeJobUUID, or the YARN application attempt ID). At task setup, if the UUID source is GeneratedLocally the committer throws PathCommitException with E_SELF_GENERATED_JOB_UUID: a task that invented its own UUID would write pendingset files the job committer can never find, silently losing data, so setup fails instead.

Source

Thrown at hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/commit/AbstractS3ACommitter.java:642

   */
  @Override
  public void setupTask(TaskAttemptContext context) throws IOException {
    TaskAttemptID attemptID = context.getTaskAttemptID();

    // update the context so that task IO in the same thread has
    // the relevant values.
    new AuditContextUpdater(context)
        .updateCurrentAuditContext();

    try (DurationInfo d = new DurationInfo(LOG, "Setup Task %s",
        attemptID)) {
      // reject attempts to set up the task where the output won't be
      // picked up
      if (!jobSetup
          && getUUIDSource() == JobUUIDSource.GeneratedLocally) {
        // on anything other than a test run, the context must not have been
        // generated locally.
        throw new PathCommitException(getOutputPath().toString(),
            "Task attempt " + attemptID
                + " " + E_SELF_GENERATED_JOB_UUID);
      }
      Path taskAttemptPath = getTaskAttemptPath(context);
      FileSystem fs = taskAttemptPath.getFileSystem(getConf());
      // delete that ta path if somehow it was there
      fs.delete(taskAttemptPath, true);
      // create an empty directory
      fs.mkdirs(taskAttemptPath);
    }
  }

  /**
   * Get the task attempt path filesystem. This may not be the same as the
   * final destination FS, and so may not be an S3A FS.
   * @param context task attempt
   * @return the filesystem
   * @throws IOException failure to instantiate

View on GitHub (pinned to 2add963021)

Solutions

Use a Spark version with SPARK-33230 applied so spark.sql.sources.writeJobUUID reaches every task's commit context
Ensure the job-level Configuration (carrying the UUID) is propagated intact to task attempts; do not rebuild task contexts from a subset of properties
Set fs.s3a.committer.generate.uuid=false (the default) so this ambiguous state cannot arise, and fs.s3a.committer.require.uuid=true to fail at job setup with the clearer E_NO_SPARK_UUID message
Log spark.sql.sources.writeJobUUID from inside a task once to verify propagation before running at scale

Example fix

# before: tasks generate their own UUID -> task setup aborts
spark.hadoop.fs.s3a.committer.generate.uuid=true

# after: require the job UUID to be present and fail fast at job setup if absent
spark.hadoop.fs.s3a.committer.generate.uuid=false
spark.hadoop.fs.s3a.committer.require.uuid=true

Defensive patterns

Strategy: validation

Validate before calling

String uuid = taskConf.getTrimmed("spark.sql.sources.writeJobUUID", "");
boolean generate = taskConf.getBoolean("fs.s3a.committer.generate.uuid", false);
if (uuid.isEmpty() && generate && !isJobSetup) {
  throw new IOException("Task would self-generate a job UUID; fix UUID propagation first");
}

Prevention

Keep fs.s3a.committer.generate.uuid=false so tasks never invent UUIDs
Verify spark.sql.sources.writeJobUUID appears in a task's config with a debug log or dump
Use Spark with SPARK-33230 so the UUID propagates to task commit contexts

When it happens

Trigger: setupTask runs with jobSetup=false while getUUIDSource() == JobUUIDSource.GeneratedLocally. That combination means fs.s3a.committer.generate.uuid=true allowed local generation because the task's configuration lacked spark.sql.sources.writeJobUUID (e.g. Spark without SPARK-33230, or a pipeline building TaskAttemptContext from a stripped Configuration).

Common situations: Spark versions older than the SPARK-33230 fix that fail to propagate the write UUID into task commit contexts; custom MR pipelines copying job configs incompletely into task configs; manually enabling fs.s3a.committer.generate.uuid without ensuring UUID propagation.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/5e2fdc3a7f8af17f. Report an issue: GitHub.