apache/hadoop · error · PathCommitException

E_NO_SPARK_UUID

E_NO_SPARK_UUID

Error message

Job/task context does not contain a unique ID in spark.sql.sources.writeJobUUID

What it means

buildJobUUID looks for spark.sql.sources.writeJobUUID in the job/task configuration; if absent and fs.s3a.committer.require.uuid=true, it throws PathCommitException E_NO_SPARK_UUID immediately. That flag exists to fail fast (at job setup) when Spark does not propagate the unique write UUID into commit contexts, as verified by SPARK-33230 - without the UUID, task output cannot be reliably correlated with the job.

Source

Thrown at hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/commit/AbstractS3ACommitter.java:1386

    String jobUUID = conf.getTrimmed(FS_S3A_COMMITTER_UUID, "");

    if (!jobUUID.isEmpty()) {
      return Pair.of(jobUUID, JobUUIDSource.CommitterUUIDProperty);
    }
    // there is no job UUID.
    // look for one from spark
    jobUUID = conf.getTrimmed(SPARK_WRITE_UUID, "");
    if (!jobUUID.isEmpty()) {
      return Pair.of(jobUUID, JobUUIDSource.SparkWriteUUID);
    }

    // there is no UUID configuration in the job/task config

    // Check the job hasn't declared a requirement for the UUID.
    // This allows or fail-fast validation of Spark behavior.
    if (conf.getBoolean(FS_S3A_COMMITTER_REQUIRE_UUID,
        DEFAULT_S3A_COMMITTER_REQUIRE_UUID)) {
      throw new PathCommitException("", E_NO_SPARK_UUID);
    }

    // see if the job can generate a random UUI`
    if (conf.getBoolean(FS_S3A_COMMITTER_GENERATE_UUID,
        DEFAULT_S3A_COMMITTER_GENERATE_UUID)) {
      // generate a random UUID. This is OK for a job, for a task
      // it means that the data may not get picked up.
      String newId = UUID.randomUUID().toString();
      LOG.warn("No job ID in configuration; generating a random ID: {}",
          newId);
      return Pair.of(newId, JobUUIDSource.GeneratedLocally);
    }
    // if no other option was supplied, return the job ID.
    // This is exactly what MR jobs expect, but is not what
    // Spark jobs can do as there is a risk of jobID collision.
    return Pair.of(jobId.toString(), JobUUIDSource.JobID);
  }

View on GitHub (pinned to 2add963021)

Solutions

1. Upgrade Spark to a release containing SPARK-33230 so spark.sql.sources.writeJobUUID is set and propagated
2. Only set fs.s3a.committer.require.uuid on Spark jobs; unset it for MR/Hive/other engines
3. If upgrading is not possible, leave fs.s3a.committer.require.uuid=false (default) and rely on the YARN application attempt ID fallback

Example fix

# before: required UUID but Spark too old to provide it
spark.hadoop.fs.s3a.committer.require.uuid=true  # on Spark 2.x without SPARK-33230

# after
spark.hadoop.fs.s3a.committer.require.uuid=false  # until Spark is upgraded

boolean sparkJob = jobConf.get("spark.yarn.app.id", null) != null
    || jobConf.getTrimmed("spark.sql.sources.writeJobUUID", "").isEmpty() == false;
if (jobConf.getBoolean("fs.s3a.committer.require.uuid", false) && !sparkJob) {
  jobConf.setBoolean("fs.s3a.committer.require.uuid", false); // non-Spark engine
}
if (sparkJob && jobConf.getTrimmed("spark.sql.sources.writeJobUUID", "").isEmpty()) {
  throw new IOException("Spark job lacks spark.sql.sources.writeJobUUID (SPARK-33230 missing)");
}

When it happens

Trigger: conf.getBoolean("fs.s3a.committer.require.uuid", false) is true and conf.getTrimmed("spark.sql.sources.writeJobUUID", "") was empty. The property is documented as 'MUST ONLY BE SET WITH SPARK JOBS'; setting it for plain MR or streaming jobs guarantees this exception.

Common situations: Enabling fs.s3a.committer.require.uuid cluster-wide while running non-Spark engines; using an older Spark that never sets spark.sql.sources.writeJobUUID; stripping Spark configuration when constructing job contexts.

Related errors

• Task attempt {attemptID} has a self-generated job UUID
• Multipart uploads are disabled for the FileSystem, the commi
• Unable to recover task %s
• Mismatch in Job ID (%s) and commit job ID (%s)
• Filesystem not supported by this committer

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/647137980ccd8acc. Report an issue: GitHub.