{"record":{"id":"647137980ccd8acc","repo":"apache/hadoop","slug":"e-no-spark-uuid","errorCode":"E_NO_SPARK_UUID","errorMessage":"Job/task context does not contain a unique ID in spark.sql.sources.writeJobUUID","messagePattern":"Job/task context does not contain a unique ID in spark\\.sql\\.sources\\.writeJobUUID","errorType":"exception","errorClass":"PathCommitException","httpStatus":null,"severity":"error","filePath":"hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/commit/AbstractS3ACommitter.java","lineNumber":1386,"sourceCode":"    String jobUUID = conf.getTrimmed(FS_S3A_COMMITTER_UUID, \"\");\n\n    if (!jobUUID.isEmpty()) {\n      return Pair.of(jobUUID, JobUUIDSource.CommitterUUIDProperty);\n    }\n    // there is no job UUID.\n    // look for one from spark\n    jobUUID = conf.getTrimmed(SPARK_WRITE_UUID, \"\");\n    if (!jobUUID.isEmpty()) {\n      return Pair.of(jobUUID, JobUUIDSource.SparkWriteUUID);\n    }\n\n    // there is no UUID configuration in the job/task config\n\n    // Check the job hasn't declared a requirement for the UUID.\n    // This allows or fail-fast validation of Spark behavior.\n    if (conf.getBoolean(FS_S3A_COMMITTER_REQUIRE_UUID,\n        DEFAULT_S3A_COMMITTER_REQUIRE_UUID)) {\n      throw new PathCommitException(\"\", E_NO_SPARK_UUID);\n    }\n\n    // see if the job can generate a random UUI`\n    if (conf.getBoolean(FS_S3A_COMMITTER_GENERATE_UUID,\n        DEFAULT_S3A_COMMITTER_GENERATE_UUID)) {\n      // generate a random UUID. This is OK for a job, for a task\n      // it means that the data may not get picked up.\n      String newId = UUID.randomUUID().toString();\n      LOG.warn(\"No job ID in configuration; generating a random ID: {}\",\n          newId);\n      return Pair.of(newId, JobUUIDSource.GeneratedLocally);\n    }\n    // if no other option was supplied, return the job ID.\n    // This is exactly what MR jobs expect, but is not what\n    // Spark jobs can do as there is a risk of jobID collision.\n    return Pair.of(jobId.toString(), JobUUIDSource.JobID);\n  }\n","sourceCodeStart":1368,"sourceCodeEnd":1404,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/commit/AbstractS3ACommitter.java#L1368-L1404","documentation":"buildJobUUID looks for spark.sql.sources.writeJobUUID in the job/task configuration; if absent and fs.s3a.committer.require.uuid=true, it throws PathCommitException E_NO_SPARK_UUID immediately. That flag exists to fail fast (at job setup) when Spark does not propagate the unique write UUID into commit contexts, as verified by SPARK-33230 - without the UUID, task output cannot be reliably correlated with the job.","triggerScenarios":"conf.getBoolean(\"fs.s3a.committer.require.uuid\", false) is true and conf.getTrimmed(\"spark.sql.sources.writeJobUUID\", \"\") was empty. The property is documented as 'MUST ONLY BE SET WITH SPARK JOBS'; setting it for plain MR or streaming jobs guarantees this exception.","commonSituations":"Enabling fs.s3a.committer.require.uuid cluster-wide while running non-Spark engines; using an older Spark that never sets spark.sql.sources.writeJobUUID; stripping Spark configuration when constructing job contexts.","solutions":["Upgrade Spark to a release containing SPARK-33230 so spark.sql.sources.writeJobUUID is set and propagated","Only set fs.s3a.committer.require.uuid on Spark jobs; unset it for MR/Hive/other engines","If upgrading is not possible, leave fs.s3a.committer.require.uuid=false (default) and rely on the YARN application attempt ID fallback"],"exampleFix":"# before: required UUID but Spark too old to provide it\nspark.hadoop.fs.s3a.committer.require.uuid=true  # on Spark 2.x without SPARK-33230\n\n# after\nspark.hadoop.fs.s3a.committer.require.uuid=false  # until Spark is upgraded","handlingStrategy":"validation","validationCode":"boolean sparkJob = jobConf.get(\"spark.yarn.app.id\", null) != null\n    || jobConf.getTrimmed(\"spark.sql.sources.writeJobUUID\", \"\").isEmpty() == false;\nif (jobConf.getBoolean(\"fs.s3a.committer.require.uuid\", false) && !sparkJob) {\n  jobConf.setBoolean(\"fs.s3a.committer.require.uuid\", false); // non-Spark engine\n}\nif (sparkJob && jobConf.getTrimmed(\"spark.sql.sources.writeJobUUID\", \"\").isEmpty()) {\n  throw new IOException(\"Spark job lacks spark.sql.sources.writeJobUUID (SPARK-33230 missing)\");\n}","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Set fs.s3a.committer.require.uuid only on Spark jobs","Upgrade Spark to a SPARK-33230-bearing release before enabling it","Fail fast at submission by checking the UUID key yourself"],"tags":["aws","s3a","committer","spark","uuid","fail-fast"],"backgroundTag":"missing-job-uuid","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-23T01:17:44.959Z"}