{"record":{"id":"5e2fdc3a7f8af17f","repo":"apache/hadoop","slug":"task-attempt-attemptid-has-a-self-generated-job","errorCode":null,"errorMessage":"Task attempt {attemptID} has a self-generated job UUID","messagePattern":"Task attempt (.+?) has a self-generated job UUID","errorType":"exception","errorClass":"PathCommitException","httpStatus":null,"severity":"error","filePath":"hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/commit/AbstractS3ACommitter.java","lineNumber":642,"sourceCode":"   */\n  @Override\n  public void setupTask(TaskAttemptContext context) throws IOException {\n    TaskAttemptID attemptID = context.getTaskAttemptID();\n\n    // update the context so that task IO in the same thread has\n    // the relevant values.\n    new AuditContextUpdater(context)\n        .updateCurrentAuditContext();\n\n    try (DurationInfo d = new DurationInfo(LOG, \"Setup Task %s\",\n        attemptID)) {\n      // reject attempts to set up the task where the output won't be\n      // picked up\n      if (!jobSetup\n          && getUUIDSource() == JobUUIDSource.GeneratedLocally) {\n        // on anything other than a test run, the context must not have been\n        // generated locally.\n        throw new PathCommitException(getOutputPath().toString(),\n            \"Task attempt \" + attemptID\n                + \" \" + E_SELF_GENERATED_JOB_UUID);\n      }\n      Path taskAttemptPath = getTaskAttemptPath(context);\n      FileSystem fs = taskAttemptPath.getFileSystem(getConf());\n      // delete that ta path if somehow it was there\n      fs.delete(taskAttemptPath, true);\n      // create an empty directory\n      fs.mkdirs(taskAttemptPath);\n    }\n  }\n\n  /**\n   * Get the task attempt path filesystem. This may not be the same as the\n   * final destination FS, and so may not be an S3A FS.\n   * @param context task attempt\n   * @return the filesystem\n   * @throws IOException failure to instantiate","sourceCodeStart":624,"sourceCodeEnd":660,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/commit/AbstractS3ACommitter.java#L624-L660","documentation":"S3A committers correlate task output with the job through a UUID that must originate at job setup (spark.sql.sources.writeJobUUID, or the YARN application attempt ID). At task setup, if the UUID source is GeneratedLocally the committer throws PathCommitException with E_SELF_GENERATED_JOB_UUID: a task that invented its own UUID would write pendingset files the job committer can never find, silently losing data, so setup fails instead.","triggerScenarios":"setupTask runs with jobSetup=false while getUUIDSource() == JobUUIDSource.GeneratedLocally. That combination means fs.s3a.committer.generate.uuid=true allowed local generation because the task's configuration lacked spark.sql.sources.writeJobUUID (e.g. Spark without SPARK-33230, or a pipeline building TaskAttemptContext from a stripped Configuration).","commonSituations":"Spark versions older than the SPARK-33230 fix that fail to propagate the write UUID into task commit contexts; custom MR pipelines copying job configs incompletely into task configs; manually enabling fs.s3a.committer.generate.uuid without ensuring UUID propagation.","solutions":["Use a Spark version with SPARK-33230 applied so spark.sql.sources.writeJobUUID reaches every task's commit context","Ensure the job-level Configuration (carrying the UUID) is propagated intact to task attempts; do not rebuild task contexts from a subset of properties","Set fs.s3a.committer.generate.uuid=false (the default) so this ambiguous state cannot arise, and fs.s3a.committer.require.uuid=true to fail at job setup with the clearer E_NO_SPARK_UUID message","Log spark.sql.sources.writeJobUUID from inside a task once to verify propagation before running at scale"],"exampleFix":"# before: tasks generate their own UUID -> task setup aborts\nspark.hadoop.fs.s3a.committer.generate.uuid=true\n\n# after: require the job UUID to be present and fail fast at job setup if absent\nspark.hadoop.fs.s3a.committer.generate.uuid=false\nspark.hadoop.fs.s3a.committer.require.uuid=true","handlingStrategy":"validation","validationCode":"String uuid = taskConf.getTrimmed(\"spark.sql.sources.writeJobUUID\", \"\");\nboolean generate = taskConf.getBoolean(\"fs.s3a.committer.generate.uuid\", false);\nif (uuid.isEmpty() && generate && !isJobSetup) {\n  throw new IOException(\"Task would self-generate a job UUID; fix UUID propagation first\");\n}","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Keep fs.s3a.committer.generate.uuid=false so tasks never invent UUIDs","Verify spark.sql.sources.writeJobUUID appears in a task's config with a debug log or dump","Use Spark with SPARK-33230 so the UUID propagates to task commit contexts"],"tags":["aws","s3a","committer","spark","uuid","configuration"],"backgroundTag":"missing-job-uuid","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}