{"record":{"id":"a0ea87a0cd322c03","repo":"apache/hadoop","slug":"mismatch-in-job-id-s-and-commit-job-id-s-a0ea87","errorCode":null,"errorMessage":"Mismatch in Job ID (%s) and commit job ID (%s)","messagePattern":"Mismatch in Job ID \\((.+?)\\) and commit job ID \\((.+?)\\)","errorType":"exception","errorClass":"PathCommitException","httpStatus":null,"severity":"error","filePath":"hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/commit/AbstractS3ACommitter.java","lineNumber":753,"sourceCode":"   * @throws IOException failure\n   */\n  private void loadAndCommit(\n      final CommitContext commitContext,\n      final ActiveCommit activeCommit,\n      final FileStatus status) throws IOException {\n\n    final Path path = status.getPath();\n    commitContext.switchToIOStatisticsContext();\n    try (DurationInfo ignored =\n             new DurationInfo(LOG,\n                 \"Loading and committing files in pendingset %s\", path)) {\n      PendingSet pendingSet = PersistentCommitData.load(\n          activeCommit.getSourceFS(),\n          status,\n          commitContext.getPendingSetSerializer());\n      String jobId = pendingSet.getJobId();\n      if (!StringUtils.isEmpty(jobId) && !getUUID().equals(jobId)) {\n        throw new PathCommitException(path,\n            String.format(\"Mismatch in Job ID (%s) and commit job ID (%s)\",\n                getUUID(), jobId));\n      }\n      TaskPool.foreach(pendingSet.getCommits())\n          .stopOnFailure()\n          .suppressExceptions(false)\n          .executeWith(commitContext.getInnerSubmitter())\n          .onFailure((commit, exception) ->\n              commitContext.abortSingleCommit(commit))\n          .abortWith(commitContext::abortSingleCommit)\n          .revertWith(commitContext::revertCommit)\n          .run(commit -> {\n            commitContext.commitOrFail(commit);\n            activeCommit.uploadCommitted(\n                commit.getDestinationKey(), commit.getLength());\n          });\n      activeCommit.pendingsetCommitted(pendingSet.getIOStatistics());\n    }","sourceCodeStart":735,"sourceCodeEnd":771,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/commit/AbstractS3ACommitter.java#L735-L771","documentation":"During job commit the committer loads each pendingset file and compares the jobId recorded in it with its own job UUID; a non-empty mismatch throws PathCommitException 'Mismatch in Job ID'. This guard stops one job from committing (or aborting) another job's pending multipart uploads, which could publish foreign data or delete a live job's output.","triggerScenarios":"calculateCommitActions loads a PendingSet under the job's commit directory whose getJobId() differs from getUUID(). The file was written by a different job run: stale _temporary/pending directories from a previous attempt sharing the output path, or two concurrent jobs colliding on the same paths.","commonSituations":"Rerunning a failed job into the same output directory without cleaning _temporary; Spark jobs without unique write UUIDs (pre-SPARK-33230) overwriting each other's working dirs; staging directory reuse between runs.","solutions":["Write to a fresh output path, or delete the destination's _temporary directory (and staging dirs) before rerunning a failed job","Ensure spark.sql.sources.writeJobUUID is unique per job (SPARK-33230+) so concurrent jobs cannot collide","Never reuse task attempt paths/IDs across job runs; let the framework generate new ones"],"exampleFix":"# before: rerun into a dirty output path\nspark-submit --conf spark.sql.sources.writeJobUUID=8f1c0c10-...  # reused/stale UUID\n\n# after: clean state, unique UUID per run\nhadoop fs -rm -r s3a://bucket/out/_temporary\nspark-submit  # Spark sets spark.sql.sources.writeJobUUID per job; do not pin it","handlingStrategy":"validation","validationCode":"// before submitting, ensure no foreign pendingsets sit in the output working dir\nPath tmp = new Path(outputPath, \"_temporary\");\nif (destFS.exists(tmp)) {\n  LOG.warn(\"Stale {} from a previous run; removing\", tmp);\n  destFS.delete(tmp, true);\n}","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Always write to a fresh output path or clean _temporary on rerun","Never pin spark.sql.sources.writeJobUUID across jobs; let Spark generate it per job","Treat a Job ID mismatch as evidence of path reuse, and audit job output directories"],"tags":["aws","s3a","committer","job-id","data-integrity"],"backgroundTag":"stale-commit-metadata","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-23T01:17:44.959Z"}