{"record":{"id":"2288adfce913ea9c","repo":"apache/hadoop","slug":"mismatch-in-job-id-s-and-commit-job-id-s","errorCode":null,"errorMessage":"Mismatch in Job ID (%s) and commit job ID (%s)","messagePattern":"Mismatch in Job ID \\((.+?)\\) and commit job ID \\((.+?)\\)","errorType":"exception","errorClass":"IllegalStateException","httpStatus":null,"severity":"critical","filePath":"hadoop-cloud-storage-project/hadoop-tos/src/main/java/org/apache/hadoop/fs/tosfs/commit/Committer.java","lineNumber":295,"sourceCode":"   */\n  private void loadAndRevert(ExecutorService pool, FileStatus pendingSetFile) {\n    PendingSet pendingSet = PendingSet.deserialize(destFs, pendingSetFile);\n    Tasks.foreach(pendingSet.commits())\n        .suppressFailureWhenFinished()\n        .executeWith(pool)\n        .run(ops::revert);\n  }\n\n  /**\n   * Load {@link PendingSet} from file and commit those {@link Pending} commits.\n   */\n  private void loadAndCommit(CommitContext commitCtxt, ExecutorService pool,\n      FileStatus pendingSetFile) {\n    PendingSet pendingSet = PendingSet.deserialize(destFs, pendingSetFile);\n    // Verify that whether the job id is matched.\n    String jobID = pendingSet.jobId();\n    if (!StringUtils.isNoneEmpty(jobID) && !Objects.equals(jobID, jobId())) {\n      throw new IllegalStateException(\n          String.format(\"Mismatch in Job ID (%s) and commit job ID (%s)\", jobId(), jobID));\n    }\n\n    Tasks.foreach(pendingSet.commits())\n        .stopOnFailure()\n        .throwFailureWhenFinished()\n        .executeWith(pool)\n        .onFailure((pending, exception) -> ops.abort(pending))\n        .abortWith(ops::abort)\n        .revertWith(ops::revert)\n        .run(pending -> {\n          ops.commit(pending);\n          commitCtxt.addDestKey(pending.destKey());\n        });\n  }\n\n  @Override\n  public void abortJob(JobContext context, JobStatus.State state) {","sourceCodeStart":277,"sourceCodeEnd":313,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-cloud-storage-project/hadoop-tos/src/main/java/org/apache/hadoop/fs/tosfs/commit/Committer.java#L277-L313","documentation":"During commitJob, loadAndCommit() deserializes each pendingset file and compares the serialized job id against the committer's job id, throwing IllegalStateException on mismatch. Note the guard in this code region reads !StringUtils.isNoneEmpty(jobID) && !Objects.equals(jobID, jobId()): it fires for an empty serialized job id, and a non-empty mismatched job id actually passes -- treat that inversion as a known defect when diagnosing. The intended condition: the pendingset on disk belongs to a different job than the one committing.","triggerScenarios":"commitJob reading pendingset-*.json artifacts in the destination/staging area that were written by a different (earlier) job; an empty jobId() in the serialized PendingSet (which always triggers, since '' != current job id); reusing an output or magic/staging directory across job submissions.","commonSituations":"Re-submitting a job into the same output dir without cleaning previous commit artifacts; two jobs sharing mapreduce.output.fileoutputformat.outputdir or the same magic directory; app- vs job-attempt id mixups producing empty serialized job ids; recovered/ restarted job attempts finding stale pendingsets.","solutions":["Use a unique output directory per job/run (e.g. append the application id or timestamp) so pendingsets never mix","Clean stale commit artifacts (_SUCCESS, pendingset-*.json, magic dirs) from the shared output location before resubmission","Verify the job id propagation: ensure the context's jobId is set when PendingSet is serialized, so the file never carries an empty id","Upgrade hadoop-tos once the inverted guard (!StringUtils.isNoneEmpty) is fixed upstream, since non-empty mismatches currently pass silently"],"exampleFix":"# before: two runs share the output dir, second commit reads first job's pendingset\nmapreduce.output.fileoutputformat.outputdir=tos://bucket/out\n\n# after: per-run output dir\nmapreduce.output.fileoutputformat.outputdir=tos://bucket/out/${yarn.app.attempt.id}","handlingStrategy":"fallback","validationCode":"// before submit: ensure the output dir holds no foreign commit artifacts\nPath out = new Path(conf.get(\"mapreduce.output.fileoutputformat.outputdir\"));\ntry (FileSystem fs = out.getFileSystem(conf)) {\n  if (fs.exists(out)) {\n    for (FileStatus st : fs.listStatus(out)) {\n      if (st.getPath().getName().startsWith(\"_SUCCESS\")\n          || st.getPath().getName().startsWith(\"pendingset-\")) {\n        fs.delete(st.getPath(), true);\n      }\n    }\n  }\n}","typeGuard":null,"tryCatchPattern":"// framework-level: committer failure surfaces at commitJob\ncatch (IllegalStateException e) {\n  if (e.getMessage() != null && e.getMessage().contains(\"Mismatch in Job ID\")) {\n    // clean the stale output/magic dir and resubmit the job\n  }\n}","preventionTips":["Give every run a unique output directory (append application/attempt id)","Never share an output or magic directory between concurrent jobs","Clean stale _SUCCESS/pendingset artifacts before resubmitting into a reused dir","Track the upstream fix for the inverted empty-jobId guard: non-empty mismatches currently pass silently"],"tags":["committer","job-id-mismatch","stale-metadata","magic-commit","tos"],"backgroundTag":"job-id-mismatch","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}