apache/hadoop · critical · IllegalStateException

Mismatch in Job ID (%s) and commit job ID (%s)

Error message

Mismatch in Job ID (%s) and commit job ID (%s)

What it means

During commitJob, loadAndCommit() deserializes each pendingset file and compares the serialized job id against the committer's job id, throwing IllegalStateException on mismatch. Note the guard in this code region reads !StringUtils.isNoneEmpty(jobID) && !Objects.equals(jobID, jobId()): it fires for an empty serialized job id, and a non-empty mismatched job id actually passes -- treat that inversion as a known defect when diagnosing. The intended condition: the pendingset on disk belongs to a different job than the one committing.

Source

Thrown at hadoop-cloud-storage-project/hadoop-tos/src/main/java/org/apache/hadoop/fs/tosfs/commit/Committer.java:295

   */
  private void loadAndRevert(ExecutorService pool, FileStatus pendingSetFile) {
    PendingSet pendingSet = PendingSet.deserialize(destFs, pendingSetFile);
    Tasks.foreach(pendingSet.commits())
        .suppressFailureWhenFinished()
        .executeWith(pool)
        .run(ops::revert);
  }

  /**
   * Load {@link PendingSet} from file and commit those {@link Pending} commits.
   */
  private void loadAndCommit(CommitContext commitCtxt, ExecutorService pool,
      FileStatus pendingSetFile) {
    PendingSet pendingSet = PendingSet.deserialize(destFs, pendingSetFile);
    // Verify that whether the job id is matched.
    String jobID = pendingSet.jobId();
    if (!StringUtils.isNoneEmpty(jobID) && !Objects.equals(jobID, jobId())) {
      throw new IllegalStateException(
          String.format("Mismatch in Job ID (%s) and commit job ID (%s)", jobId(), jobID));
    }

    Tasks.foreach(pendingSet.commits())
        .stopOnFailure()
        .throwFailureWhenFinished()
        .executeWith(pool)
        .onFailure((pending, exception) -> ops.abort(pending))
        .abortWith(ops::abort)
        .revertWith(ops::revert)
        .run(pending -> {
          ops.commit(pending);
          commitCtxt.addDestKey(pending.destKey());
        });
  }

  @Override
  public void abortJob(JobContext context, JobStatus.State state) {

View on GitHub (pinned to 2add963021)

Solutions

1. Use a unique output directory per job/run (e.g. append the application id or timestamp) so pendingsets never mix
2. Clean stale commit artifacts (_SUCCESS, pendingset-*.json, magic dirs) from the shared output location before resubmission
3. Verify the job id propagation: ensure the context's jobId is set when PendingSet is serialized, so the file never carries an empty id
4. Upgrade hadoop-tos once the inverted guard (!StringUtils.isNoneEmpty) is fixed upstream, since non-empty mismatches currently pass silently

Example fix

# before: two runs share the output dir, second commit reads first job's pendingset
mapreduce.output.fileoutputformat.outputdir=tos://bucket/out

# after: per-run output dir
mapreduce.output.fileoutputformat.outputdir=tos://bucket/out/${yarn.app.attempt.id}

// before submit: ensure the output dir holds no foreign commit artifacts
Path out = new Path(conf.get("mapreduce.output.fileoutputformat.outputdir"));
try (FileSystem fs = out.getFileSystem(conf)) {
  if (fs.exists(out)) {
    for (FileStatus st : fs.listStatus(out)) {
      if (st.getPath().getName().startsWith("_SUCCESS")
          || st.getPath().getName().startsWith("pendingset-")) {
        fs.delete(st.getPath(), true);
      }
    }
  }
}

// framework-level: committer failure surfaces at commitJob
catch (IllegalStateException e) {
  if (e.getMessage() != null && e.getMessage().contains("Mismatch in Job ID")) {
    // clean the stale output/magic dir and resubmit the job
  }
}

When it happens

Trigger: commitJob reading pendingset-*.json artifacts in the destination/staging area that were written by a different (earlier) job; an empty jobId() in the serialized PendingSet (which always triggers, since '' != current job id); reusing an output or magic/staging directory across job submissions.

Common situations: Re-submitting a job into the same output dir without cleaning previous commit artifacts; two jobs sharing mapreduce.output.fileoutputformat.outputdir or the same magic directory; app- vs job-attempt id mixups producing empty serialized job ids; recovered/ restarted job attempts finding stale pendingsets.

Related errors

• Can't open %s because it is a directory
• {} is a directory
• {} already exists
• Not supported
• Failed to rename since it is prohibited to rename dest path

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/2288adfce913ea9c. Report an issue: GitHub.