apache/hadoop · error · IOException

Invalid state of the job for cleanup. State found " + jobRun

Error message

Invalid state of the job for cleanup. State found " + jobRunStateForCleanup + " expecting SUCCEEDED, FAILED or KILLED

What it means

At job completion the cleanup task calls the OutputCommitter with the job's final run state, which must be one of SUCCEEDED, FAILED, or KILLED (abortJob or commitJob respectively). Any other JobStatus.State reaching this branch — RUNNING, PREP, UNDEFINED — is an internal invariant violation and this IOException is thrown, failing the job-cleanup attempt. It indicates the job's terminal state was never properly set before cleanup ran.

Source

Thrown at hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/Task.java:1484

    getProgress().setStatus("cleanup");
    statusUpdate(umbilical);
    // do the cleanup
    LOG.info("Cleaning up job");
    if (jobRunStateForCleanup == JobStatus.State.FAILED 
        || jobRunStateForCleanup == JobStatus.State.KILLED) {
      LOG.info("Aborting job with runstate : " + jobRunStateForCleanup.name());
      if (conf.getUseNewMapper()) {
        committer.abortJob(jobContext, jobRunStateForCleanup);
      } else {
        org.apache.hadoop.mapred.OutputCommitter oldCommitter = 
          (org.apache.hadoop.mapred.OutputCommitter)committer;
        oldCommitter.abortJob(jobContext, jobRunStateForCleanup);
      }
    } else if (jobRunStateForCleanup == JobStatus.State.SUCCEEDED){
      LOG.info("Committing job");
      committer.commitJob(jobContext);
    } else {
      throw new IOException("Invalid state of the job for cleanup. State found "
                            + jobRunStateForCleanup + " expecting "
                            + JobStatus.State.SUCCEEDED + ", " 
                            + JobStatus.State.FAILED + " or "
                            + JobStatus.State.KILLED);
    }
    
    // delete the staging area for the job
    JobConf conf = new JobConf(jobContext.getConfiguration());
    if (!keepTaskFiles(conf)) {
      String jobTempDir = conf.get(MRJobConfig.MAPREDUCE_JOB_DIR);
      Path jobTempDirPath = new Path(jobTempDir);
      FileSystem fs = jobTempDirPath.getFileSystem(conf);
      fs.delete(jobTempDirPath, true);
    }
    done(umbilical, reporter);
  }
  
  protected boolean keepTaskFiles(JobConf conf) {

View on GitHub (pinned to 2add963021)

Solutions

1. Check the JobTracker/ResourceManager logs for the state of the job right before cleanup to identify the race.
2. Retry/resubmit the job — a one-off race usually clears on the next run.
3. If it reproduces, verify any custom OutputCommitter or job-control code does not trigger cleanup before the terminal state is set.
4. Upgrade the cluster if the version matches a known job-completion race fix.

// if you control job-control code, assert a terminal state before cleanup
if (jobRunStateForCleanup == null
    || (jobRunStateForCleanup != JobStatus.State.SUCCEEDED
        && jobRunStateForCleanup != JobStatus.State.FAILED
        && jobRunStateForCleanup != JobStatus.State.KILLED)) {
  throw new IllegalStateException("Job state not terminal before cleanup: " + jobRunStateForCleanup);
}

try {
  task.commit(new TaskUmbilicalProtocol() {/*...*/});
} catch (IOException e) {
  if (e.getMessage().contains("Invalid state of the job for cleanup")) {
    // internal race: check JT/RM logs for the job's terminal state, then resubmit
  }
}

When it happens

Trigger: Framework code or a custom committer path invokes job-cleanup logic while jobRunStateForCleanup is still RUNNING/PREP (race between job completion events), or a patched/ custom JobTracker passes an unset state. Rare; almost always surfaces in job cleanup logs.

Common situations: JobTracker failover or restart racing job completion; forks with customized commit logic; versions with completion-state bugs. Ordinary user jobs should not hit this.

Understand the failure class

Background: "Invalid state transition" errors: "status must be X, actually Y", "already rejected/charging/uninstalled", "cannot ... while running" — what they mean when a library rejects your call — this error's family across 31 libraries.

Related errors

• Unable to recover task %s, output: %s
• Only 1 or 2 algorithm version is supported
• Failed to delete {}
• Failed to rename {} to {}
• Failed to create {clazz}:{e}

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/7e676081dbe51c4b. Report an issue: GitHub.