{"record":{"id":"7e676081dbe51c4b","repo":"apache/hadoop","slug":"invalid-state-of-the-job-for-cleanup-state-found","errorCode":null,"errorMessage":"Invalid state of the job for cleanup. State found \" + jobRunStateForCleanup + \" expecting SUCCEEDED, FAILED or KILLED","messagePattern":"Invalid state of the job for cleanup\\. State found \" \\+ jobRunStateForCleanup \\+ \" expecting SUCCEEDED, FAILED or KILLED","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/Task.java","lineNumber":1484,"sourceCode":"    getProgress().setStatus(\"cleanup\");\n    statusUpdate(umbilical);\n    // do the cleanup\n    LOG.info(\"Cleaning up job\");\n    if (jobRunStateForCleanup == JobStatus.State.FAILED \n        || jobRunStateForCleanup == JobStatus.State.KILLED) {\n      LOG.info(\"Aborting job with runstate : \" + jobRunStateForCleanup.name());\n      if (conf.getUseNewMapper()) {\n        committer.abortJob(jobContext, jobRunStateForCleanup);\n      } else {\n        org.apache.hadoop.mapred.OutputCommitter oldCommitter = \n          (org.apache.hadoop.mapred.OutputCommitter)committer;\n        oldCommitter.abortJob(jobContext, jobRunStateForCleanup);\n      }\n    } else if (jobRunStateForCleanup == JobStatus.State.SUCCEEDED){\n      LOG.info(\"Committing job\");\n      committer.commitJob(jobContext);\n    } else {\n      throw new IOException(\"Invalid state of the job for cleanup. State found \"\n                            + jobRunStateForCleanup + \" expecting \"\n                            + JobStatus.State.SUCCEEDED + \", \" \n                            + JobStatus.State.FAILED + \" or \"\n                            + JobStatus.State.KILLED);\n    }\n    \n    // delete the staging area for the job\n    JobConf conf = new JobConf(jobContext.getConfiguration());\n    if (!keepTaskFiles(conf)) {\n      String jobTempDir = conf.get(MRJobConfig.MAPREDUCE_JOB_DIR);\n      Path jobTempDirPath = new Path(jobTempDir);\n      FileSystem fs = jobTempDirPath.getFileSystem(conf);\n      fs.delete(jobTempDirPath, true);\n    }\n    done(umbilical, reporter);\n  }\n  \n  protected boolean keepTaskFiles(JobConf conf) {","sourceCodeStart":1466,"sourceCodeEnd":1502,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/Task.java#L1466-L1502","documentation":"At job completion the cleanup task calls the OutputCommitter with the job's final run state, which must be one of SUCCEEDED, FAILED, or KILLED (abortJob or commitJob respectively). Any other JobStatus.State reaching this branch — RUNNING, PREP, UNDEFINED — is an internal invariant violation and this IOException is thrown, failing the job-cleanup attempt. It indicates the job's terminal state was never properly set before cleanup ran.","triggerScenarios":"Framework code or a custom committer path invokes job-cleanup logic while jobRunStateForCleanup is still RUNNING/PREP (race between job completion events), or a patched/ custom JobTracker passes an unset state. Rare; almost always surfaces in job cleanup logs.","commonSituations":"JobTracker failover or restart racing job completion; forks with customized commit logic; versions with completion-state bugs. Ordinary user jobs should not hit this.","solutions":["Check the JobTracker/ResourceManager logs for the state of the job right before cleanup to identify the race.","Retry/resubmit the job — a one-off race usually clears on the next run.","If it reproduces, verify any custom OutputCommitter or job-control code does not trigger cleanup before the terminal state is set.","Upgrade the cluster if the version matches a known job-completion race fix."],"exampleFix":null,"handlingStrategy":"try-catch","validationCode":"// if you control job-control code, assert a terminal state before cleanup\nif (jobRunStateForCleanup == null\n    || (jobRunStateForCleanup != JobStatus.State.SUCCEEDED\n        && jobRunStateForCleanup != JobStatus.State.FAILED\n        && jobRunStateForCleanup != JobStatus.State.KILLED)) {\n  throw new IllegalStateException(\"Job state not terminal before cleanup: \" + jobRunStateForCleanup);\n}","typeGuard":null,"tryCatchPattern":"try {\n  task.commit(new TaskUmbilicalProtocol() {/*...*/});\n} catch (IOException e) {\n  if (e.getMessage().contains(\"Invalid state of the job for cleanup\")) {\n    // internal race: check JT/RM logs for the job's terminal state, then resubmit\n  }\n}","preventionTips":["Do not invoke job-cleanup paths from custom committers before the terminal state is recorded.","Watch for this during JobTracker failover testing; one-off races usually clear on retry.","Pin the cluster to a maintenance release if a known completion-race bug applies."],"tags":["hadoop","mapreduce","job-cleanup","output-committer","state-machine"],"backgroundTag":"invalid-state-transition","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}