apache/hadoop · error · YarnRuntimeException

Could not load history file ${historyFileAbsolute}

Error message

Could not load history file ${historyFileAbsolute}

What it means

The Job History Server found the job's history file path (historyFileAbsolute != null) but the parser could not read it: createJobHistoryParser() or parser.parse() raised an IOException, which CompletedJob wraps in a YarnRuntimeException. The JHS log line at WARN carries the underlying IOException and names the real cause (missing file, permissions, unreadable block). The lazy job load fails, so any client RPC touching that job's details surfaces this error.

Source

Thrown at hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/CompletedJob.java:385

  }

  //History data is leisurely loaded when task level data is requested
  protected synchronized void loadFullHistoryData(boolean loadTasks,
      Path historyFileAbsolute) throws IOException {
    LOG.info("Loading history file: [" + historyFileAbsolute + "]");
    if (this.jobInfo != null) {
      return;
    }
    
    if (historyFileAbsolute != null) {
      JobHistoryParser parser = null;
      try {
        parser = createJobHistoryParser(historyFileAbsolute);
        this.jobInfo = parser.parse();
      } catch (IOException e) {
        String errorMsg = "Could not load history file " + historyFileAbsolute;
        LOG.warn(errorMsg, e);
        throw new YarnRuntimeException(errorMsg, e);
      }
      IOException parseException = parser.getParseException(); 
      if (parseException != null) {
        String errorMsg = "Could not parse history file " + historyFileAbsolute;
        LOG.warn(errorMsg, parseException);
        throw new YarnRuntimeException(errorMsg, parseException);
      }
    } else {
      String errorMsg = "History file not found";
      LOG.warn(errorMsg);
      throw new IOException(errorMsg);
    }
    if (loadTasks) {
      loadAllTasks();
      LOG.info("TaskInfo loaded");
    }    
  }

View on GitHub (pinned to 2add963021)

Solutions

1. Check the JHS log for the WARN line — the wrapped IOException names the actual cause (AccessControlException, FileNotFoundException, BlockMissingException)
2. Verify the .jhist exists and is readable by the JHS user: hdfs dfs -ls <done-dir>/<serial-part> and hdfs dfs -cat on the file
3. Fix ownership/permissions of the done directory (chown to the mapred/JHS user) and restart the JHS
4. If the file is corrupt or orphaned, delete it (or let the cleaner expire it via mapreduce.jobhistory.max-age-ms) so the JHS stops indexing it

// pre-flight before JHS serves a job: confirm the history file exists and is readable
FileContext fc = FileContext.getFileContext(historyFileAbsolute.toUri(), conf);
if (!fc.util().exists(historyFileAbsolute)) {
  // skip / mark job unavailable instead of failing load
}
FileStatus st = fc.getFileStatus(historyFileAbsolute);
FsAction need = FsAction.READ;
if (!st.getPermission().getUserAction().implies(need)
    && !fsIsOwnedByCurrentUser(st)) {
  // flag permission problem before parser runs
}

try {
  job = new CompletedJob(...); // or history.getJob(jobId)
} catch (YarnRuntimeException e) {
  if (e.getMessage() != null && e.getMessage().startsWith("Could not load history file")) {
    Throwable cause = e.getCause(); // real IOException: permission / missing file / bad block
    LOG.warn("History unavailable for " + jobId + ": " + cause);
    return unavailableJob(jobId); // degrade, don't crash the JHS request loop
  }
  throw e;
}

When it happens

Trigger: Lazy load of a CompletedJob (first getCounters/getTaskReports/getDiagnostics RPC on a history job) when the .jhist file cannot be read: HDFS permissions changed after submission, the file was deleted between the index scan and the load, or the file is unreadable due to Datanode/block loss.

Common situations: History file half-moved from intermediate-done-dir to done-dir because the JHS was killed mid moveToDone; done directory permissions tightened after a security audit; HDFS block corruption on the history filesystem; history stored on NFS/local dir with wrong ownership after a JHS host migration.

Related errors

• Could not parse history file ${historyFileAbsolute}
• Failed to initialize existing directories
• "Not creating intermediate history logDir: [" + doneDirPath
• unable to load configuration for job: {}
• Cannot find job submission directory! It should just be crea

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/e7b774a3271cb627. Report an issue: GitHub.