{"record":{"id":"e7b774a3271cb627","repo":"apache/hadoop","slug":"could-not-load-history-file-historyfileabsolute","errorCode":null,"errorMessage":"Could not load history file ${historyFileAbsolute}","messagePattern":"Could not load history file (.+?)","errorType":"exception","errorClass":"YarnRuntimeException","httpStatus":null,"severity":"error","filePath":"hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/CompletedJob.java","lineNumber":385,"sourceCode":"  }\n\n  //History data is leisurely loaded when task level data is requested\n  protected synchronized void loadFullHistoryData(boolean loadTasks,\n      Path historyFileAbsolute) throws IOException {\n    LOG.info(\"Loading history file: [\" + historyFileAbsolute + \"]\");\n    if (this.jobInfo != null) {\n      return;\n    }\n    \n    if (historyFileAbsolute != null) {\n      JobHistoryParser parser = null;\n      try {\n        parser = createJobHistoryParser(historyFileAbsolute);\n        this.jobInfo = parser.parse();\n      } catch (IOException e) {\n        String errorMsg = \"Could not load history file \" + historyFileAbsolute;\n        LOG.warn(errorMsg, e);\n        throw new YarnRuntimeException(errorMsg, e);\n      }\n      IOException parseException = parser.getParseException(); \n      if (parseException != null) {\n        String errorMsg = \"Could not parse history file \" + historyFileAbsolute;\n        LOG.warn(errorMsg, parseException);\n        throw new YarnRuntimeException(errorMsg, parseException);\n      }\n    } else {\n      String errorMsg = \"History file not found\";\n      LOG.warn(errorMsg);\n      throw new IOException(errorMsg);\n    }\n    if (loadTasks) {\n      loadAllTasks();\n      LOG.info(\"TaskInfo loaded\");\n    }    \n  }\n","sourceCodeStart":367,"sourceCodeEnd":403,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/CompletedJob.java#L367-L403","documentation":"The Job History Server found the job's history file path (historyFileAbsolute != null) but the parser could not read it: createJobHistoryParser() or parser.parse() raised an IOException, which CompletedJob wraps in a YarnRuntimeException. The JHS log line at WARN carries the underlying IOException and names the real cause (missing file, permissions, unreadable block). The lazy job load fails, so any client RPC touching that job's details surfaces this error.","triggerScenarios":"Lazy load of a CompletedJob (first getCounters/getTaskReports/getDiagnostics RPC on a history job) when the .jhist file cannot be read: HDFS permissions changed after submission, the file was deleted between the index scan and the load, or the file is unreadable due to Datanode/block loss.","commonSituations":"History file half-moved from intermediate-done-dir to done-dir because the JHS was killed mid moveToDone; done directory permissions tightened after a security audit; HDFS block corruption on the history filesystem; history stored on NFS/local dir with wrong ownership after a JHS host migration.","solutions":["Check the JHS log for the WARN line — the wrapped IOException names the actual cause (AccessControlException, FileNotFoundException, BlockMissingException)","Verify the .jhist exists and is readable by the JHS user: hdfs dfs -ls <done-dir>/<serial-part> and hdfs dfs -cat on the file","Fix ownership/permissions of the done directory (chown to the mapred/JHS user) and restart the JHS","If the file is corrupt or orphaned, delete it (or let the cleaner expire it via mapreduce.jobhistory.max-age-ms) so the JHS stops indexing it"],"exampleFix":null,"handlingStrategy":"try-catch","validationCode":"// pre-flight before JHS serves a job: confirm the history file exists and is readable\nFileContext fc = FileContext.getFileContext(historyFileAbsolute.toUri(), conf);\nif (!fc.util().exists(historyFileAbsolute)) {\n  // skip / mark job unavailable instead of failing load\n}\nFileStatus st = fc.getFileStatus(historyFileAbsolute);\nFsAction need = FsAction.READ;\nif (!st.getPermission().getUserAction().implies(need)\n    && !fsIsOwnedByCurrentUser(st)) {\n  // flag permission problem before parser runs\n}","typeGuard":null,"tryCatchPattern":"try {\n  job = new CompletedJob(...); // or history.getJob(jobId)\n} catch (YarnRuntimeException e) {\n  if (e.getMessage() != null && e.getMessage().startsWith(\"Could not load history file\")) {\n    Throwable cause = e.getCause(); // real IOException: permission / missing file / bad block\n    LOG.warn(\"History unavailable for \" + jobId + \": \" + cause);\n    return unavailableJob(jobId); // degrade, don't crash the JHS request loop\n  }\n  throw e;\n}","preventionTips":["Run the JHS under a dedicated service user that owns the done and intermediate-done directories","Monitor the JHS WARN log for 'Could not load history file' — early sign of permission drift or corruption","Restart the JHS cleanly after unclean shutdowns so interrupted moveToDone scans heal","Snapshot/backup the HDFS done-dir; lost blocks in .jhist files make jobs permanently unloadable"],"tags":["mapreduce","job-history-server","jhist","hdfs","file-read","lazy-load"],"backgroundTag":"history-file-unreadable","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}