apache/hadoop · error · YarnRuntimeException

Could not parse history file ${historyFileAbsolute}

Error message

Could not parse history file ${historyFileAbsolute}

What it means

JobHistoryParser.parse() returned but stored a deferred parse exception (parser.getParseException() != null), meaning the .jhist content is malformed — truncated event stream or invalid syntax. CompletedJob wraps it in YarnRuntimeException('Could not parse history file ...') and the job becomes unloadable. This differs from 4401: the file was readable, its content is simply not valid history.

Source

Thrown at hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/CompletedJob.java:391

    if (this.jobInfo != null) {
      return;
    }
    
    if (historyFileAbsolute != null) {
      JobHistoryParser parser = null;
      try {
        parser = createJobHistoryParser(historyFileAbsolute);
        this.jobInfo = parser.parse();
      } catch (IOException e) {
        String errorMsg = "Could not load history file " + historyFileAbsolute;
        LOG.warn(errorMsg, e);
        throw new YarnRuntimeException(errorMsg, e);
      }
      IOException parseException = parser.getParseException(); 
      if (parseException != null) {
        String errorMsg = "Could not parse history file " + historyFileAbsolute;
        LOG.warn(errorMsg, parseException);
        throw new YarnRuntimeException(errorMsg, parseException);
      }
    } else {
      String errorMsg = "History file not found";
      LOG.warn(errorMsg);
      throw new IOException(errorMsg);
    }
    if (loadTasks) {
      loadAllTasks();
      LOG.info("TaskInfo loaded");
    }    
  }

  @Override
  public List<String> getDiagnostics() {
    return Collections.singletonList(jobInfo.getErrorInfo());
  }

  @Override

View on GitHub (pinned to 2add963021)

Solutions

1. Read the wrapped parse exception in the JHS log — it shows the offending position/event in the file
2. Confirm truncation manually: hdfs dfs -cat <file>.jhist | tail — an unterminated last line confirms a partial write
3. Delete or quarantine the corrupt .jhist (it will never parse) so the JHS stops retrying it
4. If history migrates between clusters, keep both on compatible Hadoop versions or convert the files

// cheap sanity check before parse: non-empty and last byte terminates the record stream
FileStatus st = fc.getFileStatus(historyFileAbsolute);
if (st.getLen() == 0) {
  // zero-length .jhist — will never parse, quarantine early
}

try {
  job = history.getJob(jobId);
} catch (YarnRuntimeException e) {
  if (e.getMessage() != null && e.getMessage().startsWith("Could not parse history file")) {
    quarantineHistoryFile(jobId); // move aside so the JHS stops failing on every access
    return unavailableJob(jobId);
  }
  throw e;
}

When it happens

Trigger: Loading a job whose .jhist is truncated (AM killed with SIGKILL before history flush finished, a JSON/Avro line cut mid-write), corrupted by disk or HDFS bit rot, or written by an incompatible Hadoop version whose event schema differs.

Common situations: Jobs killed -9 right before completion so history flush is interrupted; .jhist files copied between clusters running different Hadoop versions; failing datanode disks corrupting under-replicated history blocks; history files touched by scripts.

Related errors

• Could not load history file ${historyFileAbsolute}
• Expected nodetype
• Expected numtype
• Expected strtype
• Unexpected: " + type

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/aee283d50ee2c9a2. Report an issue: GitHub.