apache/hadoop · critical · YarnRuntimeException

Failed to initialize existing directories

Error message

Failed to initialize existing directories

What it means

During serviceInit, JobHistory constructs a HistoryFileManager and calls initExisting() to scan the intermediate-done and done directories for existing history files. Any IOException from that scan is wrapped in YarnRuntimeException 'Failed to initialize existing directories', which aborts JHS startup with the original cause attached. Typical roots are missing or unwritable history directories or HDFS unavailability.

Source

Thrown at hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/JobHistory.java:101

  
  @Override
  protected void serviceInit(Configuration conf) throws Exception {
    LOG.info("JobHistory Init");
    this.conf = conf;
    this.appID = ApplicationId.newInstance(0, 0);
    this.appAttemptID = RecordFactoryProvider.getRecordFactory(conf)
        .newRecordInstance(ApplicationAttemptId.class);

    moveThreadInterval = conf.getLong(
        JHAdminConfig.MR_HISTORY_MOVE_INTERVAL_MS,
        JHAdminConfig.DEFAULT_MR_HISTORY_MOVE_INTERVAL_MS);

    hsManager = createHistoryFileManager();
    hsManager.init(conf);
    try {
      hsManager.initExisting();
    } catch (IOException e) {
      throw new YarnRuntimeException("Failed to initialize existing directories", e);
    }

    storage = createHistoryStorage();
    
    if (storage instanceof Service) {
      ((Service) storage).init(conf);
    }
    storage.setHistoryFileManager(hsManager);

    super.serviceInit(conf);
  }

  protected HistoryStorage createHistoryStorage() {
    return ReflectionUtils.newInstance(conf.getClass(
        JHAdminConfig.MR_HISTORY_STORAGE, CachedHistoryStorage.class,
        HistoryStorage.class), conf);
  }
  

View on GitHub (pinned to 2add963021)

Solutions

1. Read the wrapped cause in the stack trace — it names the real failure.
2. Create the configured directories with correct ownership: hadoop fs -mkdir -p for both done and intermediate-done dirs, then chown/chmod for the JHS user.
3. Ensure the JHS user has rwx on both directory trees.
4. Start JHS only after HDFS is reachable and healthy.

Example fix

# before: done dir missing or owned by another user
hadoop fs -ls /mr-history/done

# after: provision both history dirs for the mapred user
hadoop fs -mkdir -p /mr-history/done /mr-history/done_intermediate
hadoop fs -chown -R mapred:hadoop /mr-history

# pre-start readiness: history dirs exist and are writable
for d in /mr-history/done /mr-history/done_intermediate; do
  hadoop fs -test -d $d || hadoop fs -mkdir -p $d
  hadoop fs -test -w $d || echo "$d not writable"
done

catch YarnRuntimeException during JHS init and inspect getCause():
try {
  jobHistory.init(conf);
} catch (org.apache.hadoop.yarn.exceptions.YarnRuntimeException e) {
  Throwable c = e.getCause(); // original IOException names the failing directory
  throw c instanceof IOException ? (IOException) c : e;
}

When it happens

Trigger: mapreduce.jobhistory.done-dir or mapreduce.jobhistory.intermediate-done-dir missing or unwritable for the JHS user; HDFS unreachable at startup; a file occupying a directory path; ACL denying traversal of the history tree.

Common situations: Fresh installs where history directories were never created; permission drift after HDFS changes; JHS starting while HDFS is down; done-dir relocated without creating the new location.

Related errors

• Directory {dir} is in an inconsistent state: storage directo
• cannot access checkpoint directory.
• Cannot remove current dump directory: {}
• Cannot create dump directory {}
• Could not load history file ${historyFileAbsolute}

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/b67bc07c23f4f321. Report an issue: GitHub.