apache/hadoop · error · IOException

Missing Log Directory for History

Error message

Missing Log Directory for History

What it means

Thrown by JobHistoryEventHandler.setupEventWriter (MR ApplicationMaster) when it is asked to start writing job-history events for a job but stagingDirPath is still null. stagingDirPath is resolved once during serviceInit from the configured history staging directory (JobHistoryUtils.getConfiguredHistoryStagingDirPrefix, ultimately mapreduce.job.staging-dir / yarn.app.mapreduce.am.staging-dir). If serviceInit never ran to completion, every later call to open the history event writer for a job aborts with this IOException and history logging for the job is disabled.

Source

Thrown at hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/jobhistory/JobHistoryEventHandler.java:520

      throws IOException {
    FSDataOutputStream out = stagingDirFS.create(historyFilePath, true);
    return new EventWriter(out, this.jhistMode);
  }
  
  /**
   * Create an event writer for the Job represented by the jobID.
   * Writes out the job configuration to the log directory.
   * This should be the first call to history for a job
   * 
   * @param jobId the jobId.
   * @param amStartedEvent
   * @throws IOException
   */
  protected void setupEventWriter(JobId jobId, AMStartedEvent amStartedEvent)
      throws IOException {
    if (stagingDirPath == null) {
      LOG.error("Log Directory is null, returning");
      throw new IOException("Missing Log Directory for History");
    }

    MetaInfo oldFi = fileMap.get(jobId);
    Configuration conf = getConfig();

    // TODO Ideally this should be written out to the job dir
    // (.staging/jobid/files - RecoveryService will need to be patched)
    Path historyFile = JobHistoryUtils.getStagingJobHistoryFile(
        stagingDirPath, jobId, startCount);
    String user = UserGroupInformation.getCurrentUser().getShortUserName();
    if (user == null) {
      throw new IOException(
          "User is null while setting up jobhistory eventwriter");
    }

    String jobName = context.getJob(jobId).getName();
    EventWriter writer = (oldFi == null) ? null : oldFi.writer;
 

View on GitHub (pinned to 2add963021)

Solutions

1. Check the AM log for the real root cause emitted from serviceInit: 'Failed while getting the configured log directories' or 'Failed while checking for/creating history staging path' — this IOException is only a downstream symptom
2. Set a valid staging directory reachable from the AM, e.g. mapreduce.job.staging-dir=hdfs://nn:8020/tmp/hadoop-yarn/staging on the same filesystem used at submit time
3. Verify permissions on the staging dir (history staging dir permissions 770, job owner must write) and that the NameNode is up
4. If you embed JobHistoryEventHandler in your own service, ensure serviceInit(conf) completes successfully before the first event is handled

// before job submission, prove the history staging dir resolves and is writable
Configuration conf = new Configuration();
String staging = conf.get("mapreduce.job.staging-dir",
    conf.get("yarn.app.mapreduce.am.staging-dir", "/tmp/hadoop-yarn/staging"));
Path dir = new Path(staging);
FileSystem fs = FileSystem.get(dir.toUri(), conf);
if (!fs.exists(dir) && !fs.mkdirs(dir)) {
  throw new IOException("History staging dir unusable: " + dir);
}
if (!fs.getFileStatus(dir).getPermission().getUserAction().implies(FsAction.WRITE)) {
  throw new IOException("No write permission on staging dir: " + dir);
}

When it happens

Trigger: The AM handles its first JobHistoryEvent (setupEventWriter is called for the JobStarted/AMStarted path) while serviceInit failed earlier or was never invoked; stagingDirStr could not be resolved to a qualified Path; a custom embedding of JobHistoryEventHandler dispatches events before the service is initialized.

Common situations: Staging directory deleted or made unwritable between job submission and AM start; mapreduce.job.staging-dir / yarn.app.mapreduce.am.staging-dir pointing to an unreachable NameNode or garbage path; unit tests constructing the handler without going through the CompositeService lifecycle; AM started from a truncated/failed init that still receives events.

Related errors

• "Not creating intermediate history logDir: [" + doneDirPath
• Not yet implemented.
• User is null while setting up jobhistory eventwriter
• MR AM not authorized to issue delegation token
• Only one of the following keys can be specified for a single

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/f58cb6ae79921209. Report an issue: GitHub.