{"record":{"id":"f58cb6ae79921209","repo":"apache/hadoop","slug":"missing-log-directory-for-history","errorCode":null,"errorMessage":"Missing Log Directory for History","messagePattern":"Missing Log Directory for History","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/jobhistory/JobHistoryEventHandler.java","lineNumber":520,"sourceCode":"      throws IOException {\n    FSDataOutputStream out = stagingDirFS.create(historyFilePath, true);\n    return new EventWriter(out, this.jhistMode);\n  }\n  \n  /**\n   * Create an event writer for the Job represented by the jobID.\n   * Writes out the job configuration to the log directory.\n   * This should be the first call to history for a job\n   * \n   * @param jobId the jobId.\n   * @param amStartedEvent\n   * @throws IOException\n   */\n  protected void setupEventWriter(JobId jobId, AMStartedEvent amStartedEvent)\n      throws IOException {\n    if (stagingDirPath == null) {\n      LOG.error(\"Log Directory is null, returning\");\n      throw new IOException(\"Missing Log Directory for History\");\n    }\n\n    MetaInfo oldFi = fileMap.get(jobId);\n    Configuration conf = getConfig();\n\n    // TODO Ideally this should be written out to the job dir\n    // (.staging/jobid/files - RecoveryService will need to be patched)\n    Path historyFile = JobHistoryUtils.getStagingJobHistoryFile(\n        stagingDirPath, jobId, startCount);\n    String user = UserGroupInformation.getCurrentUser().getShortUserName();\n    if (user == null) {\n      throw new IOException(\n          \"User is null while setting up jobhistory eventwriter\");\n    }\n\n    String jobName = context.getJob(jobId).getName();\n    EventWriter writer = (oldFi == null) ? null : oldFi.writer;\n ","sourceCodeStart":502,"sourceCodeEnd":538,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/jobhistory/JobHistoryEventHandler.java#L502-L538","documentation":"Thrown by JobHistoryEventHandler.setupEventWriter (MR ApplicationMaster) when it is asked to start writing job-history events for a job but stagingDirPath is still null. stagingDirPath is resolved once during serviceInit from the configured history staging directory (JobHistoryUtils.getConfiguredHistoryStagingDirPrefix, ultimately mapreduce.job.staging-dir / yarn.app.mapreduce.am.staging-dir). If serviceInit never ran to completion, every later call to open the history event writer for a job aborts with this IOException and history logging for the job is disabled.","triggerScenarios":"The AM handles its first JobHistoryEvent (setupEventWriter is called for the JobStarted/AMStarted path) while serviceInit failed earlier or was never invoked; stagingDirStr could not be resolved to a qualified Path; a custom embedding of JobHistoryEventHandler dispatches events before the service is initialized.","commonSituations":"Staging directory deleted or made unwritable between job submission and AM start; mapreduce.job.staging-dir / yarn.app.mapreduce.am.staging-dir pointing to an unreachable NameNode or garbage path; unit tests constructing the handler without going through the CompositeService lifecycle; AM started from a truncated/failed init that still receives events.","solutions":["Check the AM log for the real root cause emitted from serviceInit: 'Failed while getting the configured log directories' or 'Failed while checking for/creating history staging path' — this IOException is only a downstream symptom","Set a valid staging directory reachable from the AM, e.g. mapreduce.job.staging-dir=hdfs://nn:8020/tmp/hadoop-yarn/staging on the same filesystem used at submit time","Verify permissions on the staging dir (history staging dir permissions 770, job owner must write) and that the NameNode is up","If you embed JobHistoryEventHandler in your own service, ensure serviceInit(conf) completes successfully before the first event is handled"],"exampleFix":null,"handlingStrategy":"validation","validationCode":"// before job submission, prove the history staging dir resolves and is writable\nConfiguration conf = new Configuration();\nString staging = conf.get(\"mapreduce.job.staging-dir\",\n    conf.get(\"yarn.app.mapreduce.am.staging-dir\", \"/tmp/hadoop-yarn/staging\"));\nPath dir = new Path(staging);\nFileSystem fs = FileSystem.get(dir.toUri(), conf);\nif (!fs.exists(dir) && !fs.mkdirs(dir)) {\n  throw new IOException(\"History staging dir unusable: \" + dir);\n}\nif (!fs.getFileStatus(dir).getPermission().getUserAction().implies(FsAction.WRITE)) {\n  throw new IOException(\"No write permission on staging dir: \" + dir);\n}","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Keep mapreduce.job.staging-dir identical at submit time and AM runtime, on a filesystem the AM can reach","Monitor AM logs for the serviceInit failures ('Failed while checking for/creating history staging path') that precede this error","Never dispatch JobHistoryEvents to JobHistoryEventHandler before its serviceInit has completed"],"tags":["mapreduce","job-history","staging-dir","application-master","configuration"],"backgroundTag":"missing-staging-directory","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}