{"record":{"id":"f17436615afef83d","repo":"apache/hadoop","slug":"not-creating-intermediate-history-logdir-do","errorCode":null,"errorMessage":"\"Not creating intermediate history logDir: [\" + doneDirPath + \"] based on conf: mapreduce.am.create-intermediate-jh-base-dir. Either set to true or pre-create this directory with appropriate permissions\"","messagePattern":"\"Not creating intermediate history logDir: \\[\" \\+ doneDirPath \\+ \"\\] based on conf: mapreduce\\.am\\.create-intermediate-jh-base-dir\\. Either set to true or pre-create this directory with appropriate permissions\"","errorType":"exception","errorClass":"YarnRuntimeException","httpStatus":null,"severity":"critical","filePath":"hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/jobhistory/JobHistoryEventHandler.java","lineNumber":226,"sourceCode":" + MRJobConfig.MR_AM_CREATE_JH_INTERMEDIATE_BASE_DIR);\n mkdir(\n doneDirFS,\n doneDirPath,\n new FsPermission(\n JobHistoryUtils.HISTORY_INTERMEDIATE_DONE_DIR_PERMISSIONS\n .toShort()));\n // TODO Temporary toShort till new FsPermission(FsPermissions)\n // respects\n // sticky\n } else {\n String message = \"Not creating intermediate history logDir: [\"\n + doneDirPath\n + \"] based on conf: \"\n + MRJobConfig.MR_AM_CREATE_JH_INTERMEDIATE_BASE_DIR\n + \". Either set to true or pre-create this directory with\" +\n \" appropriate permissions\";\n LOG.error(message);\n throw new YarnRuntimeException(message);\n }\n }\n } catch (IOException e) {\n LOG.error(\"Failed checking for the existence of history intermediate \" +\n \t\t\"done directory: [\" + doneDirPath + \"]\");\n throw new YarnRuntimeException(e);\n }\n\n //Check/create user directory under intermediate done dir.\n try {\n doneDirPrefixPath =\n FileContext.getFileContext(conf).makeQualified(new Path(userDoneDirStr));\n mkdir(doneDirFS, doneDirPrefixPath, JobHistoryUtils.\n getConfiguredHistoryIntermediateUserDoneDirPermissions(conf));\n } catch (IOException e) {\n LOG.error(\"Error creating user intermediate history done directory: [ \"\n + doneDirPrefixPath + \"]\", e);\n throw new YarnRuntimeException(e);","sourceCodeStart":208,"sourceCodeEnd":244,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/jobhistory/JobHistoryEventHandler.java#L208-L244","documentation":"At MR ApplicationMaster init, JobHistoryEventHandler checks the job-history intermediate done directory (mapreduce.jobhistory.intermediate-done-dir) on its filesystem. If the directory does not exist and mapreduce.am.create-intermediate-jh-base-dir is false (the default), the AM refuses to run and throws YarnRuntimeException('Not creating intermediate history logDir: ... Either set to true or pre-create this directory...'), failing the job before any task starts. The design expects the JobHistoryServer or deployment to have created that base directory.","triggerScenarios":"Fresh cluster where the intermediate done dir (default under the staging root, e.g. /history/done_intermediate on HDFS) was never created; custom mapreduce.jobhistory.intermediate-done-dir pointing to a missing path; permissions let the AM see the dir is absent but policy forbids it to mkdir (flag false); history server not yet deployed when the first job is submitted.","commonSituations":"New installations or test clusters without a running JobHistoryServer; secure clusters where the pre-created dir has wrong ownership; config drift where staging/intermediate-done-dir moved but the HDFS layout was not migrated.","solutions":["Pre-create the directory with history-appropriate permissions as part of deployment: hdfs dfs -mkdir -p ; hdfs dfs -chmod 1771/1777 (match your cluster's policy)","Or set mapreduce.am.create-intermediate-jh-base-dir=true so the AM creates the base dir itself (fine for single-user/test clusters; shared clusters should pre-create it)","Verify the exact configured value: mapreduce.jobhistory.intermediate-done-dir and the staging dir (yarn.app.mapreduce.am.staging-dir) resolve where you expect","Confirm the JobHistoryServer is deployed and created its directory tree; check the AM logs for the doneDirPath in the message"],"exampleFix":"# before: dir absent, AM refuses to start\n# after: pre-create with history-server-friendly perms, or let the AM create it\nhdfs dfs -mkdir -p /mr-history/done_intermediate\nhdfs dfs -chmod 1777 /mr-history/done_intermediate\n# or in mapred-site.xml\nmapreduce.am.create-intermediate-jh-base-dirtrue","handlingStrategy":"validation","validationCode":"// pre-flight in the job client before submit\nPath doneDir = new Path(conf.get(\"mapreduce.jobhistory.intermediate-done-dir\",\n \"/tmp/hadoop-yarn/staging/history/done_intermediate\"));\nFileSystem fs = doneDir.getFileSystem(conf);\nif (!fs.exists(doneDir)\n && !conf.getBoolean(\"mapreduce.am.create-intermediate-jh-base-dir\", false)) {\n throw new IOException(\"Pre-create \" + doneDir\n + \" or set mapreduce.am.create-intermediate-jh-base-dir=true\");\n}","typeGuard":null,"tryCatchPattern":"Catch YarnRuntimeException at AM init in integration tests and assert the message contains 'Not creating intermediate history logDir' -- then fail fast with the two remediation options instead of a generic AM launch failure.","preventionTips":["Bake 'hdfs dfs -mkdir -p ' plus permissions into cluster deployment","For test clusters set mapreduce.am.create-intermediate-jh-base-dir=true","Whenever you change mapreduce.jobhistory.intermediate-done-dir, migrate the HDFS layout in the same change"],"tags":["mapreduce","yarn","application-master","job-history","hdfs","configuration","missing-directory"],"backgroundTag":"directory-not-found","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}