{"record":{"id":"19eb471ea6b8f320","repo":"apache/hadoop","slug":"cannot-find-job-submission-directory-it-should-ju","errorCode":null,"errorMessage":"Cannot find job submission directory! It should just be created, so something wrong here.","messagePattern":"Cannot find job submission directory! It should just be created, so something wrong here\\.","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/JobResourceUploader.java","lineNumber":834,"sourceCode":"\n  // copy user specified log4j.property file in local\n  // to HDFS with putting on distributed cache and adding its parent directory\n  // to classpath.\n  @SuppressWarnings(\"deprecation\")\n  private void copyLog4jPropertyFile(Job job, Path submitJobDir,\n      short replication) throws IOException {\n    Configuration conf = job.getConfiguration();\n\n    String file =\n        validateFilePath(\n            conf.get(MRJobConfig.MAPREDUCE_JOB_LOG4J_PROPERTIES_FILE), conf);\n    LOG.debug(\"default FileSystem: \" + jtFs.getUri());\n    FsPermission mapredSysPerms =\n        new FsPermission(JobSubmissionFiles.JOB_DIR_PERMISSION);\n    try {\n      jtFs.getFileStatus(submitJobDir);\n    } catch (FileNotFoundException e) {\n      throw new IOException(\"Cannot find job submission directory! \"\n          + \"It should just be created, so something wrong here.\", e);\n    }\n\n    Path fileDir = JobSubmissionFiles.getJobLog4jFile(submitJobDir);\n\n    // first copy local log4j.properties file to HDFS under submitJobDir\n    if (file != null) {\n      FileSystem.mkdirs(jtFs, fileDir, mapredSysPerms);\n      URI tmpURI = null;\n      try {\n        tmpURI = new URI(file);\n      } catch (URISyntaxException e) {\n        throw new IllegalArgumentException(e);\n      }\n      Path tmp = new Path(tmpURI);\n      Path newPath = copyRemoteFiles(fileDir, tmp, conf, replication);\n      Path path = new Path(newPath.toUri().getPath());\n      Job.addFileToClassPath(path, conf, path.getFileSystem(conf));","sourceCodeStart":816,"sourceCodeEnd":852,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/JobResourceUploader.java#L816-L852","documentation":"Late in submission, JobResourceUploader.copyLog4jPropertyFile re-checks the staging directory with jtFs.getFileStatus(submitJobDir). The directory was created moments earlier by uploadResourcesInternal, so a FileNotFoundException means it vanished (or became invisible) between creation and this call; the code wraps it as IOException('Cannot find job submission directory! It should just be created, so something wrong here.').","triggerScenarios":"Another process deleting <staging>/<user>/.staging/job_... between mkdir and the log4j step (cleanup cron, staging cleaner, another job's error handling); HDFS permission or visibility changes mid-submission; NameNode failover making a just-created directory temporarily unresolvable; a shared staging root being wiped by an admin action.","commonSituations":"Aggressive housekeeping scripts purging .staging on a schedule; two submitters racing on the same job id where the loser's cleanup removes the winner's dir; safe-mode or NN instability during submission windows.","solutions":["Simply resubmit — a fresh attempt gets a new job id and staging dir; this is a transient race","Ensure no cron/cleanup deletes the staging root while jobs are being submitted (schedule around it or exclude active job dirs)","Verify HDFS permissions on yarn.app.mapreduce.am.staging-dir/<user>/.staging for the submitting user","If recurrent, check NameNode logs for the window between job-dir creation and this failure"],"exampleFix":"# before: cleanup cron runs mid-submission\ncrontab: */5 * * * * hdfs dfs -rm -r /tmp/hadoop-yarn/staging\n# job submission -> IOException: Cannot find job submission directory!\n\n# after: only clean stale dirs (older than 1 day), never during submission windows\nfind-based cleanup: hdfs dfs -ls ... | filter by modification time before deleting","handlingStrategy":"retry","validationCode":"// pre-flight: staging root must exist and be writable before submitting\nPath stagingRoot = new Path(conf.get(\"yarn.app.mapreduce.am.staging-dir\", \"/tmp/hadoop-yarn/staging\"));\nFileSystem fs = stagingRoot.getFileSystem(conf);\nif (!fs.exists(stagingRoot)) {\n  throw new IOException(\"Staging root missing: \" + stagingRoot);\n}\nfs.access(stagingRoot, FsAction.WRITE);","typeGuard":null,"tryCatchPattern":"try {\n  job.submit();\n} catch (IOException e) {\n  if (e.getMessage() != null && e.getMessage().contains(\"Cannot find job submission directory\")) {\n    // transient race: staging dir vanished mid-submission — a fresh Job gets a new dir\n    job = Job.getInstance(conf);\n    job.submit();\n  } else {\n    throw e;\n  }\n}","preventionTips":["Schedule staging-dir cleanup only for directories older than a day, never live submissions","Ensure the submitting user owns <staging>/<user>/.staging with write permission","Check NameNode health/safe-mode before submission windows if this recurs"],"tags":["staging-directory","race-condition","hdfs","job-submission","mapreduce"],"backgroundTag":"missing-staging-directory","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}