apache/hadoop · error · IOException

Cannot find job submission directory! It should just be crea

Error message

Cannot find job submission directory! It should just be created, so something wrong here.

What it means

Late in submission, JobResourceUploader.copyLog4jPropertyFile re-checks the staging directory with jtFs.getFileStatus(submitJobDir). The directory was created moments earlier by uploadResourcesInternal, so a FileNotFoundException means it vanished (or became invisible) between creation and this call; the code wraps it as IOException('Cannot find job submission directory! It should just be created, so something wrong here.').

Source

Thrown at hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/JobResourceUploader.java:834

  // copy user specified log4j.property file in local
  // to HDFS with putting on distributed cache and adding its parent directory
  // to classpath.
  @SuppressWarnings("deprecation")
  private void copyLog4jPropertyFile(Job job, Path submitJobDir,
      short replication) throws IOException {
    Configuration conf = job.getConfiguration();

    String file =
        validateFilePath(
            conf.get(MRJobConfig.MAPREDUCE_JOB_LOG4J_PROPERTIES_FILE), conf);
    LOG.debug("default FileSystem: " + jtFs.getUri());
    FsPermission mapredSysPerms =
        new FsPermission(JobSubmissionFiles.JOB_DIR_PERMISSION);
    try {
      jtFs.getFileStatus(submitJobDir);
    } catch (FileNotFoundException e) {
      throw new IOException("Cannot find job submission directory! "
          + "It should just be created, so something wrong here.", e);
    }

    Path fileDir = JobSubmissionFiles.getJobLog4jFile(submitJobDir);

    // first copy local log4j.properties file to HDFS under submitJobDir
    if (file != null) {
      FileSystem.mkdirs(jtFs, fileDir, mapredSysPerms);
      URI tmpURI = null;
      try {
        tmpURI = new URI(file);
      } catch (URISyntaxException e) {
        throw new IllegalArgumentException(e);
      }
      Path tmp = new Path(tmpURI);
      Path newPath = copyRemoteFiles(fileDir, tmp, conf, replication);
      Path path = new Path(newPath.toUri().getPath());
      Job.addFileToClassPath(path, conf, path.getFileSystem(conf));

View on GitHub (pinned to 2add963021)

Solutions

1. Simply resubmit — a fresh attempt gets a new job id and staging dir; this is a transient race
2. Ensure no cron/cleanup deletes the staging root while jobs are being submitted (schedule around it or exclude active job dirs)
3. Verify HDFS permissions on yarn.app.mapreduce.am.staging-dir/<user>/.staging for the submitting user
4. If recurrent, check NameNode logs for the window between job-dir creation and this failure

Example fix

# before: cleanup cron runs mid-submission
crontab: */5 * * * * hdfs dfs -rm -r /tmp/hadoop-yarn/staging
# job submission -> IOException: Cannot find job submission directory!

# after: only clean stale dirs (older than 1 day), never during submission windows
find-based cleanup: hdfs dfs -ls ... | filter by modification time before deleting

// pre-flight: staging root must exist and be writable before submitting
Path stagingRoot = new Path(conf.get("yarn.app.mapreduce.am.staging-dir", "/tmp/hadoop-yarn/staging"));
FileSystem fs = stagingRoot.getFileSystem(conf);
if (!fs.exists(stagingRoot)) {
  throw new IOException("Staging root missing: " + stagingRoot);
}
fs.access(stagingRoot, FsAction.WRITE);

try {
  job.submit();
} catch (IOException e) {
  if (e.getMessage() != null && e.getMessage().contains("Cannot find job submission directory")) {
    // transient race: staging dir vanished mid-submission — a fresh Job gets a new dir
    job = Job.getInstance(conf);
    job.submit();
  } else {
    throw e;
  }
}

When it happens

Trigger: Another process deleting <staging>/<user>/.staging/job_... between mkdir and the log4j step (cleanup cron, staging cleaner, another job's error handling); HDFS permission or visibility changes mid-submission; NameNode failover making a just-created directory temporarily unresolvable; a shared staging root being wiped by an admin action.

Common situations: Aggressive housekeeping scripts purging .staging on a schedule; two submitters racing on the same job id where the loser's cleanup removes the winner's dir; safe-mode or NN instability during submission windows.

Related errors

• Not submitting job. Job directory {} already exists!! This i
• Recovery block {b} where it is not under construction.
• Failed to copy full contents from '{srcFile}' to '{destFile}
• FsDatasetSpi has not been initialized
• Failed to create temporary file for {}. File {} should be c

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/19eb471ea6b8f320. Report an issue: GitHub.