apache/hadoop · error · IOException

mkdir failed for {target}

Error message

mkdir failed for {target}

What it means

CopyMapper creates target directories via RetriableDirectoryCreateCommand; if the command still throws after its retries (the underlying mkdirs failed), the mapper rethrows wrapped as 'mkdir failed for <target>'. The nested cause carries the FileSystem-level error.

Source

Thrown at hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/mapred/CopyMapper.java:304

    incrementCounter(context, Counter.COPY, 1);
    totalBytesCopied += bytesCopied;

    if (verboseLog) {
      context.write(null,
          new Text("FILE_COPIED: source=" + sourceFileStatus.getPath() + ","
          + " size=" + sourceFileStatus.getLen() + " --> "
          + "target=" + target + ", size=" + (targrtFileStatus == null ?
              0 : targrtFileStatus.getLen())));
    }
  }

  private void createTargetDirsWithRetry(String description, Path target,
      Context context, FileStatus sourceStatus, FileSystem sourceFS) throws IOException {
    try {
      new RetriableDirectoryCreateCommand(description).execute(target, context,
          sourceStatus, sourceFS);
    } catch (Exception e) {
      throw new IOException("mkdir failed for " + target, e);
    }
    incrementCounter(context, Counter.DIR_COPY, 1);
  }

  private static void updateSkipCounters(Context context,
      CopyListingFileStatus sourceFile) {
    incrementCounter(context, Counter.SKIP, 1);
    incrementCounter(context, Counter.BYTESSKIPPED, sourceFile.getLen());

  }

  private void handleFailures(IOException exception,
      CopyListingFileStatus sourceFileStatus, Path target, Context context)
      throws IOException, InterruptedException {
    LOG.error("Failure in copying " + sourceFileStatus.getPath() +
        (sourceFileStatus.isSplit()? ","
            + " offset=" + sourceFileStatus.getChunkOffset()
            + " chunkLength=" + sourceFileStatus.getChunkLength()

View on GitHub (pinned to 2add963021)

Solutions

1. Read the nested cause for the FileSystem error; check and fix write permission on the target parent
2. Check quotas (hdfs dfs -count / hdfs dfsadmin -setQuota) and raise them if exhausted
3. Remove any file occupying the directory path, then re-run with -update
4. Confirm the NameNode is reachable and out of safe mode before re-running

// before submit: ensure a directory can be created at the target root
Path parent = targetRoot.getParent();
if (!targetFS.exists(parent) && !targetFS.mkdirs(parent)) {
  throw new IOException("Cannot create target parent: " + parent);
}
if (targetFS.exists(targetRoot) && !targetFS.getFileStatus(targetRoot).isDirectory()) {
  throw new IOException("Target occupied by a file: " + targetRoot);
}

catch (IOException e) {  // wraps 'mkdir failed for <target>'
  Throwable root = e.getCause();  // underlying FS error: permission/quota/safe-mode
  LOG.error("mkdir failed, cause:", root);
}

When it happens

Trigger: No write permission on the target's parent, HDFS namespace or space quota exceeded, an existing FILE occupies the path where a directory is needed, NameNode in safe mode or unreachable, or the parent path deleted while the job ran.

Common situations: Destination owned by another user; HDFS quota exhausted; a stale file blocking a directory path; transient NameNode unavailability outlasting the retry policy.

Related errors

• Mkdirs failed to create {} (exists={}, cwd={})
• Destination '{parentFile}' directory cannot be created
• key + ": No such file or directory."
• Can't make directory for path '%s' since it is a file.
• Cannot create directories because of existing file: %s

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/31f0e15c2dd0a620. Report an issue: GitHub.