apache/hadoop · error · IOException

Mkdirs failed to create directory {dirName}

Error message

Mkdirs failed to create directory {dirName}

What it means

Thrown by the MapFile.Writer constructor when FileSystem.mkdirs(dirName) returns false — the output directory for the map (which will hold 'data' and 'index' files) could not be created. Hadoop's mkdirs returns false rather than throwing for many failure modes (permission denied, a parent existing as a regular file, quota exceeded), so this IOException is the first concrete signal.

Source

Thrown at hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/MapFile.java:340

        throw new IllegalArgumentException("key class or comparator option "
                                           + "must be set");
      }
      this.indexInterval = conf.getInt(INDEX_INTERVAL, this.indexInterval);

      Class<? extends WritableComparable> keyClass;
      if (keyClassOption == null) {
        this.comparator = comparatorOption.getValue();
        keyClass = comparator.getKeyClass();
      } else {
        keyClass= 
          (Class<? extends WritableComparable>) keyClassOption.getValue();
        this.comparator = WritableComparator.get(keyClass, conf);
      }
      this.lastKey = comparator.newKey();
      FileSystem fs = dirName.getFileSystem(conf);

      if (!fs.mkdirs(dirName)) {
        throw new IOException("Mkdirs failed to create directory " + dirName);
      }
      Path dataFile = new Path(dirName, DATA_FILE_NAME);
      Path indexFile = new Path(dirName, INDEX_FILE_NAME);

      SequenceFile.Writer.Option[] dataOptions =
        Options.prependOptions(opts, 
                               SequenceFile.Writer.file(dataFile),
                               SequenceFile.Writer.keyClass(keyClass));
      this.data = SequenceFile.createWriter(conf, dataOptions);

      SequenceFile.Writer.Option[] indexOptions =
        Options.prependOptions(opts, SequenceFile.Writer.file(indexFile),
            SequenceFile.Writer.keyClass(keyClass),
            SequenceFile.Writer.valueClass(LongWritable.class),
            SequenceFile.Writer.compression(CompressionType.BLOCK));
      this.index = SequenceFile.createWriter(conf, indexOptions);      
    }

View on GitHub (pinned to 2add963021)

Solutions

1. Check permissions on the parent path: hadoop fs -ls the parent; fix with hadoop fs -chmod/-chown or choose a path you own.
2. Remove or move any existing regular file occupying the target path.
3. Verify HDFS is healthy and out of safe mode (hdfs dfsadmin -safemode get) and quotas are not exceeded (hdfs dfsadmin -quota).
4. If failures may be transient, catch the IOException and retry Writer construction after a delay.

Example fix

// before: fails with only 'Mkdirs failed to create directory /user/bob/out'
new MapFile.Writer(conf, new Path("/user/bob/out"),
    MapFile.Writer.keyClass(Text.class), MapFile.Writer.valueClass(Text.class));

// after: pre-flight the target so the failure names the real cause
Path dir = new Path("/user/bob/out");
FileSystem fs = dir.getFileSystem(conf);
if (fs.exists(dir) && !fs.getFileStatus(dir).isDirectory()) {
  throw new IOException(dir + " exists as a FILE — move it first");
}
if (!fs.mkdirs(dir)) {
  throw new IOException("Cannot create " + dir + " — check perms/quota/safe-mode");
}

FileSystem fs = dir.getFileSystem(conf);
if (fs.exists(dir) && !fs.getFileStatus(dir).isDirectory()) {
  throw new IOException(dir + " exists as a FILE — remove it first");
}
if (!fs.mkdirs(dir)) {
  throw new IOException("Pre-check mkdirs failed for " + dir
      + " — perms/quota/safe-mode?");
}
new MapFile.Writer(conf, dir, MapFile.Writer.keyClass(Text.class));

try {
  writer = new MapFile.Writer(conf, dir, opts);
} catch (IOException e) {
  if (e.getMessage() != null && e.getMessage().startsWith("Mkdirs failed")) {
    // inspect perms on parent, HDFS safemode, quota; clean target and retry once
    diagnoseAndClean(dir, fs);
    writer = new MapFile.Writer(conf, dir, opts); // single retry after fix
  } else { throw e; }
}

When it happens

Trigger: Creating a MapFile.Writer at a path where the HDFS/local user lacks write/execute permission; a parent in the path is a regular file; the target already exists as a file (not a directory); directory quota is exhausted; safe mode on HDFS rejecting mutations.

Common situations: MapReduce/reduce-side output dirs under a path owned by another user; running as a user without rights to /user/<other>; leftover files from a previous failed run occupying the path; HDFS in safe mode or namenode unavailable causing mkdirs to fail.

Related errors

• Could not rename {oldDir} to {newDir}
• Cannot create directory {}
• Mkdirs failed to create {}
• Mkdirs failed to create ${dir}
• Permission denied: user=%s, path="%s":%s:%s:%s%s

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/87bd09bd58694390. Report an issue: GitHub.