{"record":{"id":"87bd09bd58694390","repo":"apache/hadoop","slug":"mkdirs-failed-to-create-directory-dirname","errorCode":null,"errorMessage":"Mkdirs failed to create directory {dirName}","messagePattern":"Mkdirs failed to create directory (.+?)","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/MapFile.java","lineNumber":340,"sourceCode":"        throw new IllegalArgumentException(\"key class or comparator option \"\n                                           + \"must be set\");\n      }\n      this.indexInterval = conf.getInt(INDEX_INTERVAL, this.indexInterval);\n\n      Class<? extends WritableComparable> keyClass;\n      if (keyClassOption == null) {\n        this.comparator = comparatorOption.getValue();\n        keyClass = comparator.getKeyClass();\n      } else {\n        keyClass= \n          (Class<? extends WritableComparable>) keyClassOption.getValue();\n        this.comparator = WritableComparator.get(keyClass, conf);\n      }\n      this.lastKey = comparator.newKey();\n      FileSystem fs = dirName.getFileSystem(conf);\n\n      if (!fs.mkdirs(dirName)) {\n        throw new IOException(\"Mkdirs failed to create directory \" + dirName);\n      }\n      Path dataFile = new Path(dirName, DATA_FILE_NAME);\n      Path indexFile = new Path(dirName, INDEX_FILE_NAME);\n\n      SequenceFile.Writer.Option[] dataOptions =\n        Options.prependOptions(opts, \n                               SequenceFile.Writer.file(dataFile),\n                               SequenceFile.Writer.keyClass(keyClass));\n      this.data = SequenceFile.createWriter(conf, dataOptions);\n\n      SequenceFile.Writer.Option[] indexOptions =\n        Options.prependOptions(opts, SequenceFile.Writer.file(indexFile),\n            SequenceFile.Writer.keyClass(keyClass),\n            SequenceFile.Writer.valueClass(LongWritable.class),\n            SequenceFile.Writer.compression(CompressionType.BLOCK));\n      this.index = SequenceFile.createWriter(conf, indexOptions);      \n    }\n","sourceCodeStart":322,"sourceCodeEnd":358,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/MapFile.java#L322-L358","documentation":"Thrown by the MapFile.Writer constructor when FileSystem.mkdirs(dirName) returns false — the output directory for the map (which will hold 'data' and 'index' files) could not be created. Hadoop's mkdirs returns false rather than throwing for many failure modes (permission denied, a parent existing as a regular file, quota exceeded), so this IOException is the first concrete signal.","triggerScenarios":"Creating a MapFile.Writer at a path where the HDFS/local user lacks write/execute permission; a parent in the path is a regular file; the target already exists as a file (not a directory); directory quota is exhausted; safe mode on HDFS rejecting mutations.","commonSituations":"MapReduce/reduce-side output dirs under a path owned by another user; running as a user without rights to /user/<other>; leftover files from a previous failed run occupying the path; HDFS in safe mode or namenode unavailable causing mkdirs to fail.","solutions":["Check permissions on the parent path: hadoop fs -ls the parent; fix with hadoop fs -chmod/-chown or choose a path you own.","Remove or move any existing regular file occupying the target path.","Verify HDFS is healthy and out of safe mode (hdfs dfsadmin -safemode get) and quotas are not exceeded (hdfs dfsadmin -quota).","If failures may be transient, catch the IOException and retry Writer construction after a delay."],"exampleFix":"// before: fails with only 'Mkdirs failed to create directory /user/bob/out'\nnew MapFile.Writer(conf, new Path(\"/user/bob/out\"),\n    MapFile.Writer.keyClass(Text.class), MapFile.Writer.valueClass(Text.class));\n\n// after: pre-flight the target so the failure names the real cause\nPath dir = new Path(\"/user/bob/out\");\nFileSystem fs = dir.getFileSystem(conf);\nif (fs.exists(dir) && !fs.getFileStatus(dir).isDirectory()) {\n  throw new IOException(dir + \" exists as a FILE — move it first\");\n}\nif (!fs.mkdirs(dir)) {\n  throw new IOException(\"Cannot create \" + dir + \" — check perms/quota/safe-mode\");\n}","handlingStrategy":"try-catch","validationCode":"FileSystem fs = dir.getFileSystem(conf);\nif (fs.exists(dir) && !fs.getFileStatus(dir).isDirectory()) {\n  throw new IOException(dir + \" exists as a FILE — remove it first\");\n}\nif (!fs.mkdirs(dir)) {\n  throw new IOException(\"Pre-check mkdirs failed for \" + dir\n      + \" — perms/quota/safe-mode?\");\n}\nnew MapFile.Writer(conf, dir, MapFile.Writer.keyClass(Text.class));","typeGuard":null,"tryCatchPattern":"try {\n  writer = new MapFile.Writer(conf, dir, opts);\n} catch (IOException e) {\n  if (e.getMessage() != null && e.getMessage().startsWith(\"Mkdirs failed\")) {\n    // inspect perms on parent, HDFS safemode, quota; clean target and retry once\n    diagnoseAndClean(dir, fs);\n    writer = new MapFile.Writer(conf, dir, opts); // single retry after fix\n  } else { throw e; }\n}","preventionTips":["Pre-flight the output path: exists-as-file check, parent permissions, and quota before constructing the Writer.","In job drivers, clean stale output dirs (or refuse to overwrite) rather than letting the Writer hit them.","For transient FS conditions, one bounded retry after diagnosis beats an immediate hard fail."],"tags":["mapfile","mkdirs","permissions","hdfs","hadoop-common"],"backgroundTag":"mkdirs-failed","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-23T01:17:44.959Z"}