{"record":{"id":"31f0e15c2dd0a620","repo":"apache/hadoop","slug":"mkdir-failed-for-target","errorCode":null,"errorMessage":"mkdir failed for {target}","messagePattern":"mkdir failed for (.+?)","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/mapred/CopyMapper.java","lineNumber":304,"sourceCode":"    incrementCounter(context, Counter.COPY, 1);\n    totalBytesCopied += bytesCopied;\n\n    if (verboseLog) {\n      context.write(null,\n          new Text(\"FILE_COPIED: source=\" + sourceFileStatus.getPath() + \",\"\n          + \" size=\" + sourceFileStatus.getLen() + \" --> \"\n          + \"target=\" + target + \", size=\" + (targrtFileStatus == null ?\n              0 : targrtFileStatus.getLen())));\n    }\n  }\n\n  private void createTargetDirsWithRetry(String description, Path target,\n      Context context, FileStatus sourceStatus, FileSystem sourceFS) throws IOException {\n    try {\n      new RetriableDirectoryCreateCommand(description).execute(target, context,\n          sourceStatus, sourceFS);\n    } catch (Exception e) {\n      throw new IOException(\"mkdir failed for \" + target, e);\n    }\n    incrementCounter(context, Counter.DIR_COPY, 1);\n  }\n\n  private static void updateSkipCounters(Context context,\n      CopyListingFileStatus sourceFile) {\n    incrementCounter(context, Counter.SKIP, 1);\n    incrementCounter(context, Counter.BYTESSKIPPED, sourceFile.getLen());\n\n  }\n\n  private void handleFailures(IOException exception,\n      CopyListingFileStatus sourceFileStatus, Path target, Context context)\n      throws IOException, InterruptedException {\n    LOG.error(\"Failure in copying \" + sourceFileStatus.getPath() +\n        (sourceFileStatus.isSplit()? \",\"\n            + \" offset=\" + sourceFileStatus.getChunkOffset()\n            + \" chunkLength=\" + sourceFileStatus.getChunkLength()","sourceCodeStart":286,"sourceCodeEnd":322,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/mapred/CopyMapper.java#L286-L322","documentation":"CopyMapper creates target directories via RetriableDirectoryCreateCommand; if the command still throws after its retries (the underlying mkdirs failed), the mapper rethrows wrapped as 'mkdir failed for <target>'. The nested cause carries the FileSystem-level error.","triggerScenarios":"No write permission on the target's parent, HDFS namespace or space quota exceeded, an existing FILE occupies the path where a directory is needed, NameNode in safe mode or unreachable, or the parent path deleted while the job ran.","commonSituations":"Destination owned by another user; HDFS quota exhausted; a stale file blocking a directory path; transient NameNode unavailability outlasting the retry policy.","solutions":["Read the nested cause for the FileSystem error; check and fix write permission on the target parent","Check quotas (hdfs dfs -count / hdfs dfsadmin -setQuota) and raise them if exhausted","Remove any file occupying the directory path, then re-run with -update","Confirm the NameNode is reachable and out of safe mode before re-running"],"exampleFix":null,"handlingStrategy":"validation","validationCode":"// before submit: ensure a directory can be created at the target root\nPath parent = targetRoot.getParent();\nif (!targetFS.exists(parent) && !targetFS.mkdirs(parent)) {\n  throw new IOException(\"Cannot create target parent: \" + parent);\n}\nif (targetFS.exists(targetRoot) && !targetFS.getFileStatus(targetRoot).isDirectory()) {\n  throw new IOException(\"Target occupied by a file: \" + targetRoot);\n}","typeGuard":null,"tryCatchPattern":"catch (IOException e) {  // wraps 'mkdir failed for <target>'\n  Throwable root = e.getCause();  // underlying FS error: permission/quota/safe-mode\n  LOG.error(\"mkdir failed, cause:\", root);\n}","preventionTips":["Verify write permission on the target's parent before submitting","Pre-check namespace and space quotas on the destination","Remove files that occupy directory paths in the target tree","Run jobs when the NameNode is stable and reachable"],"tags":["distcp","hadoop","mkdir","permissions","quota"],"backgroundTag":"mkdir-failed","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}