{"record":{"id":"c679f245139c0dd6","repo":"apache/beam","slug":"unable-to-create-target-directory-s-no-further-information","errorCode":null,"errorMessage":"Unable to create target directory %s. No further information provided by underlying filesystem.","messagePattern":"Unable to create target directory (.+?)\\. No further information provided by underlying filesystem\\.","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"sdks/java/io/hadoop-file-system/src/main/java/org/apache/beam/sdk/io/hdfs/HadoopFileSystem.java","lineNumber":307,"sourceCode":"\n        } else {\n          throw new IOException(\n              String.format(\n                  \"Unable to rename resource %s to %s. No further information provided by underlying filesystem.\",\n                  srcPath, destPath));\n        }\n      }\n    }\n  }\n\n  /** Ensures that the target directory exists for the given filePath. */\n  private void mkdirs(Path filePath) throws IOException {\n    final org.apache.hadoop.fs.FileSystem fs = filePath.getFileSystem(configuration);\n    final Path targetDirectory = filePath.getParent();\n    if (!fs.exists(targetDirectory)) {\n      LOG.debug(LOG_CREATE_DIRECTORY, Path.getPathWithoutSchemeAndAuthority(targetDirectory));\n      if (!fs.mkdirs(targetDirectory)) {\n        throw new IOException(\n            String.format(\n                \"Unable to create target directory %s. No further information provided by underlying filesystem.\",\n                targetDirectory));\n      }\n    }\n  }\n\n  @Override\n  protected void delete(Collection<HadoopResourceId> resourceIds) throws IOException {\n    for (HadoopResourceId resourceId : resourceIds) {\n      // ignore response as issues are surfaced with exception\n      final Path resourcePath = resourceId.toPath();\n      resourcePath.getFileSystem(configuration).delete(resourceId.toPath(), false);\n    }\n  }\n\n  @Override\n  protected HadoopResourceId matchNewResource(String singleResourceSpec, boolean isDirectory) {","sourceCodeStart":289,"sourceCodeEnd":325,"githubUrl":"https://github.com/apache/beam/blob/12126d8942aaf848030c478b4c6a28c6af861c66/sdks/java/io/hadoop-file-system/src/main/java/org/apache/beam/sdk/io/hdfs/HadoopFileSystem.java#L289-L325","documentation":"The private mkdirs() helper throws this IOException when fs.mkdirs() returns false for the parent directory of a file being created/renamed. As with other Hadoop boolean returns, no underlying reason is available, so Beam reports it plainly.","triggerScenarios":"Calling operations that internally create parent directories (rename, or create paths whose parents are missing) where fs.mkdirs(targetDirectory) fails — e.g. permission denied on the parent, or the path exists as a file.","commonSituations":"Output paths under directories the job user cannot write to; a path component existing as a regular file so a directory cannot be created there; HDFS quota exceeded.","solutions":["Check and grant write permission on the parent directory for the job's user.","Ensure no component of the target path already exists as a file.","Check HDFS quota and NameNode logs for the suppressed failure cause."],"exampleFix":"// before\nwriteTo(\"hdfs://nn/data/output/result.txt\"); // /data exists as a file\n// after\nwriteTo(\"hdfs://nn/data/output-2026/result.txt\"); // unique, valid dir path","handlingStrategy":"validation","validationCode":"Path parent = new Path(filePath).getParent();\nFileSystem fs = parent.getFileSystem(conf);\nif (fs.exists(parent) && fs.getFileStatus(parent).isFile()) {\n  throw new IllegalStateException(\"Parent path exists as a file: \" + parent);\n}","typeGuard":null,"tryCatchPattern":"try {\n  fileSystem.rename(srcs, dests);\n} catch (IOException e) {\n  if (e.getMessage().contains(\"Unable to create target directory\")) {\n    fixPermissionsAndRetry(dest);\n  }\n}","preventionTips":["Verify write permission on destination parent directories","Ensure no path component collides with an existing file","Monitor HDFS quota for output volumes"],"tags":["hadoop","hdfs","mkdir","beam"],"backgroundTag":"mkdir-permission-denied","analyzedSha":"12126d8942aaf848030c478b4c6a28c6af861c66","analyzedAt":"2026-09-13T01:50:10.254Z","contentChangedAt":"2026-09-13T01:50:10.254Z","schemaVersion":2},"datasetVersion":"2026-09-20T03:17:13.778Z"}