apache/beam · error · IOException

Unable to create target directory

Error message

Unable to create target directory %s. No further information provided by underlying filesystem.

What it means

The private mkdirs() helper throws this IOException when fs.mkdirs() returns false for the parent directory of a file being created/renamed. As with other Hadoop boolean returns, no underlying reason is available, so Beam reports it plainly.

Solutions

Check and grant write permission on the parent directory for the job's user.
Ensure no component of the target path already exists as a file.
Check HDFS quota and NameNode logs for the suppressed failure cause.

Example fix

// before
writeTo("hdfs://nn/data/output/result.txt"); // /data exists as a file
// after
writeTo("hdfs://nn/data/output-2026/result.txt"); // unique, valid dir path

Defensive patterns

Strategy: validation

Validate before calling

Path parent = new Path(filePath).getParent();
FileSystem fs = parent.getFileSystem(conf);
if (fs.exists(parent) && fs.getFileStatus(parent).isFile()) {
  throw new IllegalStateException("Parent path exists as a file: " + parent);
}

Try / catch

try {
  fileSystem.rename(srcs, dests);
} catch (IOException e) {
  if (e.getMessage().contains("Unable to create target directory")) {
    fixPermissionsAndRetry(dest);
  }
}

Prevention

Verify write permission on destination parent directories
Ensure no path component collides with an existing file
Monitor HDFS quota for output volumes

When it happens

Trigger: Calling operations that internally create parent directories (rename, or create paths whose parents are missing) where fs.mkdirs(targetDirectory) fails — e.g. permission denied on the parent, or the path exists as a file.

Common situations: Output paths under directories the job user cannot write to; a path component existing as a regular file so a directory cannot be created there; HDFS quota exceeded.

Understand the failure class

Background: mkdir permission denied (EACCES): failed to create directory errors explained — this error's family across 32 libraries.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/c679f245139c0dd6. Report an issue: GitHub.

Appendix: source

Thrown at sdks/java/io/hadoop-file-system/src/main/java/org/apache/beam/sdk/io/hdfs/HadoopFileSystem.java:307

        } else {
          throw new IOException(
              String.format(
                  "Unable to rename resource %s to %s. No further information provided by underlying filesystem.",
                  srcPath, destPath));
        }
      }
    }
  }

  /** Ensures that the target directory exists for the given filePath. */
  private void mkdirs(Path filePath) throws IOException {
    final org.apache.hadoop.fs.FileSystem fs = filePath.getFileSystem(configuration);
    final Path targetDirectory = filePath.getParent();
    if (!fs.exists(targetDirectory)) {
      LOG.debug(LOG_CREATE_DIRECTORY, Path.getPathWithoutSchemeAndAuthority(targetDirectory));
      if (!fs.mkdirs(targetDirectory)) {
        throw new IOException(
            String.format(
                "Unable to create target directory %s. No further information provided by underlying filesystem.",
                targetDirectory));
      }
    }
  }

  @Override
  protected void delete(Collection<HadoopResourceId> resourceIds) throws IOException {
    for (HadoopResourceId resourceId : resourceIds) {
      // ignore response as issues are surfaced with exception
      final Path resourcePath = resourceId.toPath();
      resourcePath.getFileSystem(configuration).delete(resourceId.toPath(), false);
    }
  }

  @Override
  protected HadoopResourceId matchNewResource(String singleResourceSpec, boolean isDirectory) {

View on GitHub (pinned to 12126d8942)