apache/hadoop · error · IOException

Destination '{parentFile}' directory cannot be created

Error message

Destination '{parentFile}' directory cannot be created

What it means

Thrown as IOException from FsDatasetImpl.computeChecksum (reached via copyBlockFiles with calculateChecksum=true) when the destination meta file's parent directory cannot be created: parentFile.mkdirs() returns false AND parentFile.isDirectory() is false — i.e. creation failed and it is not already a directory. The message contains the offending parent path.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java:1362

    final File srcMeta = new File(srcReplica.getMetadataURI());

    DataChecksum checksum;
    try (FileInputStream fis =
             srcReplica.getFileIoProvider().getFileInputStream(
                 srcReplica.getVolume(), srcMeta)) {
      checksum = BlockMetadataHeader.readDataChecksum(
          fis, DFSUtilClient.getIoFileBufferSize(conf), srcMeta);
    }

    final byte[] data = new byte[1 << 16];
    final byte[] crcs = new byte[checksum.getChecksumSize(data.length)];

    DataOutputStream metaOut = null;
    try {
      File parentFile = dstMeta.getParentFile();
      if (parentFile != null) {
        if (!parentFile.mkdirs() && !parentFile.isDirectory()) {
          throw new IOException("Destination '" + parentFile
              + "' directory cannot be created");
        }
      }
      metaOut = new DataOutputStream(new BufferedOutputStream(
          Files.newOutputStream(dstMeta.toPath()), smallBufferSize));
      BlockMetadataHeader.writeHeader(metaOut, checksum);

      int offset = 0;
      try (InputStream dataIn = srcReplica.getDataInputStream(0)) {

        for (int n; (n = dataIn.read(data, offset, data.length - offset)) != -1; ) {
          if (n > 0) {
            n += offset;
            offset = n % checksum.getBytesPerChecksum();
            final int length = n - offset;

            if (length > 0) {
              checksum.calculateChunkedSums(data, 0, length, crcs, 0);

View on GitHub (pinned to 2add963021)

Solutions

ls -ld the parent path from the message; if a regular file blocks the path, remove/rename it (usually debris from a failed restore).
chown -R hdfs:hadoop (or your DN user) the volume root and chmod u+w the parent chain.
mount/df the destination filesystem — read-only remounts and inode exhaustion (df -i) both fail mkdirs silently.
Retry the copy/move operation; with the parent creatable, checksum recompute proceeds.

Defensive patterns

Strategy: validation

Validate before calling

// Ensure dst meta parent exists and is a writable dir before checksum copy.
File parent = dstMeta.getParentFile();
if (parent != null) {
  Files.createDirectories(parent.toPath()); // throws a descriptive reason
  if (!parent.isDirectory() || !parent.canWrite()) {
    throw new IOException("Cannot use destination dir: " + parent);
  }
}

Try / catch

// Surface the failing parent path to operators; retry after repair.
try {
  fsDataset.copyBlockFiles(src, dstMeta, dstFile, true, bufSize, conf);
} catch (IOException e) {
  if (String.valueOf(e.getMessage()).contains("cannot be created")) {
    LOG.error("Destination dir unusable (see {}), fix mount/permissions",
        dstMeta.getParent());
  }
  throw e;
}

Prevention

Pre-create destination dir trees with Files.createDirectories (which reports the real cause) instead of relying on mkdirs().
After any volume restore/clone, verify no regular file occupies a directory path in the data-dir tree.
Mount data volumes read-write and alert on read-only remounts (a common precursor to silent mkdirs failure).

When it happens

Trigger: Recomputing a replica's checksum into dstMeta on a volume where the parent subdir (e.g. .../current/finalized subdir or a lazy-persist destination) cannot be created: permissions deny mkdir, a non-directory file exists at the parent path, the filesystem is read-only or out of inodes.

Common situations: Destination volume mounted read-only after an array failure; a stray file occupies the dir path (partial restore/copy of data dirs); DataNode user lost group ownership on the volume; inode exhaustion on small filesystems.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/167dd0bfd8332417. Report an issue: GitHub.