apache/hadoop · error · IOException

Failed to hardLink {srcReplica} block file to {dstFile}

Error message

Failed to hardLink {srcReplica} block file to {dstFile}

What it means

Thrown as IOException from the hard-link variant of block-file duplication in FsDatasetImpl (~line 1060) when HardLink.createHardLink(new File(srcReplicaUri), dstFile) fails. This path is chosen when a replica can be duplicated without a full copy (same filesystem, e.g. providing a local replica to a reader or mover on the same volume); the parent dir is mkdirs'd first (logged at trace), then the block file is hard-linked, and any IOException from the link call is wrapped with cause and paths.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java:1060

    return new File[] {dstMeta, dstFile};
  }

  static File[] hardLinkBlockFiles(ReplicaInfo srcReplica, File dstMeta,
      File dstFile)
      throws IOException {
    FsVolumeSpi srcReplicaVolume = srcReplica.getVolume();
    File destParentFile = dstFile.getParentFile();
    // Create parent folder if not exists.
    boolean isDirCreated = srcReplica.getFileIoProvider()
        .mkdirs(srcReplicaVolume, destParentFile);
    LOG.trace("Dir creation of {} on volume {} {}", destParentFile,
        srcReplicaVolume, isDirCreated ? "succeeded" : "failed");
    URI srcReplicaUri = srcReplica.getBlockURI();
    try {
      HardLink.createHardLink(
          new File(srcReplicaUri), dstFile);
    } catch (IOException e) {
      throw new IOException("Failed to hardLink "
          + srcReplica + " block file to "
          + dstFile, e);
    }
    try {
      HardLink.createHardLink(
          new File(srcReplica.getMetadataURI()), dstMeta);
    } catch (IOException e) {
      throw new IOException("Failed to hardLink "
          + srcReplica + " metadata to "
          + dstMeta, e);
    }
    LOG.debug("Linked {} to {} . Dest meta file: {}", srcReplicaUri, dstFile,
        dstMeta);
    return new File[]{dstMeta, dstFile};
  }

  /**
   * Move block files from one storage to another storage.

View on GitHub (pinned to 2add963021)

Solutions

Check the cause code: EXDEV means src and dst are on different filesystems — fall back to the copy path (the caller of the hardlink helper decides; see copyBlockFiles) or configure the volumes appropriately.
Verify the DataNode user can write and link in the destination parent dir (ls -ld on the dst path from the message).
Remove a stale existing dstFile from an earlier failed attempt.
If the FS can't hard-link (FUSE/object store), move that data dir to a real local filesystem or disable the hardlink code path.

Defensive patterns

Strategy: fallback

Validate before calling

// Only attempt the hardlink path when src and dst share a store.
try {
  FileStore srcStore = Files.getFileStore(
      Paths.get(srcReplica.getBlockURI()));
  FileStore dstStore = Files.getFileStore(dstFile.toPath());
  if (!srcStore.equals(dstStore)) {
    return copyInstead(srcReplica, dstFile); // different FS: link impossible
  }
} catch (IOException ignore) { // fall through to link attempt
}

Try / catch

// Link failed: fall back to full copy — the standard remedy for EXDEV.
try {
  return hardLinkBlockFiles(srcReplica, dstFile, dstMeta);
} catch (IOException e) {
  LOG.debug("Hardlink unavailable ({}), falling back to copy", e.getCause());
  return FsDatasetImpl.copyBlockFiles(srcReplica, dstMeta, dstFile,
      false, bufSize, conf);
}

Prevention

Verify src/dst are on the same filesystem before hard-link code paths (Files.getFileStore comparison).
Avoid object-store/FUSE-backed data dirs where link(2) is unsupported.
On failure, prefer copy fallback over retrying the link — EXDEV never resolves by retry.

When it happens

Trigger: Hard-linking a replica's block file where the link cannot be created: dst on a different filesystem than src (EXDEV — hard links never cross mounts), dst parent unwritable despite the mkdirs attempt, dst file already exists, or the filesystem does not support hard links (some object/FUSE mounts, NFS with odd export settings).

Common situations: Two data dirs configured on the same mount in a tiering setup but actually on different underlying devices; overlayfs/container filesystems restricting link; destination subdir created by another thread concurrently; leftover dst from an interrupted previous attempt.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/c5246a772fec0ce7. Report an issue: GitHub.