apache/hadoop · error · IOException

Failed to hardLink {srcReplica} metadata to {dstMeta}

Error message

Failed to hardLink {srcReplica} metadata to {dstMeta}

What it means

Thrown as IOException from the hard-link helper in FsDatasetImpl (~line 1068) when the second link — HardLink.createHardLink(new File(srcReplica.getMetadataURI()), dstMeta) — fails after the block-file link already succeeded. Cause is chained; message names srcReplica and dstMeta. A failure here leaves an orphan block-file link at dstFile which the caller's failure path is expected to clean up.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java:1068

    // Create parent folder if not exists.
    boolean isDirCreated = srcReplica.getFileIoProvider()
        .mkdirs(srcReplicaVolume, destParentFile);
    LOG.trace("Dir creation of {} on volume {} {}", destParentFile,
        srcReplicaVolume, isDirCreated ? "succeeded" : "failed");
    URI srcReplicaUri = srcReplica.getBlockURI();
    try {
      HardLink.createHardLink(
          new File(srcReplicaUri), dstFile);
    } catch (IOException e) {
      throw new IOException("Failed to hardLink "
          + srcReplica + " block file to "
          + dstFile, e);
    }
    try {
      HardLink.createHardLink(
          new File(srcReplica.getMetadataURI()), dstMeta);
    } catch (IOException e) {
      throw new IOException("Failed to hardLink "
          + srcReplica + " metadata to "
          + dstMeta, e);
    }
    LOG.debug("Linked {} to {} . Dest meta file: {}", srcReplicaUri, dstFile,
        dstMeta);
    return new File[]{dstMeta, dstFile};
  }

  /**
   * Move block files from one storage to another storage.
   * @return Returns the Old replicaInfo
   * @throws IOException
   */
  @Override
  public ReplicaInfo moveBlockAcrossStorage(ExtendedBlock block,
      StorageType targetStorageType, String targetStorageId)
      throws IOException {
    ReplicaInfo replicaInfo = getReplicaInfo(block);

View on GitHub (pinned to 2add963021)

Solutions

Confirm the source metadata URI from the message still exists on disk (ls the path); if the replica was deleted, abandon the operation — it will be re-driven.
Clean orphan dstFile/dstMeta pairs from the failed attempt before retrying.
Check inode availability on the destination: df -i <mount>; expand or clean the filesystem.
Re-run the triggering operation (read/mover); linking is idempotent once the leftovers are removed.

Defensive patterns

Strategy: fallback

Validate before calling

// Pre-verify both source files exist and dst paths are free before linking.
if (!new File(srcReplica.getMetadataURI()).exists()
    || dstMeta.exists()) {
  return Optional.empty(); // link set cannot proceed cleanly
}

Try / catch

// Second link failed: remove the orphan block link, fall back to copy.
try {
  return hardLinkBlockFiles(srcReplica, dstFile, dstMeta);
} catch (IOException e) {
  if (dstFile.exists()) Files.deleteIfExists(dstFile.toPath()); // orphan
  return FsDatasetImpl.copyBlockFiles(srcReplica, dstMeta, dstFile,
      false, bufSize, conf);
}

Prevention

Treat link helpers as two-phase: on phase-2 failure always unwind phase-1.
Pre-check dstMeta existence to avoid EEXIST from interrupted prior runs.
Run concurrent replica operations under the dataset lock so invalidation races cannot delete sources mid-link.

When it happens

Trigger: Same hardlink flow as 2412 with the meta link failing specifically: source .meta file missing (replica invalidated between the two links), dstMeta already present from an interrupted run, ENOSPC on inode creation, or permission change on the destination finalized subdir mid-operation.

Common situations: Concurrent block invalidation deleting the replica while a reader link is set up; retry after a crashed attempt leaving dstMeta behind; security software altering dir perms; inode exhaustion on the destination filesystem.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/e3b7010e7cb2016c. Report an issue: GitHub.