{"record":{"id":"c5246a772fec0ce7","repo":"apache/hadoop","slug":"failed-to-hardlink-srcreplica-block-file-to-dst","errorCode":null,"errorMessage":"Failed to hardLink {srcReplica} block file to {dstFile}","messagePattern":"Failed to hardLink (.+?) block file to (.+?)","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java","lineNumber":1060,"sourceCode":"    return new File[] {dstMeta, dstFile};\n  }\n\n  static File[] hardLinkBlockFiles(ReplicaInfo srcReplica, File dstMeta,\n      File dstFile)\n      throws IOException {\n    FsVolumeSpi srcReplicaVolume = srcReplica.getVolume();\n    File destParentFile = dstFile.getParentFile();\n    // Create parent folder if not exists.\n    boolean isDirCreated = srcReplica.getFileIoProvider()\n        .mkdirs(srcReplicaVolume, destParentFile);\n    LOG.trace(\"Dir creation of {} on volume {} {}\", destParentFile,\n        srcReplicaVolume, isDirCreated ? \"succeeded\" : \"failed\");\n    URI srcReplicaUri = srcReplica.getBlockURI();\n    try {\n      HardLink.createHardLink(\n          new File(srcReplicaUri), dstFile);\n    } catch (IOException e) {\n      throw new IOException(\"Failed to hardLink \"\n          + srcReplica + \" block file to \"\n          + dstFile, e);\n    }\n    try {\n      HardLink.createHardLink(\n          new File(srcReplica.getMetadataURI()), dstMeta);\n    } catch (IOException e) {\n      throw new IOException(\"Failed to hardLink \"\n          + srcReplica + \" metadata to \"\n          + dstMeta, e);\n    }\n    LOG.debug(\"Linked {} to {} . Dest meta file: {}\", srcReplicaUri, dstFile,\n        dstMeta);\n    return new File[]{dstMeta, dstFile};\n  }\n\n  /**\n   * Move block files from one storage to another storage.","sourceCodeStart":1042,"sourceCodeEnd":1078,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java#L1042-L1078","documentation":"Thrown as IOException from the hard-link variant of block-file duplication in FsDatasetImpl (~line 1060) when HardLink.createHardLink(new File(srcReplicaUri), dstFile) fails. This path is chosen when a replica can be duplicated without a full copy (same filesystem, e.g. providing a local replica to a reader or mover on the same volume); the parent dir is mkdirs'd first (logged at trace), then the block file is hard-linked, and any IOException from the link call is wrapped with cause and paths.","triggerScenarios":"Hard-linking a replica's block file where the link cannot be created: dst on a different filesystem than src (EXDEV — hard links never cross mounts), dst parent unwritable despite the mkdirs attempt, dst file already exists, or the filesystem does not support hard links (some object/FUSE mounts, NFS with odd export settings).","commonSituations":"Two data dirs configured on the same mount in a tiering setup but actually on different underlying devices; overlayfs/container filesystems restricting link; destination subdir created by another thread concurrently; leftover dst from an interrupted previous attempt.","solutions":["Check the cause code: EXDEV means src and dst are on different filesystems — fall back to the copy path (the caller of the hardlink helper decides; see copyBlockFiles) or configure the volumes appropriately.","Verify the DataNode user can write and link in the destination parent dir (ls -ld on the dst path from the message).","Remove a stale existing dstFile from an earlier failed attempt.","If the FS can't hard-link (FUSE/object store), move that data dir to a real local filesystem or disable the hardlink code path."],"exampleFix":null,"handlingStrategy":"fallback","validationCode":"// Only attempt the hardlink path when src and dst share a store.\ntry {\n  FileStore srcStore = Files.getFileStore(\n      Paths.get(srcReplica.getBlockURI()));\n  FileStore dstStore = Files.getFileStore(dstFile.toPath());\n  if (!srcStore.equals(dstStore)) {\n    return copyInstead(srcReplica, dstFile); // different FS: link impossible\n  }\n} catch (IOException ignore) { // fall through to link attempt\n}","typeGuard":null,"tryCatchPattern":"// Link failed: fall back to full copy — the standard remedy for EXDEV.\ntry {\n  return hardLinkBlockFiles(srcReplica, dstFile, dstMeta);\n} catch (IOException e) {\n  LOG.debug(\"Hardlink unavailable ({}), falling back to copy\", e.getCause());\n  return FsDatasetImpl.copyBlockFiles(srcReplica, dstMeta, dstFile,\n      false, bufSize, conf);\n}","preventionTips":["Verify src/dst are on the same filesystem before hard-link code paths (Files.getFileStore comparison).","Avoid object-store/FUSE-backed data dirs where link(2) is unsupported.","On failure, prefer copy fallback over retrying the link — EXDEV never resolves by retry."],"tags":["hadoop","hdfs","datanode","fsdataset","hardlink","same-disk-tiering","io"],"backgroundTag":"hardlink-failed","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}