{"record":{"id":"e3b7010e7cb2016c","repo":"apache/hadoop","slug":"failed-to-hardlink-srcreplica-metadata-to-dstme","errorCode":null,"errorMessage":"Failed to hardLink {srcReplica} metadata to {dstMeta}","messagePattern":"Failed to hardLink (.+?) metadata to (.+?)","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java","lineNumber":1068,"sourceCode":"    // Create parent folder if not exists.\n    boolean isDirCreated = srcReplica.getFileIoProvider()\n        .mkdirs(srcReplicaVolume, destParentFile);\n    LOG.trace(\"Dir creation of {} on volume {} {}\", destParentFile,\n        srcReplicaVolume, isDirCreated ? \"succeeded\" : \"failed\");\n    URI srcReplicaUri = srcReplica.getBlockURI();\n    try {\n      HardLink.createHardLink(\n          new File(srcReplicaUri), dstFile);\n    } catch (IOException e) {\n      throw new IOException(\"Failed to hardLink \"\n          + srcReplica + \" block file to \"\n          + dstFile, e);\n    }\n    try {\n      HardLink.createHardLink(\n          new File(srcReplica.getMetadataURI()), dstMeta);\n    } catch (IOException e) {\n      throw new IOException(\"Failed to hardLink \"\n          + srcReplica + \" metadata to \"\n          + dstMeta, e);\n    }\n    LOG.debug(\"Linked {} to {} . Dest meta file: {}\", srcReplicaUri, dstFile,\n        dstMeta);\n    return new File[]{dstMeta, dstFile};\n  }\n\n  /**\n   * Move block files from one storage to another storage.\n   * @return Returns the Old replicaInfo\n   * @throws IOException\n   */\n  @Override\n  public ReplicaInfo moveBlockAcrossStorage(ExtendedBlock block,\n      StorageType targetStorageType, String targetStorageId)\n      throws IOException {\n    ReplicaInfo replicaInfo = getReplicaInfo(block);","sourceCodeStart":1050,"sourceCodeEnd":1086,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java#L1050-L1086","documentation":"Thrown as IOException from the hard-link helper in FsDatasetImpl (~line 1068) when the second link — HardLink.createHardLink(new File(srcReplica.getMetadataURI()), dstMeta) — fails after the block-file link already succeeded. Cause is chained; message names srcReplica and dstMeta. A failure here leaves an orphan block-file link at dstFile which the caller's failure path is expected to clean up.","triggerScenarios":"Same hardlink flow as 2412 with the meta link failing specifically: source .meta file missing (replica invalidated between the two links), dstMeta already present from an interrupted run, ENOSPC on inode creation, or permission change on the destination finalized subdir mid-operation.","commonSituations":"Concurrent block invalidation deleting the replica while a reader link is set up; retry after a crashed attempt leaving dstMeta behind; security software altering dir perms; inode exhaustion on the destination filesystem.","solutions":["Confirm the source metadata URI from the message still exists on disk (ls the path); if the replica was deleted, abandon the operation — it will be re-driven.","Clean orphan dstFile/dstMeta pairs from the failed attempt before retrying.","Check inode availability on the destination: df -i <mount>; expand or clean the filesystem.","Re-run the triggering operation (read/mover); linking is idempotent once the leftovers are removed."],"exampleFix":null,"handlingStrategy":"fallback","validationCode":"// Pre-verify both source files exist and dst paths are free before linking.\nif (!new File(srcReplica.getMetadataURI()).exists()\n    || dstMeta.exists()) {\n  return Optional.empty(); // link set cannot proceed cleanly\n}","typeGuard":null,"tryCatchPattern":"// Second link failed: remove the orphan block link, fall back to copy.\ntry {\n  return hardLinkBlockFiles(srcReplica, dstFile, dstMeta);\n} catch (IOException e) {\n  if (dstFile.exists()) Files.deleteIfExists(dstFile.toPath()); // orphan\n  return FsDatasetImpl.copyBlockFiles(srcReplica, dstMeta, dstFile,\n      false, bufSize, conf);\n}","preventionTips":["Treat link helpers as two-phase: on phase-2 failure always unwind phase-1.","Pre-check dstMeta existence to avoid EEXIST from interrupted prior runs.","Run concurrent replica operations under the dataset lock so invalidation races cannot delete sources mid-link."],"tags":["hadoop","hdfs","datanode","fsdataset","hardlink","metadata-file","io"],"backgroundTag":"hardlink-failed","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}