{"record":{"id":"0873594673e713c7","repo":"apache/hadoop","slug":"copy-of-file-file-size-file-length-into-fi","errorCode":null,"errorMessage":"Copy of file ${file} size ${file.length()} into file ${tmpFile} resulted in a size of ${tmpFile.length()}","messagePattern":"Copy of file (.+?) size (.+?) into file (.+?) resulted in a size of (.+?)","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/LocalReplica.java","lineNumber":203,"sourceCode":"   * temporary file to the original name. This will cause any\n   * hardlinks to the original file to be removed. The temporary\n   * files are created in the same directory. The temporary files will\n   * be recovered (especially on Windows) on datanode restart.\n   */\n  private void breakHardlinks(File file, Block b) throws IOException {\n    final FileIoProvider fileIoProvider = getFileIoProvider();\n    final File tmpFile = DatanodeUtil.createFileWithExistsCheck(\n        getVolume(), b, DatanodeUtil.getUnlinkTmpFile(file), fileIoProvider);\n    try {\n      try (FileInputStream in = fileIoProvider.getFileInputStream(\n          getVolume(), file)) {\n        try (FileOutputStream out = fileIoProvider.getFileOutputStream(\n            getVolume(), tmpFile)) {\n          IOUtils.copyBytes(in, out, 16 * 1024);\n        }\n      }\n      if (file.length() != tmpFile.length()) {\n        throw new IOException(\"Copy of file \" + file + \" size \" + file.length()\n            + \" into file \" + tmpFile + \" resulted in a size of \"\n            + tmpFile.length());\n      }\n      fileIoProvider.replaceFile(getVolume(), tmpFile, file);\n    } catch (IOException e) {\n      if (!fileIoProvider.delete(getVolume(), tmpFile)) {\n        DataNode.LOG.info(\"detachFile failed to delete temporary file \" +\n                          tmpFile);\n      }\n      throw e;\n    }\n  }\n\n  /**\n   * This function \"breaks hardlinks\" to the current replica file.\n   *\n   * When doing a DataNode upgrade, we create a bunch of hardlinks to each block\n   * file.  This cleverly ensures that both the old and the new storage","sourceCodeStart":185,"sourceCodeEnd":221,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/LocalReplica.java#L185-L221","documentation":"During breakHardLinksIfNeeded(), LocalReplica copies a hard-linked block file to an unlink-tmp file and then verifies the copied size matches the source. A mismatch means the copy did not reproduce the source byte-for-byte — the source changed length while being read (concurrent truncate/append), or the read/write silently shorted (failing disk, full volume, odd NFS semantics). The tmp file is deleted and the IOException propagates.","triggerScenarios":"Upgrade/downgrade flows calling breakHardLinksIfNeeded() while another thread mutates the same block file; ENOSPC during the copy; a storage layer returning short reads without error.","commonSituations":"Rolling upgrades with hard-linked block files on nearly-full volumes; marginal disks producing silent corruption; a writer (append/recovery) racing the detach on the same replica.","solutions":["Check the volume for health and headroom (`df -h`, dmesg/SMART); free space, then retry the upgrade/detach.","Ensure the replica is quiescent (no append/recovery in flight) before running the detach; retry once it is.","If one replica persistently fails, re-replicate the block from other replicas and delete the suspect replica or volume — the local copy may be corrupt.","Capture DataNode logs and file a JIRA if a verified-healthy disk reproduces it."],"exampleFix":null,"handlingStrategy":"retry","validationCode":"// Predict ENOSPC before detach flows\nFile v = new File(volume.getBaseURI());\nif (v.getUsableSpace() < replica.getNumBytes()) {\n  // free space on the volume before copying block files\n}","typeGuard":null,"tryCatchPattern":"try {\n  replica.breakHardLinksIfNeeded();\n} catch (IOException e) {\n  if (e.getMessage().contains(\"resulted in a size of\")) {\n    // verify disk health, wait for the replica to be quiescent, retry once;\n    // persistent failure -> re-replicate the block and drop this replica\n  }\n}","preventionTips":["Run upgrades/detaches only while replicas are quiescent (no appends or recovery in flight)","Keep data volumes below capacity so hard-link copies never hit ENOSPC","Surface SMART/dmesg disk errors before running block-copying maintenance operations"],"tags":["block","hardlink","io","disk-corruption","hdfs","datanode"],"backgroundTag":"file-copy-size-mismatch","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}