apache/hadoop · error · IOException

Copy of file ${file} size ${file.length()} into file ${tmpFi

Error message

Copy of file ${file} size ${file.length()} into file ${tmpFile} resulted in a size of ${tmpFile.length()}

What it means

During breakHardLinksIfNeeded(), LocalReplica copies a hard-linked block file to an unlink-tmp file and then verifies the copied size matches the source. A mismatch means the copy did not reproduce the source byte-for-byte — the source changed length while being read (concurrent truncate/append), or the read/write silently shorted (failing disk, full volume, odd NFS semantics). The tmp file is deleted and the IOException propagates.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/LocalReplica.java:203

   * temporary file to the original name. This will cause any
   * hardlinks to the original file to be removed. The temporary
   * files are created in the same directory. The temporary files will
   * be recovered (especially on Windows) on datanode restart.
   */
  private void breakHardlinks(File file, Block b) throws IOException {
    final FileIoProvider fileIoProvider = getFileIoProvider();
    final File tmpFile = DatanodeUtil.createFileWithExistsCheck(
        getVolume(), b, DatanodeUtil.getUnlinkTmpFile(file), fileIoProvider);
    try {
      try (FileInputStream in = fileIoProvider.getFileInputStream(
          getVolume(), file)) {
        try (FileOutputStream out = fileIoProvider.getFileOutputStream(
            getVolume(), tmpFile)) {
          IOUtils.copyBytes(in, out, 16 * 1024);
        }
      }
      if (file.length() != tmpFile.length()) {
        throw new IOException("Copy of file " + file + " size " + file.length()
            + " into file " + tmpFile + " resulted in a size of "
            + tmpFile.length());
      }
      fileIoProvider.replaceFile(getVolume(), tmpFile, file);
    } catch (IOException e) {
      if (!fileIoProvider.delete(getVolume(), tmpFile)) {
        DataNode.LOG.info("detachFile failed to delete temporary file " +
                          tmpFile);
      }
      throw e;
    }
  }

  /**
   * This function "breaks hardlinks" to the current replica file.
   *
   * When doing a DataNode upgrade, we create a bunch of hardlinks to each block
   * file.  This cleverly ensures that both the old and the new storage

View on GitHub (pinned to 2add963021)

Solutions

1. Check the volume for health and headroom (`df -h`, dmesg/SMART); free space, then retry the upgrade/detach.
2. Ensure the replica is quiescent (no append/recovery in flight) before running the detach; retry once it is.
3. If one replica persistently fails, re-replicate the block from other replicas and delete the suspect replica or volume — the local copy may be corrupt.
4. Capture DataNode logs and file a JIRA if a verified-healthy disk reproduces it.

// Predict ENOSPC before detach flows
File v = new File(volume.getBaseURI());
if (v.getUsableSpace() < replica.getNumBytes()) {
  // free space on the volume before copying block files
}

try {
  replica.breakHardLinksIfNeeded();
} catch (IOException e) {
  if (e.getMessage().contains("resulted in a size of")) {
    // verify disk health, wait for the replica to be quiescent, retry once;
    // persistent failure -> re-replicate the block and drop this replica
  }
}

When it happens

Trigger: Upgrade/downgrade flows calling breakHardLinksIfNeeded() while another thread mutates the same block file; ENOSPC during the copy; a storage layer returning short reads without error.

Common situations: Rolling upgrades with hard-linked block files on nearly-full volumes; marginal disks producing silent corruption; a writer (append/recovery) racing the detach on the same replica.

Related errors

• detachBlock:Block not found. ${this}
• Failed to hardLink {srcReplica} block file to {dstFile}
• Failed to hardLink {srcReplica} metadata to {dstMeta}
• The meta file length {metaIn.getLength()} is less than the e
• Failed to mkdirs {}

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/0873594673e713c7. Report an issue: GitHub.