apache/hadoop · error · IOException

Failed to copy {srcReplica} block file to {dstFile}

Error message

Failed to copy {srcReplica} block file to {dstFile}

What it means

Thrown as IOException from FsDatasetImpl.copyBlockFiles when srcReplica.copyBlockdata(dstFile) fails after the metadata copy step already ran. The cause is chained and the message names srcReplica and dstFile. Callers get an all-or-nothing signal: a data copy failure leaves a possibly-complete meta file at dstMeta that the caller must discard, which the lazy-persist/move callers handle by aborting the whole replica operation.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java:1031

  static File[] copyBlockFiles(ReplicaInfo srcReplica, File dstMeta,
                               File dstFile, boolean calculateChecksum,
                               int smallBufferSize, final Configuration conf)
      throws IOException {

    if (calculateChecksum) {
      computeChecksum(srcReplica, dstMeta, smallBufferSize, conf);
    } else {
      try {
        srcReplica.copyMetadata(dstMeta.toURI());
      } catch (IOException e) {
        throw new IOException("Failed to copy " + srcReplica + " metadata to "
            + dstMeta, e);
      }
    }
    try {
      srcReplica.copyBlockdata(dstFile.toURI());
    } catch (IOException e) {
      throw new IOException("Failed to copy " + srcReplica + " block file to "
          + dstFile, e);
    }
    if (LOG.isDebugEnabled()) {
      if (calculateChecksum) {
        LOG.debug("Copied " + srcReplica.getMetadataURI() + " meta to "
            + dstMeta + " and calculated checksum");
      } else {
        LOG.debug("Copied " + srcReplica.getBlockURI() + " to " + dstFile);
      }
    }
    return new File[] {dstMeta, dstFile};
  }

  static File[] hardLinkBlockFiles(ReplicaInfo srcReplica, File dstMeta,
      File dstFile)
      throws IOException {
    FsVolumeSpi srcReplicaVolume = srcReplica.getVolume();
    File destParentFile = dstFile.getParentFile();

View on GitHub (pinned to 2add963021)

Solutions

1. Compare block length vs destination free space before retry: df the dst volume; raise dfs.datanode.du.reserved if the volume was overcommitted.
2. Delete the orphan dstMeta left by the failed attempt (message gives the path) or let the caller's cleanup discard it, then retry.
3. If source reads fail, run hdfs fsck to confirm the block has other healthy replicas and evict/decommission the bad volume.
4. Stagger mover jobs (bandwidth caps: dfs.datanode.balance.bandwidthPerSec, dfs.datanode.mover.max-concurrency) to avoid destination saturation.

// Ensure the block fits before the large copy.
long need = srcReplica.getNumBytes();
long free = dstFile.getParentFile().getUsableSpace();
if (free < need) {
  throw new IOException("dst has " + free + " free, need " + need);
}

// Data copy failed after meta copy: discard partial dst artifacts, retry.
try {
  FsDatasetImpl.copyBlockFiles(src, dstMeta, dstFile,
      calculateChecksum, bufSize, conf);
} catch (IOException e) {
  deleteQuietly(dstFile);
  deleteQuietly(dstMeta);
  schedulePersistenceRetry(src); // lazy-persist/mover will re-attempt
}

When it happens

Trigger: copyBlockFiles where the meta step succeeded but the (much larger) block data copy fails: destination fills mid-copy (ENOSPC), source block file truncated/corrupt (EIO/checksum read failure at the IO-provider level), or destination file creation denied.

Common situations: RAM_DISK lazy-write flush filling the backing DISK; mover copying a multi-GB block onto a volume with less free space than the block; bit-rot on the source disk surfacing as read failure; concurrent scanner quarantining the source replica mid-copy.

Related errors

• Failed to copy {srcReplica} metadata to {dstMeta}
• Failed to move block file for {b} from {blockURI} to {absolu
• Replica {replicaInfo} cannot be moved from storageType : {st
• Found duplicated storage UUID: %s in %s.
• Storage type %s already exists on same mount: %s.

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/eaec333aa4d28b23. Report an issue: GitHub.