apache/hadoop · error · IOException

Failed to copy {srcReplica} metadata to {dstMeta}

Error message

Failed to copy {srcReplica} metadata to {dstMeta}

What it means

Thrown as IOException from the static FsDatasetImpl.copyBlockFiles(ReplicaInfo, File dstMeta, File dstFile, boolean calculateChecksum, ...) when srcReplica.copyMetadata(dstMeta) fails in the calculateChecksum=false branch (the checksum-recompute branch writes the meta itself and cannot hit this). Used by lazy-write RAM_DISK→DISK flushing and replica-copy paths; the cause is chained.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java:1024

    final File destDir = DatanodeUtil.idToBlockDir(destRoot, blockId);
    // blockName is same as the filename for the block
    final File dstFile = new File(destDir, srcReplica.getBlockName());
    final File dstMeta = FsDatasetUtil.getMetaFile(dstFile, genStamp);
    return hardLinkBlockFiles(srcReplica, dstMeta, dstFile);
  }

  static File[] copyBlockFiles(ReplicaInfo srcReplica, File dstMeta,
                               File dstFile, boolean calculateChecksum,
                               int smallBufferSize, final Configuration conf)
      throws IOException {

    if (calculateChecksum) {
      computeChecksum(srcReplica, dstMeta, smallBufferSize, conf);
    } else {
      try {
        srcReplica.copyMetadata(dstMeta.toURI());
      } catch (IOException e) {
        throw new IOException("Failed to copy " + srcReplica + " metadata to "
            + dstMeta, e);
      }
    }
    try {
      srcReplica.copyBlockdata(dstFile.toURI());
    } catch (IOException e) {
      throw new IOException("Failed to copy " + srcReplica + " block file to "
          + dstFile, e);
    }
    if (LOG.isDebugEnabled()) {
      if (calculateChecksum) {
        LOG.debug("Copied " + srcReplica.getMetadataURI() + " meta to "
            + dstMeta + " and calculated checksum");
      } else {
        LOG.debug("Copied " + srcReplica.getBlockURI() + " to " + dstFile);
      }
    }
    return new File[] {dstMeta, dstFile};

View on GitHub (pinned to 2add963021)

Solutions

1. Read the chained cause for the OS error; map it to source (missing meta) vs destination (EACCES/ENOSPC) using the paths in the message.
2. Ensure destination volumes have headroom (dfs.datanode.du.reserved) and correct ownership of current/finalized subdirs.
3. If the source was evicted from RAM_DISK, this copy is unnecessary — let the replica be re-replicated; the DN discards transient replicas that failed to persist.
4. Re-trigger the write or mover operation after volume repair; copyBlockFiles is retried per-replica by the lazy-persist tracker.

// Before copyBlockFiles(calculateChecksum=false): source meta present,
// destination writable.
if (!new File(srcReplica.getMetadataURI()).exists()) return skip();
File dp = dstMeta.getParentFile();
if (dp != null && !(dp.isDirectory() || dp.mkdirs())) return skip();

// Lazy-persist semantics: failure is transient per-replica; retry via tracker.
try {
  FsDatasetImpl.copyBlockFiles(src, dstMeta, dstFile, false, bufSize, conf);
} catch (IOException e) {
  LOG.warn("Meta copy failed for {}: {}", src, e.getCause());
  // do not fail the writer; the lazy writer retries persistence later
}

When it happens

Trigger: copyBlockFiles with calculateChecksum=false copying a replica's meta file to a destination that is unwritable/full, or from a source meta that vanished (RAM_DISK eviction race, deleted replica) — copyMetadata propagates the failure and this wrapper names srcReplica and dstMeta.

Common situations: Lazy-persist flush racing replica eviction from RAM_DISK; destination DISK volume full when RAM_DISK data must be persisted; permission drift on the finalized subdir; tests copying replicas onto tmpfs that filled up.

Related errors

• Failed to copy {srcReplica} block file to {dstFile}
• Replica {replicaInfo} cannot be moved from storageType : {st
• Failed to move meta file for {b} from {metadataURI} to {dstm
• Failed to hardLink {srcReplica} metadata to {dstMeta}
• Found duplicated storage UUID: %s in %s.

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/86bf74e82d0de924. Report an issue: GitHub.