{"record":{"id":"86bf74e82d0de924","repo":"apache/hadoop","slug":"failed-to-copy-srcreplica-metadata-to-dstmeta","errorCode":null,"errorMessage":"Failed to copy {srcReplica} metadata to {dstMeta}","messagePattern":"Failed to copy (.+?) metadata to (.+?)","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java","lineNumber":1024,"sourceCode":"    final File destDir = DatanodeUtil.idToBlockDir(destRoot, blockId);\n    // blockName is same as the filename for the block\n    final File dstFile = new File(destDir, srcReplica.getBlockName());\n    final File dstMeta = FsDatasetUtil.getMetaFile(dstFile, genStamp);\n    return hardLinkBlockFiles(srcReplica, dstMeta, dstFile);\n  }\n\n  static File[] copyBlockFiles(ReplicaInfo srcReplica, File dstMeta,\n                               File dstFile, boolean calculateChecksum,\n                               int smallBufferSize, final Configuration conf)\n      throws IOException {\n\n    if (calculateChecksum) {\n      computeChecksum(srcReplica, dstMeta, smallBufferSize, conf);\n    } else {\n      try {\n        srcReplica.copyMetadata(dstMeta.toURI());\n      } catch (IOException e) {\n        throw new IOException(\"Failed to copy \" + srcReplica + \" metadata to \"\n            + dstMeta, e);\n      }\n    }\n    try {\n      srcReplica.copyBlockdata(dstFile.toURI());\n    } catch (IOException e) {\n      throw new IOException(\"Failed to copy \" + srcReplica + \" block file to \"\n          + dstFile, e);\n    }\n    if (LOG.isDebugEnabled()) {\n      if (calculateChecksum) {\n        LOG.debug(\"Copied \" + srcReplica.getMetadataURI() + \" meta to \"\n            + dstMeta + \" and calculated checksum\");\n      } else {\n        LOG.debug(\"Copied \" + srcReplica.getBlockURI() + \" to \" + dstFile);\n      }\n    }\n    return new File[] {dstMeta, dstFile};","sourceCodeStart":1006,"sourceCodeEnd":1042,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java#L1006-L1042","documentation":"Thrown as IOException from the static FsDatasetImpl.copyBlockFiles(ReplicaInfo, File dstMeta, File dstFile, boolean calculateChecksum, ...) when srcReplica.copyMetadata(dstMeta) fails in the calculateChecksum=false branch (the checksum-recompute branch writes the meta itself and cannot hit this). Used by lazy-write RAM_DISK→DISK flushing and replica-copy paths; the cause is chained.","triggerScenarios":"copyBlockFiles with calculateChecksum=false copying a replica's meta file to a destination that is unwritable/full, or from a source meta that vanished (RAM_DISK eviction race, deleted replica) — copyMetadata propagates the failure and this wrapper names srcReplica and dstMeta.","commonSituations":"Lazy-persist flush racing replica eviction from RAM_DISK; destination DISK volume full when RAM_DISK data must be persisted; permission drift on the finalized subdir; tests copying replicas onto tmpfs that filled up.","solutions":["Read the chained cause for the OS error; map it to source (missing meta) vs destination (EACCES/ENOSPC) using the paths in the message.","Ensure destination volumes have headroom (dfs.datanode.du.reserved) and correct ownership of current/finalized subdirs.","If the source was evicted from RAM_DISK, this copy is unnecessary — let the replica be re-replicated; the DN discards transient replicas that failed to persist.","Re-trigger the write or mover operation after volume repair; copyBlockFiles is retried per-replica by the lazy-persist tracker."],"exampleFix":null,"handlingStrategy":"retry","validationCode":"// Before copyBlockFiles(calculateChecksum=false): source meta present,\n// destination writable.\nif (!new File(srcReplica.getMetadataURI()).exists()) return skip();\nFile dp = dstMeta.getParentFile();\nif (dp != null && !(dp.isDirectory() || dp.mkdirs())) return skip();","typeGuard":null,"tryCatchPattern":"// Lazy-persist semantics: failure is transient per-replica; retry via tracker.\ntry {\n  FsDatasetImpl.copyBlockFiles(src, dstMeta, dstFile, false, bufSize, conf);\n} catch (IOException e) {\n  LOG.warn(\"Meta copy failed for {}: {}\", src, e.getCause());\n  // do not fail the writer; the lazy writer retries persistence later\n}","preventionTips":["Keep a healthy [DISK] tier available whenever [RAM_DISK]/LAZY_PERSIST is configured — persistence depends on it.","Monitor 'RamDiskBlocksLazyPersisted' vs 'RamDiskBlocksWriteFallback' metrics to catch persistence failures early.","Avoid pointing lazy-persist destinations at volumes shared with archive/scan-heavy workloads."],"tags":["hadoop","hdfs","datanode","fsdataset","lazy-persist","ram-disk","copy","metadata-file"],"backgroundTag":"file-copy-failed","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-23T01:17:44.959Z"}