{"record":{"id":"eaec333aa4d28b23","repo":"apache/hadoop","slug":"failed-to-copy-srcreplica-block-file-to-dstfile","errorCode":null,"errorMessage":"Failed to copy {srcReplica} block file to {dstFile}","messagePattern":"Failed to copy (.+?) block file to (.+?)","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java","lineNumber":1031,"sourceCode":"  static File[] copyBlockFiles(ReplicaInfo srcReplica, File dstMeta,\n                               File dstFile, boolean calculateChecksum,\n                               int smallBufferSize, final Configuration conf)\n      throws IOException {\n\n    if (calculateChecksum) {\n      computeChecksum(srcReplica, dstMeta, smallBufferSize, conf);\n    } else {\n      try {\n        srcReplica.copyMetadata(dstMeta.toURI());\n      } catch (IOException e) {\n        throw new IOException(\"Failed to copy \" + srcReplica + \" metadata to \"\n            + dstMeta, e);\n      }\n    }\n    try {\n      srcReplica.copyBlockdata(dstFile.toURI());\n    } catch (IOException e) {\n      throw new IOException(\"Failed to copy \" + srcReplica + \" block file to \"\n          + dstFile, e);\n    }\n    if (LOG.isDebugEnabled()) {\n      if (calculateChecksum) {\n        LOG.debug(\"Copied \" + srcReplica.getMetadataURI() + \" meta to \"\n            + dstMeta + \" and calculated checksum\");\n      } else {\n        LOG.debug(\"Copied \" + srcReplica.getBlockURI() + \" to \" + dstFile);\n      }\n    }\n    return new File[] {dstMeta, dstFile};\n  }\n\n  static File[] hardLinkBlockFiles(ReplicaInfo srcReplica, File dstMeta,\n      File dstFile)\n      throws IOException {\n    FsVolumeSpi srcReplicaVolume = srcReplica.getVolume();\n    File destParentFile = dstFile.getParentFile();","sourceCodeStart":1013,"sourceCodeEnd":1049,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java#L1013-L1049","documentation":"Thrown as IOException from FsDatasetImpl.copyBlockFiles when srcReplica.copyBlockdata(dstFile) fails after the metadata copy step already ran. The cause is chained and the message names srcReplica and dstFile. Callers get an all-or-nothing signal: a data copy failure leaves a possibly-complete meta file at dstMeta that the caller must discard, which the lazy-persist/move callers handle by aborting the whole replica operation.","triggerScenarios":"copyBlockFiles where the meta step succeeded but the (much larger) block data copy fails: destination fills mid-copy (ENOSPC), source block file truncated/corrupt (EIO/checksum read failure at the IO-provider level), or destination file creation denied.","commonSituations":"RAM_DISK lazy-write flush filling the backing DISK; mover copying a multi-GB block onto a volume with less free space than the block; bit-rot on the source disk surfacing as read failure; concurrent scanner quarantining the source replica mid-copy.","solutions":["Compare block length vs destination free space before retry: df the dst volume; raise dfs.datanode.du.reserved if the volume was overcommitted.","Delete the orphan dstMeta left by the failed attempt (message gives the path) or let the caller's cleanup discard it, then retry.","If source reads fail, run hdfs fsck to confirm the block has other healthy replicas and evict/decommission the bad volume.","Stagger mover jobs (bandwidth caps: dfs.datanode.balance.bandwidthPerSec, dfs.datanode.mover.max-concurrency) to avoid destination saturation."],"exampleFix":null,"handlingStrategy":"retry","validationCode":"// Ensure the block fits before the large copy.\nlong need = srcReplica.getNumBytes();\nlong free = dstFile.getParentFile().getUsableSpace();\nif (free < need) {\n  throw new IOException(\"dst has \" + free + \" free, need \" + need);\n}","typeGuard":null,"tryCatchPattern":"// Data copy failed after meta copy: discard partial dst artifacts, retry.\ntry {\n  FsDatasetImpl.copyBlockFiles(src, dstMeta, dstFile,\n      calculateChecksum, bufSize, conf);\n} catch (IOException e) {\n  deleteQuietly(dstFile);\n  deleteQuietly(dstMeta);\n  schedulePersistenceRetry(src); // lazy-persist/mover will re-attempt\n}","preventionTips":["Set dfs.datanode.du.reserved on destination volumes above your max block size.","Clean orphan dst files/metas on failure paths — partial copies poison retries.","Watch for source-read errors in the cause: route those to re-replication instead of copy retries."],"tags":["hadoop","hdfs","datanode","fsdataset","lazy-persist","copy","block-file","disk-full"],"backgroundTag":"file-copy-failed","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}