{"record":{"id":"6abdaeb678eb0367","repo":"apache/hadoop","slug":"failed-to-move-block-file-for-b-from-blockuri","errorCode":null,"errorMessage":"Failed to move block file for {b} from {blockURI} to {absolutePath}","messagePattern":"Failed to move block file for (.+?) from (.+?) to (.+?)","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java","lineNumber":971,"sourceCode":"        throw e;\n      }\n    }\n  }\n\n  static File moveBlockFiles(Block b, ReplicaInfo replicaInfo, File destdir)\n      throws IOException {\n    final File dstfile = new File(destdir, b.getBlockName());\n    final File dstmeta = FsDatasetUtil.getMetaFile(dstfile, b.getGenerationStamp());\n    try {\n      replicaInfo.renameMeta(dstmeta.toURI());\n    } catch (IOException e) {\n      throw new IOException(\"Failed to move meta file for \" + b\n          + \" from \" + replicaInfo.getMetadataURI() + \" to \" + dstmeta, e);\n    }\n    try {\n      replicaInfo.renameData(dstfile.toURI());\n    } catch (IOException e) {\n      throw new IOException(\"Failed to move block file for \" + b\n          + \" from \" + replicaInfo.getBlockURI() + \" to \"\n          + dstfile.getAbsolutePath(), e);\n    }\n    if (LOG.isDebugEnabled()) {\n      LOG.debug(\"addFinalizedBlock: Moved \" + replicaInfo.getMetadataURI()\n          + \" to \" + dstmeta + \" and \" + replicaInfo.getBlockURI()\n          + \" to \" + dstfile);\n    }\n    return dstfile;\n  }\n\n  /**\n   * Copy the block and meta files for the given block to the given destination.\n   * @return the new meta and block files.\n   * @throws IOException\n   */\n  static File[] copyBlockFiles(long blockId, long genStamp,\n      ReplicaInfo srcReplica, File destRoot, boolean calculateChecksum,","sourceCodeStart":953,"sourceCodeEnd":989,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java#L953-L989","documentation":"Thrown as IOException from FsDatasetImpl.moveBlockFiles when replicaInfo.renameData(dstfile) fails moving the block data file after the metadata rename already succeeded. The cause is chained and the message shows block, source blockURI, and destination absolute path. Because meta moved but data did not, the replica is left half-moved — callers rely on the exception to abort and later reconcile (the old replica entry is not swapped in until the whole move succeeds).","triggerScenarios":"Same moveBlockFiles flow as 2408 but the second step fails: destination block file already exists, dest dir full/read-only, source block file unreadable (corrupt disk), or EXDEV cross-device rename unsupported by the FileIOProvider chain.","commonSituations":"Mover moving blocks between DISK and ARCHIVE/SSD volumes when the archive mount fills; leftover destination files from a previously interrupted move; dying source disk returning EIO on rename; containerized DN with mismounted volumes.","solutions":["Check the destination path from the message: remove stale half-moved leftovers (block file and its .meta) so the retry starts clean.","Free space/inodes on the destination volume or extend it; verify write permissions for the DataNode user.","If the source volume is erroring (EIO in cause), decommission or replace that volume and let re-replication heal the block instead of moving it.","Retry the storage-policy move once the volume is healthy (hdfs mover -p <path>)."],"exampleFix":null,"handlingStrategy":"try-catch","validationCode":"// Validate destination capacity vs block length before the two-step move.\nlong usable = dstVolume.getAvailable();\nif (usable < replicaInfo.getNumBytes() + overhead) {\n  throw new IOException(\"dest volume lacks space for \"\n      + replicaInfo.getNumBytes() + \" bytes\");\n}","typeGuard":null,"tryCatchPattern":"// Data-move failed after meta moved: clean the half-moved dst pair, retry later.\ntry {\n  FsDatasetImpl.moveBlockFiles(b, replicaInfo, destdir);\n} catch (IOException e) {\n  cleanupPartialMove(destdir, b); // remove dst block + meta leftovers\n  LOG.warn(\"Block move failed for {} ({}), queued for retry\", b, e.getCause());\n  scheduleRetry(b);\n}","preventionTips":["Always clean partial destination files when a move fails midway — leftovers cause the next attempt's 'exists' failure.","Size destination volumes with headroom for the largest blocks (max block size) times mover concurrency.","Cap mover concurrency (dfs.datanode.mover.max-concurrency) to bound destination pressure."],"tags":["hadoop","hdfs","datanode","fsdataset","mover","rename","block-file","io"],"backgroundTag":"file-rename-failed","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}