{"record":{"id":"ece53c451f85684a","repo":"apache/hadoop","slug":"failed-to-move-meta-file-for-b-from-metadatauri","errorCode":null,"errorMessage":"Failed to move meta file for {b} from {metadataURI} to {dstmeta}","messagePattern":"Failed to move meta file for (.+?) from (.+?) to (.+?)","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java","lineNumber":965,"sourceCode":"        } catch (IOException e) {\n          IOUtils.cleanupWithLogger(null, blockInStream);\n          throw e;\n        }\n      } catch (IOException e) {\n        IOUtils.cleanupWithLogger(null, ref);\n        throw e;\n      }\n    }\n  }\n\n  static File moveBlockFiles(Block b, ReplicaInfo replicaInfo, File destdir)\n      throws IOException {\n    final File dstfile = new File(destdir, b.getBlockName());\n    final File dstmeta = FsDatasetUtil.getMetaFile(dstfile, b.getGenerationStamp());\n    try {\n      replicaInfo.renameMeta(dstmeta.toURI());\n    } catch (IOException e) {\n      throw new IOException(\"Failed to move meta file for \" + b\n          + \" from \" + replicaInfo.getMetadataURI() + \" to \" + dstmeta, e);\n    }\n    try {\n      replicaInfo.renameData(dstfile.toURI());\n    } catch (IOException e) {\n      throw new IOException(\"Failed to move block file for \" + b\n          + \" from \" + replicaInfo.getBlockURI() + \" to \"\n          + dstfile.getAbsolutePath(), e);\n    }\n    if (LOG.isDebugEnabled()) {\n      LOG.debug(\"addFinalizedBlock: Moved \" + replicaInfo.getMetadataURI()\n          + \" to \" + dstmeta + \" and \" + replicaInfo.getBlockURI()\n          + \" to \" + dstfile);\n    }\n    return dstfile;\n  }\n\n  /**","sourceCodeStart":947,"sourceCodeEnd":983,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java#L947-L983","documentation":"Thrown as IOException from the static FsDatasetImpl.moveBlockFiles(Block, ReplicaInfo, File) when replicaInfo.renameMeta(dstmeta) fails while relocating a finalized replica's metadata file into the destination directory (used by moveBlockAcrossStorage and replica move paths). The original IOException is chained as the cause, and the message carries the block, source metadataURI, and intended destination so the failing rename is fully identified.","triggerScenarios":"moveBlockFiles invoked during moveBlockAcrossStorage (admin mover / setStoragePolicy-triggered moves): renameMeta fails because the destination dir is unwritable, the meta file is missing on source, destination meta already exists, the filesystem is full, or the rename crosses filesystem boundaries where the provider must copy and that copy fails.","commonSituations":"Destination volume mounted read-only or out of inodes/space; source meta lost to disk corruption; mover running while block scanner quarantines the replica; permissions changed under the DataNode user; NFS-backed data dir with flaky rename semantics.","solutions":["Inspect the cause chain in the log — the nested IOException states the OS-level reason (ENOENT/EACCES/ENOSPC/EXDEV).","Fix the destination volume: df -h and df -i the target mount, chmod/chown the target subdir to the DataNode user, remount read-write.","Confirm the source meta file named by metadataURI still exists and the volume holding it is healthy (check DN volume-failure metrics).","Re-run the mover (hdfs mover / Mover tool) after repair; already-finalized replicas that failed mid-move are retried idempotently."],"exampleFix":null,"handlingStrategy":"try-catch","validationCode":"// Before moving, confirm both source meta exists and destination is writable.\nif (!new File(srcReplica.getMetadataURI()).exists()) {\n  throw new IllegalStateException(\"source meta missing - skip move\");\n}\nFile dstParent = destdir;\ndstParent.mkdirs();\nif (!dstParent.canWrite()) {\n  throw new IllegalStateException(\"dest not writable: \" + dstParent);\n}","typeGuard":null,"tryCatchPattern":"// Preserve the cause; report paths for ops; move is retryable per-replica.\ntry {\n  FsDatasetImpl.moveBlockFiles(b, replicaInfo, destdir);\n} catch (IOException e) {\n  LOG.warn(\"Meta move failed for {} (cause {}), will retry on next pass\",\n      b, e.getCause());\n  scheduleRetry(b); // mover semantics: idempotent retry after volume fix\n}","preventionTips":["Pre-flight volume checks (writable, space, inodes) on the destination before batch moves.","Alert on ENOSPC/EACCES in DataNode logs during mover windows.","Reserve headroom with dfs.datanode.du.reserved so mover copies cannot fill a volume."],"tags":["hadoop","hdfs","datanode","fsdataset","mover","rename","metadata-file","io"],"backgroundTag":"file-rename-failed","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-23T01:17:44.959Z"}