{"record":{"id":"5d5514e2fd60657d","repo":"apache/hadoop","slug":"replica-replicainfo-cannot-be-moved-from-storage","errorCode":null,"errorMessage":"Replica {replicaInfo} cannot be moved from storageType : {storageType}","messagePattern":"Replica (.+?) cannot be moved from storageType : (.+?)","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"warning","filePath":"hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java","lineNumber":1103,"sourceCode":"      throws IOException {\n    ReplicaInfo replicaInfo = getReplicaInfo(block);\n    if (replicaInfo.getState() != ReplicaState.FINALIZED) {\n      throw new ReplicaNotFoundException(\n          ReplicaNotFoundException.UNFINALIZED_REPLICA + block);\n    }\n    if (replicaInfo.getNumBytes() != block.getNumBytes()) {\n      throw new IOException(\"Corrupted replica \" + replicaInfo\n          + \" with a length of \" + replicaInfo.getNumBytes()\n          + \" expected length is \" + block.getNumBytes());\n    }\n    if (replicaInfo.getVolume().getStorageType() == targetStorageType) {\n      throw new ReplicaAlreadyExistsException(\"Replica \" + replicaInfo\n          + \" already exists on storage \" + targetStorageType);\n    }\n\n    if (replicaInfo.isOnTransientStorage()) {\n      // Block movement from RAM_DISK will be done by LazyPersist mechanism\n      throw new IOException(\"Replica \" + replicaInfo\n          + \" cannot be moved from storageType : \"\n          + replicaInfo.getVolume().getStorageType());\n    }\n\n    FsVolumeReference volumeRef = null;\n    boolean shouldConsiderSameMountVolume =\n        shouldConsiderSameMountVolume(replicaInfo.getVolume(),\n            targetStorageType, targetStorageId);\n    boolean useVolumeOnSameMount = false;\n\n    try (AutoCloseableLock lock = lockManager.readLock(LockLevel.BLOCK_POOl,\n        block.getBlockPoolId())) {\n      if (shouldConsiderSameMountVolume) {\n        volumeRef = volumes.getVolumeByMount(targetStorageType,\n            ((FsVolumeImpl) replicaInfo.getVolume()).getMount(),\n            block.getNumBytes());\n        if (volumeRef != null) {\n          useVolumeOnSameMount = true;","sourceCodeStart":1085,"sourceCodeEnd":1121,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java#L1085-L1121","documentation":"Thrown as IOException from FsDatasetImpl.moveBlockAcrossStorage when replicaInfo.isOnTransientStorage() — the replica sits on RAM_DISK (lazy-persist tier). Moves out of RAM_DISK are deliberately not done synchronously; the LazyPersist mechanism flushes RAM_DISK replicas to a durable DISK volume asynchronously, so the explicit move API refuses.","triggerScenarios":"Requesting a storage-type move for a block cached on RAM_DISK via dfs.datanode.data.dir [RAM_DISK] with lazy persist enabled (dfs.datanode.lazy.writer) — e.g. mover targeting a policy change on a lazily-written file.","commonSituations":"Lazy-persist writes (LAZY_PERSIST policy) followed by a policy change and mover run; users assuming mover evacuates RAM_DISK like any volume; RAM_DISK running low and admin attempting a manual drain via mover.","solutions":["Let the lazy writer do the eviction/flush: ensure a [DISK] volume exists and dfs.datanode.lazy.writer.threads.interval is sane; RAM_DISK replicas persist automatically.","Check dfs.datanode.max.locked.memory and RAM_DISK capacity — tight RAM forces faster lazy eviction, after which the block is on DISK and can be moved.","If the policy target is DISK, simply wait for lazy persist; if a different tier is required, move after persistence completes (verify via fsck locations).","Do not retry moveBlockAcrossStorage for the RAM_DISK replica — it will keep refusing by design."],"exampleFix":null,"handlingStrategy":"validation","validationCode":"// Do not ask to move RAM_DISK replicas; wait for lazy persist instead.\nReplicaInfo info = fsDataset.getReplica(\n    block.getBlockPoolId(), block.getBlockId());\nif (info != null && info.isOnTransientStorage()) {\n  // LazyPersist will flush it to DISK; skip explicit move\n  continue;\n}","typeGuard":null,"tryCatchPattern":"// Expected refusal for RAM_DISK: log and rely on lazy persist.\ntry {\n  fsDataset.moveBlockAcrossStorage(block, target, null);\n} catch (IOException e) {\n  if (String.valueOf(e.getMessage()).contains(\"cannot be moved from storageType\")) {\n    LOG.debug(\"{} on transient storage; lazy persist will relocate\", block);\n    continue;\n  }\n  throw e;\n}","preventionTips":["Filter transient-storage (RAM_DISK) replicas out of move requests before calling the API.","Keep a DISK volume and lazy-writer enabled whenever LAZY_PERSIST is used so flushes actually happen.","Size dfs.datanode.max.locked.memory below physical RAM to force eviction/persist instead of manual moves."],"tags":["hadoop","hdfs","datanode","fsdataset","lazy-persist","ram-disk","mover"],"backgroundTag":"transient-storage-move-refused","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}