apache/hadoop · warning · IOException

Replica {replicaInfo} cannot be moved from storageType : {st

Error message

Replica {replicaInfo} cannot be moved from storageType : {storageType}

What it means

Thrown as IOException from FsDatasetImpl.moveBlockAcrossStorage when replicaInfo.isOnTransientStorage() — the replica sits on RAM_DISK (lazy-persist tier). Moves out of RAM_DISK are deliberately not done synchronously; the LazyPersist mechanism flushes RAM_DISK replicas to a durable DISK volume asynchronously, so the explicit move API refuses.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java:1103

      throws IOException {
    ReplicaInfo replicaInfo = getReplicaInfo(block);
    if (replicaInfo.getState() != ReplicaState.FINALIZED) {
      throw new ReplicaNotFoundException(
          ReplicaNotFoundException.UNFINALIZED_REPLICA + block);
    }
    if (replicaInfo.getNumBytes() != block.getNumBytes()) {
      throw new IOException("Corrupted replica " + replicaInfo
          + " with a length of " + replicaInfo.getNumBytes()
          + " expected length is " + block.getNumBytes());
    }
    if (replicaInfo.getVolume().getStorageType() == targetStorageType) {
      throw new ReplicaAlreadyExistsException("Replica " + replicaInfo
          + " already exists on storage " + targetStorageType);
    }

    if (replicaInfo.isOnTransientStorage()) {
      // Block movement from RAM_DISK will be done by LazyPersist mechanism
      throw new IOException("Replica " + replicaInfo
          + " cannot be moved from storageType : "
          + replicaInfo.getVolume().getStorageType());
    }

    FsVolumeReference volumeRef = null;
    boolean shouldConsiderSameMountVolume =
        shouldConsiderSameMountVolume(replicaInfo.getVolume(),
            targetStorageType, targetStorageId);
    boolean useVolumeOnSameMount = false;

    try (AutoCloseableLock lock = lockManager.readLock(LockLevel.BLOCK_POOl,
        block.getBlockPoolId())) {
      if (shouldConsiderSameMountVolume) {
        volumeRef = volumes.getVolumeByMount(targetStorageType,
            ((FsVolumeImpl) replicaInfo.getVolume()).getMount(),
            block.getNumBytes());
        if (volumeRef != null) {
          useVolumeOnSameMount = true;

View on GitHub (pinned to 2add963021)

Solutions

1. Let the lazy writer do the eviction/flush: ensure a [DISK] volume exists and dfs.datanode.lazy.writer.threads.interval is sane; RAM_DISK replicas persist automatically.
2. Check dfs.datanode.max.locked.memory and RAM_DISK capacity — tight RAM forces faster lazy eviction, after which the block is on DISK and can be moved.
3. If the policy target is DISK, simply wait for lazy persist; if a different tier is required, move after persistence completes (verify via fsck locations).
4. Do not retry moveBlockAcrossStorage for the RAM_DISK replica — it will keep refusing by design.

// Do not ask to move RAM_DISK replicas; wait for lazy persist instead.
ReplicaInfo info = fsDataset.getReplica(
    block.getBlockPoolId(), block.getBlockId());
if (info != null && info.isOnTransientStorage()) {
  // LazyPersist will flush it to DISK; skip explicit move
  continue;
}

// Expected refusal for RAM_DISK: log and rely on lazy persist.
try {
  fsDataset.moveBlockAcrossStorage(block, target, null);
} catch (IOException e) {
  if (String.valueOf(e.getMessage()).contains("cannot be moved from storageType")) {
    LOG.debug("{} on transient storage; lazy persist will relocate", block);
    continue;
  }
  throw e;
}

When it happens

Trigger: Requesting a storage-type move for a block cached on RAM_DISK via dfs.datanode.data.dir [RAM_DISK] with lazy persist enabled (dfs.datanode.lazy.writer) — e.g. mover targeting a policy change on a lazily-written file.

Common situations: Lazy-persist writes (LAZY_PERSIST policy) followed by a policy change and mover run; users assuming mover evacuates RAM_DISK like any volume; RAM_DISK running low and admin attempting a manual drain via mover.

Related errors

• Failed to copy {srcReplica} metadata to {dstMeta}
• Failed to move meta file for {b} from {metadataURI} to {dstm
• Failed to move block file for {b} from {blockURI} to {absolu
• Failed to copy {srcReplica} block file to {dstFile}
• Cannot append to an unfinalized replica {block}

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/5d5514e2fd60657d. Report an issue: GitHub.