apache/hadoop · error · IOException

Failed to move block file for {b} from {blockURI} to {absolu

Error message

Failed to move block file for {b} from {blockURI} to {absolutePath}

What it means

Thrown as IOException from FsDatasetImpl.moveBlockFiles when replicaInfo.renameData(dstfile) fails moving the block data file after the metadata rename already succeeded. The cause is chained and the message shows block, source blockURI, and destination absolute path. Because meta moved but data did not, the replica is left half-moved — callers rely on the exception to abort and later reconcile (the old replica entry is not swapped in until the whole move succeeds).

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java:971

        throw e;
      }
    }
  }

  static File moveBlockFiles(Block b, ReplicaInfo replicaInfo, File destdir)
      throws IOException {
    final File dstfile = new File(destdir, b.getBlockName());
    final File dstmeta = FsDatasetUtil.getMetaFile(dstfile, b.getGenerationStamp());
    try {
      replicaInfo.renameMeta(dstmeta.toURI());
    } catch (IOException e) {
      throw new IOException("Failed to move meta file for " + b
          + " from " + replicaInfo.getMetadataURI() + " to " + dstmeta, e);
    }
    try {
      replicaInfo.renameData(dstfile.toURI());
    } catch (IOException e) {
      throw new IOException("Failed to move block file for " + b
          + " from " + replicaInfo.getBlockURI() + " to "
          + dstfile.getAbsolutePath(), e);
    }
    if (LOG.isDebugEnabled()) {
      LOG.debug("addFinalizedBlock: Moved " + replicaInfo.getMetadataURI()
          + " to " + dstmeta + " and " + replicaInfo.getBlockURI()
          + " to " + dstfile);
    }
    return dstfile;
  }

  /**
   * Copy the block and meta files for the given block to the given destination.
   * @return the new meta and block files.
   * @throws IOException
   */
  static File[] copyBlockFiles(long blockId, long genStamp,
      ReplicaInfo srcReplica, File destRoot, boolean calculateChecksum,

View on GitHub (pinned to 2add963021)

Solutions

Check the destination path from the message: remove stale half-moved leftovers (block file and its .meta) so the retry starts clean.
Free space/inodes on the destination volume or extend it; verify write permissions for the DataNode user.
If the source volume is erroring (EIO in cause), decommission or replace that volume and let re-replication heal the block instead of moving it.
Retry the storage-policy move once the volume is healthy (hdfs mover -p <path>).

Defensive patterns

Strategy: try-catch

Validate before calling

// Validate destination capacity vs block length before the two-step move.
long usable = dstVolume.getAvailable();
if (usable < replicaInfo.getNumBytes() + overhead) {
  throw new IOException("dest volume lacks space for "
      + replicaInfo.getNumBytes() + " bytes");
}

Try / catch

// Data-move failed after meta moved: clean the half-moved dst pair, retry later.
try {
  FsDatasetImpl.moveBlockFiles(b, replicaInfo, destdir);
} catch (IOException e) {
  cleanupPartialMove(destdir, b); // remove dst block + meta leftovers
  LOG.warn("Block move failed for {} ({}), queued for retry", b, e.getCause());
  scheduleRetry(b);
}

Prevention

Always clean partial destination files when a move fails midway — leftovers cause the next attempt's 'exists' failure.
Size destination volumes with headroom for the largest blocks (max block size) times mover concurrency.
Cap mover concurrency (dfs.datanode.mover.max-concurrency) to bound destination pressure.

When it happens

Trigger: Same moveBlockFiles flow as 2408 but the second step fails: destination block file already exists, dest dir full/read-only, source block file unreadable (corrupt disk), or EXDEV cross-device rename unsupported by the FileIOProvider chain.

Common situations: Mover moving blocks between DISK and ARCHIVE/SSD volumes when the archive mount fills; leftover destination files from a previously interrupted move; dying source disk returning EIO on rename; containerized DN with mismounted volumes.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/6abdaeb678eb0367. Report an issue: GitHub.