apache/hadoop · error · IOException

Failed to move meta file for {b} from {metadataURI} to {dstm

Error message

Failed to move meta file for {b} from {metadataURI} to {dstmeta}

What it means

Thrown as IOException from the static FsDatasetImpl.moveBlockFiles(Block, ReplicaInfo, File) when replicaInfo.renameMeta(dstmeta) fails while relocating a finalized replica's metadata file into the destination directory (used by moveBlockAcrossStorage and replica move paths). The original IOException is chained as the cause, and the message carries the block, source metadataURI, and intended destination so the failing rename is fully identified.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java:965

        } catch (IOException e) {
          IOUtils.cleanupWithLogger(null, blockInStream);
          throw e;
        }
      } catch (IOException e) {
        IOUtils.cleanupWithLogger(null, ref);
        throw e;
      }
    }
  }

  static File moveBlockFiles(Block b, ReplicaInfo replicaInfo, File destdir)
      throws IOException {
    final File dstfile = new File(destdir, b.getBlockName());
    final File dstmeta = FsDatasetUtil.getMetaFile(dstfile, b.getGenerationStamp());
    try {
      replicaInfo.renameMeta(dstmeta.toURI());
    } catch (IOException e) {
      throw new IOException("Failed to move meta file for " + b
          + " from " + replicaInfo.getMetadataURI() + " to " + dstmeta, e);
    }
    try {
      replicaInfo.renameData(dstfile.toURI());
    } catch (IOException e) {
      throw new IOException("Failed to move block file for " + b
          + " from " + replicaInfo.getBlockURI() + " to "
          + dstfile.getAbsolutePath(), e);
    }
    if (LOG.isDebugEnabled()) {
      LOG.debug("addFinalizedBlock: Moved " + replicaInfo.getMetadataURI()
          + " to " + dstmeta + " and " + replicaInfo.getBlockURI()
          + " to " + dstfile);
    }
    return dstfile;
  }

  /**

View on GitHub (pinned to 2add963021)

Solutions

1. Inspect the cause chain in the log — the nested IOException states the OS-level reason (ENOENT/EACCES/ENOSPC/EXDEV).
2. Fix the destination volume: df -h and df -i the target mount, chmod/chown the target subdir to the DataNode user, remount read-write.
3. Confirm the source meta file named by metadataURI still exists and the volume holding it is healthy (check DN volume-failure metrics).
4. Re-run the mover (hdfs mover / Mover tool) after repair; already-finalized replicas that failed mid-move are retried idempotently.

// Before moving, confirm both source meta exists and destination is writable.
if (!new File(srcReplica.getMetadataURI()).exists()) {
  throw new IllegalStateException("source meta missing - skip move");
}
File dstParent = destdir;
dstParent.mkdirs();
if (!dstParent.canWrite()) {
  throw new IllegalStateException("dest not writable: " + dstParent);
}

// Preserve the cause; report paths for ops; move is retryable per-replica.
try {
  FsDatasetImpl.moveBlockFiles(b, replicaInfo, destdir);
} catch (IOException e) {
  LOG.warn("Meta move failed for {} (cause {}), will retry on next pass",
      b, e.getCause());
  scheduleRetry(b); // mover semantics: idempotent retry after volume fix
}

When it happens

Trigger: moveBlockFiles invoked during moveBlockAcrossStorage (admin mover / setStoragePolicy-triggered moves): renameMeta fails because the destination dir is unwritable, the meta file is missing on source, destination meta already exists, the filesystem is full, or the rename crosses filesystem boundaries where the provider must copy and that copy fails.

Common situations: Destination volume mounted read-only or out of inodes/space; source meta lost to disk corruption; mover running while block scanner quarantines the replica; permissions changed under the DataNode user; NFS-backed data dir with flaky rename semantics.

Related errors

• Failed to move block file for {b} from {blockURI} to {absolu
• Failed to hardLink {srcReplica} metadata to {dstMeta}
• Failed to copy {srcReplica} metadata to {dstMeta}
• Failed to hardLink {srcReplica} block file to {dstFile}
• Cannot append to an unfinalized replica {block}

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/ece53c451f85684a. Report an issue: GitHub.