apache/hadoop · error · IOException

Metafile is empty, r={r}

Error message

Metafile is empty, r={r}

What it means

IOException thrown by FsDatasetImpl.checkReplicaFiles when the replica's meta (checksum) file exists but is zero bytes (getMetadataLength() == 0). An empty .meta file lacks the checksum header (version + checksum type), so checksums cannot be parsed and the replica cannot be recovered, validated, or finalized safely.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java:2342

  }

  /** Check the files of a replica. */
  static void checkReplicaFiles(final ReplicaInfo r) throws IOException {
    //check replica's data exists
    if (!r.blockDataExists()) {
      throw new FileNotFoundException("Block data not found, r=" + r);
    }
    if (r.getBytesOnDisk() != r.getBlockDataLength()) {
      throw new IOException("Block length mismatch, len="
          + r.getBlockDataLength() + " but r=" + r);
    }

    //check replica's meta file
    if (!r.metadataExists()) {
      throw new IOException(r.getMetadataURI() + " does not exist, r=" + r);
    }
    if (r.getMetadataLength() == 0) {
      throw new IOException("Metafile is empty, r=" + r);
    }
  }

  /**
   * We're informed that a block is no longer valid. Delete it.
   */
  @Override // FsDatasetSpi
  public void invalidate(String bpid, Block invalidBlks[]) throws IOException {
    invalidate(bpid, invalidBlks, true);
  }

  private void invalidate(String bpid, Block[] invalidBlks, boolean async)
      throws IOException {
    final List<String> errors = new ArrayList<String>();
    for (int i = 0; i < invalidBlks.length; i++) {
      final ReplicaInfo info;
      final FsVolumeImpl v;
      try (AutoCloseableLock lock = lockManager.readLock(LockLevel.BLOCK_POOl, bpid)) {

View on GitHub (pinned to 2add963021)

Solutions

1. Treat the replica as corrupt: use hdfs fsck to confirm other replicas exist, then invalidate this replica for re-replication.
2. If unique, salvage data via recoverLease and read with checksum verification disabled, accepting risk.
3. Investigate crash-consistency: enable/verify journaling filesystems, check UPS/battery on RAID cache.
4. Restart the DataNode after fixing the underlying storage reliability issue.

boolean isEmptyMetaFile(IOException e) {
  return e.getMessage() != null && e.getMessage().startsWith("Metafile is empty");
}

try {
  dataset.initReplicaRecovery(rBlock);
} catch (IOException e) {
  if (e.getMessage() != null && e.getMessage().startsWith("Metafile is empty")) {
    // unreadable checksums: drop this replica from recovery set, verify others via fsck
    return recoverFromOtherReplicas(rBlock);
  }
  throw e;
}

When it happens

Trigger: checkReplicaFiles(r) during recovery/finalize where the .meta file was created but its header was never written - typical of a crash between FileOutputStream create and the first flush of the meta header.

Common situations: Hard crash / power loss at the moment a block transfer started writing its meta file; disk that reports successful writes but loses them; interrupted backup restore truncating meta files.

Related errors

• {metadataURI} does not exist, r={r}
• Unknown op {} in data stream
• file VERSION is invalid.
• A disk IO error occurred
• file VERSION is invalid.

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/c528dc12731792de. Report an issue: GitHub.