apache/hadoop · error · IOException

getBytesOnDisk() < getVisibleLength(), rip={replica}

Error message

getBytesOnDisk() < getVisibleLength(), rip={replica}

What it means

IOException thrown by FsDatasetImpl.initReplicaRecoveryImpl when a TEMPORARY/RBW replica being enlisted for block recovery has fewer bytes on disk than its visible length (getBytesOnDisk() < getVisibleLength()). Recovery would finalize a length clients can already see but that is not physically on disk, so the DataNode refuses.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java:3090

      Block block, long recoveryId)
          throws IOException, MustStopExistingWriter {
    final ReplicaInfo replica = map.get(bpid, block.getBlockId());
    //check replica
    if (replica == null) {
      return null;
    }

    //stop writer if there is any
    if (replica.getState() == ReplicaState.TEMPORARY ||
        replica.getState() == ReplicaState.RBW) {
      final ReplicaInPipeline rip = (ReplicaInPipeline)replica;
      if (!rip.attemptToSetWriter(null, Thread.currentThread())) {
        throw new MustStopExistingWriter(rip);
      }

      //check replica bytes on disk.
      if (replica.getBytesOnDisk() < replica.getVisibleLength()) {
        throw new IOException("getBytesOnDisk() < getVisibleLength(), rip="
            + replica);
      }

      //check the replica's files
      checkReplicaFiles(replica);
    }

    //check generation stamp
    if (replica.getGenerationStamp() < block.getGenerationStamp()) {
      throw new IOException(
          "replica.getGenerationStamp() < block.getGenerationStamp(), block="
          + block + ", replica=" + replica);
    }

    //check recovery id
    if (replica.getGenerationStamp() >= recoveryId) {
      throw new IOException("THIS IS NOT SUPPOSED TO HAPPEN:"
          + " replica.getGenerationStamp() >= recoveryId = " + recoveryId

View on GitHub (pinned to 2add963021)

Solutions

1. Recovery will proceed on other replicas: verify with hdfs fsck that at least one pipeline member is healthy.
2. Investigate DataNode logs around the write window for disk errors or 'Slow flush' warnings.
3. Test the disk (smartctl) and replace it if truncation is hardware-caused.
4. If all replicas show this, salvage with hdfs debug recoverLease and accept the visible-length data may be unrecoverable.

boolean isTruncatedReplica(IOException e) {
  return e.getMessage() != null && e.getMessage().startsWith("getBytesOnDisk() < getVisibleLength()");
}

try {
  ReplicaRecoveryInfo info = dataset.initReplicaRecovery(rBlock);
} catch (IOException e) {
  if (e.getMessage() != null && e.getMessage().startsWith("getBytesOnDisk() < getVisibleLength()")) {
    // this replica cannot honor acked length: recover from other pipeline members
    return recoverFromOtherReplicas(rBlock);
  }
  throw e;
}

When it happens

Trigger: initReplicaRecovery on a replica in pipeline state where the visible length (acked to the NameNode through hflush) exceeds what was actually persisted to the block file - e.g., acked-in-memory bytes lost before fsync on a crash, or a truncated file after disk error.

Common situations: DataNode crash between packet ack and disk flush; replica visible length advanced by hflush while disk write lagged; failing disk silently truncating writes; replica resurrected from an inconsistent snapshot.

Related errors

• Replica was found but missing fields.
• {} has no enough internal blocks(current: {}), unable to sta
• Replica gen stamp < block genstamp, block={block}, replica={
• ProvidedReplica does not yet support writes
• Missing a valid replica to recover from

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/05973fc8d2bd9db2. Report an issue: GitHub.