What causes "Block length mismatch, len={len} but r={r}" in apache/hadoop?

Retry the operation after the concurrent writer/recovery session completes - a benign race self-heals.

apache/hadoop · error · IOException

Block length mismatch, len={len} but r={r}

Error message

Block length mismatch, len={len} but r={r}

What it means

IOException thrown by FsDatasetImpl.checkReplicaFiles when the replica's recorded on-disk length (getBytesOnDisk()) differs from the actual block data file length (getBlockDataLength()). The in-memory byte count and the filesystem metadata disagree, which would corrupt length accounting if recovery proceeded.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java:2333

      if (r.blockDataExists()) {
        return r;
      }
      // if file is not null, but doesn't exist - possibly disk failed
      datanode.checkDiskErrorAsync(r.getVolume());
    }

    LOG.debug("blockId={}, replica={}", blockId, r);
    return null;
  }

  /** Check the files of a replica. */
  static void checkReplicaFiles(final ReplicaInfo r) throws IOException {
    //check replica's data exists
    if (!r.blockDataExists()) {
      throw new FileNotFoundException("Block data not found, r=" + r);
    }
    if (r.getBytesOnDisk() != r.getBlockDataLength()) {
      throw new IOException("Block length mismatch, len="
          + r.getBlockDataLength() + " but r=" + r);
    }

    //check replica's meta file
    if (!r.metadataExists()) {
      throw new IOException(r.getMetadataURI() + " does not exist, r=" + r);
    }
    if (r.getMetadataLength() == 0) {
      throw new IOException("Metafile is empty, r=" + r);
    }
  }

  /**
   * We're informed that a block is no longer valid. Delete it.
   */
  @Override // FsDatasetSpi
  public void invalidate(String bpid, Block invalidBlks[]) throws IOException {
    invalidate(bpid, invalidBlks, true);

View on GitHub (pinned to 2add963021)

Solutions

Retry the operation after the concurrent writer/recovery session completes - a benign race self-heals.
If persistent, validate the block file (hdfs fsck -blockcheck; compare lengths via 'ls -l' on the block file path shown in the message).
Restart the DataNode so replica lengths are re-derived from disk during volume scanning.
If the file was externally tampered with, delete the replica and let the NameNode re-replicate.

Defensive patterns

Strategy: try-catch

Type guard

boolean isBlockLengthMismatch(IOException e) {
  return e.getMessage() != null && e.getMessage().startsWith("Block length mismatch");
}

Try / catch

try {
  dataset.initReplicaRecovery(rBlock);
} catch (IOException e) {
  if (e.getMessage() != null && e.getMessage().startsWith("Block length mismatch")) {
    // transient if a writer raced the check; retry after pipeline settles, else re-replicate
    return retryAfterWritersQuiet(rBlock);
  }
  throw e;
}

Prevention

Never append to block files outside DataNode processes.
After crashes, let the DataNode volume scanner (dfs.datanode.scan.period.hours) reconcile lengths.
Treat repeat mismatches on one volume as disk replacement criteria.

When it happens

Trigger: checkReplicaFiles(r) during replica recovery/finalize while the block file was appended or truncated outside HDFS's accounting, or the replica object's cached length is stale relative to the file.

Common situations: Concurrent write updating the file while recovery validates it; file modified externally; crash between file append and volumeMap update; filesystem corruption (e.g., wrong file restored by backup).

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/ef57311fe6ec72ed. Report an issue: GitHub.