{"record":{"id":"ef57311fe6ec72ed","repo":"apache/hadoop","slug":"block-length-mismatch-len-len-but-r-r","errorCode":null,"errorMessage":"Block length mismatch, len={len} but r={r}","messagePattern":"Block length mismatch, len=(.+?) but r=(.+?)","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java","lineNumber":2333,"sourceCode":"      if (r.blockDataExists()) {\n        return r;\n      }\n      // if file is not null, but doesn't exist - possibly disk failed\n      datanode.checkDiskErrorAsync(r.getVolume());\n    }\n\n    LOG.debug(\"blockId={}, replica={}\", blockId, r);\n    return null;\n  }\n\n  /** Check the files of a replica. */\n  static void checkReplicaFiles(final ReplicaInfo r) throws IOException {\n    //check replica's data exists\n    if (!r.blockDataExists()) {\n      throw new FileNotFoundException(\"Block data not found, r=\" + r);\n    }\n    if (r.getBytesOnDisk() != r.getBlockDataLength()) {\n      throw new IOException(\"Block length mismatch, len=\"\n          + r.getBlockDataLength() + \" but r=\" + r);\n    }\n\n    //check replica's meta file\n    if (!r.metadataExists()) {\n      throw new IOException(r.getMetadataURI() + \" does not exist, r=\" + r);\n    }\n    if (r.getMetadataLength() == 0) {\n      throw new IOException(\"Metafile is empty, r=\" + r);\n    }\n  }\n\n  /**\n   * We're informed that a block is no longer valid. Delete it.\n   */\n  @Override // FsDatasetSpi\n  public void invalidate(String bpid, Block invalidBlks[]) throws IOException {\n    invalidate(bpid, invalidBlks, true);","sourceCodeStart":2315,"sourceCodeEnd":2351,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java#L2315-L2351","documentation":"IOException thrown by FsDatasetImpl.checkReplicaFiles when the replica's recorded on-disk length (getBytesOnDisk()) differs from the actual block data file length (getBlockDataLength()). The in-memory byte count and the filesystem metadata disagree, which would corrupt length accounting if recovery proceeded.","triggerScenarios":"checkReplicaFiles(r) during replica recovery/finalize while the block file was appended or truncated outside HDFS's accounting, or the replica object's cached length is stale relative to the file.","commonSituations":"Concurrent write updating the file while recovery validates it; file modified externally; crash between file append and volumeMap update; filesystem corruption (e.g., wrong file restored by backup).","solutions":["Retry the operation after the concurrent writer/recovery session completes - a benign race self-heals.","If persistent, validate the block file (hdfs fsck -blockcheck; compare lengths via 'ls -l' on the block file path shown in the message).","Restart the DataNode so replica lengths are re-derived from disk during volume scanning.","If the file was externally tampered with, delete the replica and let the NameNode re-replicate."],"exampleFix":null,"handlingStrategy":"try-catch","validationCode":null,"typeGuard":"boolean isBlockLengthMismatch(IOException e) {\n  return e.getMessage() != null && e.getMessage().startsWith(\"Block length mismatch\");\n}","tryCatchPattern":"try {\n  dataset.initReplicaRecovery(rBlock);\n} catch (IOException e) {\n  if (e.getMessage() != null && e.getMessage().startsWith(\"Block length mismatch\")) {\n    // transient if a writer raced the check; retry after pipeline settles, else re-replicate\n    return retryAfterWritersQuiet(rBlock);\n  }\n  throw e;\n}","preventionTips":["Never append to block files outside DataNode processes.","After crashes, let the DataNode volume scanner (dfs.datanode.scan.period.hours) reconcile lengths.","Treat repeat mismatches on one volume as disk replacement criteria."],"tags":["hdfs","datanode","replica-files","length-mismatch","consistency"],"backgroundTag":"block-length-mismatch","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-23T01:17:44.959Z"}