{"record":{"id":"ccb94cd94dd828e7","repo":"apache/hadoop","slug":"replica-not-found-for-block-the-block-may-have","errorCode":null,"errorMessage":"Replica not found for {block}. The block may have been removed recently by the balancer or by intentionally reducing the replication factor. This condition is usually harmless. To be certain, please check the preceding datanode log messages for signs of a more serious issue.","messagePattern":"Replica not found for (.+?)\\. The block may have been removed recently by the balancer or by intentionally reducing the replication factor\\. This condition is usually harmless\\. To be certain, please check the preceding datanode log messages for signs of a more serious issue\\.","errorType":"exception","errorClass":"ReplicaNotFoundException","httpStatus":null,"severity":"warning","filePath":"hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockSender.java","lineNumber":302,"sourceCode":"      }\n      if (replicaVisibleLength < 0) {\n        throw new IOException(\"Replica is not readable, block=\"\n            + block + \", replica=\" + replica);\n      }\n      if (DataNode.LOG.isDebugEnabled()) {\n        DataNode.LOG.debug(\"block=\" + block + \", replica=\" + replica);\n      }\n\n      // transferToFully() fails on 32 bit platforms for block sizes >= 2GB,\n      // use normal transfer in those cases\n      this.transferToAllowed = datanode.getDnConf().transferToAllowed &&\n        (!is32Bit || length <= Integer.MAX_VALUE);\n\n      // Obtain a reference before reading data\n      FsVolumeSpi volume = datanode.data.getVolume(block);\n      if (volume == null) {\n        LOG.warn(\"Cannot find FsVolumeSpi to obtain a reference for block: {}\", block);\n        throw new ReplicaNotFoundException(block);\n      }\n      volumeRef = volume.obtainReference();\n\n      /* \n       * (corruptChecksumOK, meta_file_exist): operation\n       * True,   True: will verify checksum  \n       * True,  False: No verify, e.g., need to read data from a corrupted file \n       * False,  True: will verify checksum\n       * False, False: throws IOException file not found\n       */\n      DataChecksum csum = null;\n      if (verifyChecksum || sendChecksum) {\n        LengthInputStream metaIn = null;\n        boolean keepMetaInOpen = false;\n        try {\n          DataNodeFaultInjector.get().throwTooManyOpenFiles();\n          metaIn = datanode.data.getMetaDataInputStream(block);\n          if (!corruptChecksumOk || metaIn != null) {","sourceCodeStart":284,"sourceCodeEnd":320,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockSender.java#L284-L320","documentation":"Before reading, BlockSender must pin the replica's volume via datanode.data.getVolume(block); a null FsVolumeSpi means no volume in the dataset currently owns this block, so ReplicaNotFoundException is thrown. The message itself states the common truth: the block was removed (balancer move, replication-factor reduction, deletion) after the NameNode handed out its location, and the condition is usually harmless — the client just needs fresher locations.","triggerScenarios":"A client or peer DN opens a read for a block whose files were already deleted on this datanode: dfs.datanode.replication factor lowered, the balancer moved the replica off, or an HDFS delete/unlease removed it between the NN block-report and the read request.","commonSituations":"Reads racing with decreasing replication factor; balancer aggressively moving blocks; clients using stale located-block caches; decommissioning datanodes while reads are in flight.","solutions":["Treat as expected noise: the client automatically re-fetches block locations from the NameNode and reads a surviving replica.","Verify block health once with 'hdfs fsck /file -files -blocks -locations' if the message repeats for the same block.","Reduce staleness in custom clients by re-resolving LocatedBlocks on ReplicaNotFoundException instead of failing the read.","Rule out a serious cause by checking preceding DN log lines for disk or dataset errors, as the message advises."],"exampleFix":"// before: custom client fails on stale location\ntry {\n  reader = new BlockReaderFactory(...).build();\n} catch (ReplicaNotFoundException e) {\n  throw new IOException(\"read failed\", e);\n}\n\n// after: refetch locations and retry on another replica\ntry {\n  reader = new BlockReaderFactory(...).build();\n} catch (ReplicaNotFoundException e) {\n  locatedBlocks = namenode.getBlockLocations(file, offset, len);\n  reader = openReaderFrom(locatedBlocks); // retry with fresh locations\n}","handlingStrategy":"fallback","validationCode":"// Client-side: verify the DN you are about to read from is still in fresh locations\nLocatedBlock lb = namenode.getBlockLocations(file, offset, 1).get(0);\nboolean stillHosted = Arrays.stream(lb.getLocations())\n    .anyMatch(di -> di.getXferAddr().equals(targetXferAddr));\nif (!stillHosted) chooseAnotherReplica(lb);","typeGuard":"static boolean isReplicaNotFound(IOException e) {\n  return e instanceof ReplicaNotFoundException ||\n      (e.getMessage() != null && e.getMessage().startsWith(\"Replica not found for\"));\n}","tryCatchPattern":"try {\n  reader = buildBlockReader(block, targetDn);\n} catch (ReplicaNotFoundException e) {\n  lb = namenode.getBlockLocations(file, offset, len).get(0); // refresh locations\n  reader = buildBlockReader(lb, pickReplica(lb));          // read from another DN\n}","preventionTips":["Refresh block locations before each long-running read session instead of trusting cached DN lists.","Expect this during balancer runs or replication-factor reductions; log at DEBUG unless it repeats for one block.","Check preceding DN logs when it recurs — occasionally it masks a volume failure."],"tags":["hdfs","datanode","read-path","replica","balancer"],"backgroundTag":"replica-not-found","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-23T01:17:44.959Z"}