{"record":{"id":"44c7565561885231","repo":"apache/hadoop","slug":"could-not-obtain-block-file-no-live-nodes-c","errorCode":null,"errorMessage":"Could not obtain block: {} file={} No live nodes contain current block Block locations: {} Dead nodes: {} Ignored nodes: {}","messagePattern":"Could not obtain block: (.+?) file=(.+?) No live nodes contain current block Block locations: (.+?) Dead nodes: (.+?) Ignored nodes: (.+?)","errorType":"exception","errorClass":"BlockMissingException","httpStatus":null,"severity":"critical","filePath":"hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSInputStream.java","lineNumber":1018,"sourceCode":"\n  /**\n   * RefetchLocations should only be called when there are no active requests\n   * to datanodes. In the hedged read case this means futures should be empty.\n   * @param block The locatedBlock to get new datanode locations for.\n   * @param ignoredNodes A list of ignored nodes. This list can be null and can be cleared.\n   * @return the locatedBlock with updated datanode locations.\n   * @throws IOException\n   */\n  private LocatedBlock refetchLocations(LocatedBlock block,\n      Collection<DatanodeInfo> ignoredNodes) throws IOException {\n    String errMsg = getBestNodeDNAddrPairErrorString(block.getLocations(),\n            dfsClient.getDeadNodes(this), ignoredNodes);\n    String blockInfo = block.getBlock() + \" file=\" + src;\n    if (failures >= dfsClient.getConf().getMaxBlockAcquireFailures()) {\n      String description = \"Could not obtain block: \" + blockInfo;\n      DFSClient.LOG.warn(description + errMsg\n          + \". Throwing a BlockMissingException\");\n      throw new BlockMissingException(src, description + errMsg,\n          block.getStartOffset());\n    }\n\n    DatanodeInfo[] nodes = block.getLocations();\n    if (nodes == null || nodes.length == 0) {\n      DFSClient.LOG.info(\"No node available for \" + blockInfo);\n    }\n    DFSClient.LOG.info(\"Could not obtain \" + block.getBlock()\n        + \" from any node: \" + errMsg\n        + \". Will get new block locations from namenode and retry...\");\n    try {\n      // Introducing a random factor to the wait time before another retry.\n      // The wait time is dependent on # of failures and a random factor.\n      // At the first time of getting a BlockMissingException, the wait time\n      // is a random number between 0..3000 ms. If the first retry\n      // still fails, we will wait 3000 ms grace period before the 2nd retry.\n      // Also at the second retry, the waiting window is expanded to 6000 ms\n      // alleviating the request rate from the server. Similarly the 3rd retry","sourceCodeStart":1000,"sourceCodeEnd":1036,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSInputStream.java#L1000-L1036","documentation":"BlockMissingException from refetchLocations(): the failure counter reached dfs.client.max.block.acquireFailures (default 3) while every listed location for the block is dead, locally ignored, or unusable - the message lists live block locations, dead nodes, and ignored nodes. The byte range this block covers is unreadable right now; this is the client's terminal 'block unavailable' signal.","triggerScenarios":"All DNs holding the block are down/decommissioned/network-unreachable; the client ignored every replica after prior failures; replicas lost before re-replication completed.","commonSituations":"Reading files whose replicas concentrated on failed nodes (small replication factor plus node loss); rack/network outage; reading under-replicated files right after DataNode loss.","solutions":["Check cluster health first: dead/under-replicated blocks in the NN UI, DN processes and mounts","Run hdfs fsck -files -blocks -locations on the file to see whether any replica survives anywhere","Bring the DNs holding replicas back (restart/mount disks), then retry the read","If replicas are truly gone, restore the file from source/backup; consider raising replication factor (dfs.replication) to prevent recurrence"],"exampleFix":null,"handlingStrategy":"retry","validationCode":null,"typeGuard":"static boolean isBlockMissing(IOException e) {\n  return e instanceof org.apache.hadoop.hdfs.BlockMissingException;\n}","tryCatchPattern":"try {\n  data = readAll(in);\n} catch (BlockMissingException e) {\n  // all live locations failed: retry later after cluster recovery, or fall back to another source\n  scheduleRetryOrFallbackSource(e);\n}","preventionTips":["Keep dfs.replication >= 3 for valuable data so one DN loss never strands a block","Watch NN under-replicated/dead-node metrics and re-replicate promptly after node loss","Run periodic hdfs fsck so missing replicas are detected before readers need them"],"tags":["hdfs","block-missing","data-loss","read"],"backgroundTag":"block-missing","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-23T01:17:44.959Z"}