apache/hadoop · critical · BlockMissingException

Could not obtain block: {} file={} No live nodes contain cur

Error message

Could not obtain block: {} file={} No live nodes contain current block Block locations: {} Dead nodes: {} Ignored nodes: {}

What it means

BlockMissingException from refetchLocations(): the failure counter reached dfs.client.max.block.acquireFailures (default 3) while every listed location for the block is dead, locally ignored, or unusable - the message lists live block locations, dead nodes, and ignored nodes. The byte range this block covers is unreadable right now; this is the client's terminal 'block unavailable' signal.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSInputStream.java:1018

  /**
   * RefetchLocations should only be called when there are no active requests
   * to datanodes. In the hedged read case this means futures should be empty.
   * @param block The locatedBlock to get new datanode locations for.
   * @param ignoredNodes A list of ignored nodes. This list can be null and can be cleared.
   * @return the locatedBlock with updated datanode locations.
   * @throws IOException
   */
  private LocatedBlock refetchLocations(LocatedBlock block,
      Collection<DatanodeInfo> ignoredNodes) throws IOException {
    String errMsg = getBestNodeDNAddrPairErrorString(block.getLocations(),
            dfsClient.getDeadNodes(this), ignoredNodes);
    String blockInfo = block.getBlock() + " file=" + src;
    if (failures >= dfsClient.getConf().getMaxBlockAcquireFailures()) {
      String description = "Could not obtain block: " + blockInfo;
      DFSClient.LOG.warn(description + errMsg
          + ". Throwing a BlockMissingException");
      throw new BlockMissingException(src, description + errMsg,
          block.getStartOffset());
    }

    DatanodeInfo[] nodes = block.getLocations();
    if (nodes == null || nodes.length == 0) {
      DFSClient.LOG.info("No node available for " + blockInfo);
    }
    DFSClient.LOG.info("Could not obtain " + block.getBlock()
        + " from any node: " + errMsg
        + ". Will get new block locations from namenode and retry...");
    try {
      // Introducing a random factor to the wait time before another retry.
      // The wait time is dependent on # of failures and a random factor.
      // At the first time of getting a BlockMissingException, the wait time
      // is a random number between 0..3000 ms. If the first retry
      // still fails, we will wait 3000 ms grace period before the 2nd retry.
      // Also at the second retry, the waiting window is expanded to 6000 ms
      // alleviating the request rate from the server. Similarly the 3rd retry

View on GitHub (pinned to 2add963021)

Solutions

Check cluster health first: dead/under-replicated blocks in the NN UI, DN processes and mounts
Run hdfs fsck -files -blocks -locations on the file to see whether any replica survives anywhere
Bring the DNs holding replicas back (restart/mount disks), then retry the read
If replicas are truly gone, restore the file from source/backup; consider raising replication factor (dfs.replication) to prevent recurrence

Defensive patterns

Strategy: retry

Type guard

static boolean isBlockMissing(IOException e) {
  return e instanceof org.apache.hadoop.hdfs.BlockMissingException;
}

Try / catch

try {
  data = readAll(in);
} catch (BlockMissingException e) {
  // all live locations failed: retry later after cluster recovery, or fall back to another source
  scheduleRetryOrFallbackSource(e);
}

Prevention

Keep dfs.replication >= 3 for valuable data so one DN loss never strands a block
Watch NN under-replicated/dead-node metrics and re-replicate promptly after node loss
Run periodic hdfs fsck so missing replicas are detected before readers need them

When it happens

Trigger: All DNs holding the block are down/decommissioned/network-unreachable; the client ignored every replica after prior failures; replicas lost before re-replication completed.

Common situations: Reading files whose replicas concentrated on failed nodes (small replication factor plus node loss); rack/network outage; reading under-replicated files right after DataNode loss.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/44c7565561885231. Report an issue: GitHub.