apache/hadoop · error · IOException

Could not obtain the last block locations.

Error message

Could not obtain the last block locations.

What it means

Thrown while opening a file whose last block is still under construction. DFSInputStream.openInfo asks the NameNode for located blocks, and when the last block's length comes back as -1 (no DataNode has reported the in-progress replica yet), the client retries up to retriesForLastBlockLength times (3 by default), sleeping dfs.client.retry.interval.get-last-block-length.ms between tries. This IOException means those retries were exhausted: the DNs holding the last block still have not reported it to the NN.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSInputStream.java:254

        if (locatedBlocks == null || refreshLocatedBlocks) {
          newLocatedBlocks = fetchAndCheckLocatedBlocks(locatedBlocks);
        } else {
          newLocatedBlocks = locatedBlocks;
        }

        long lastBlockLength = getLastBlockLength(newLocatedBlocks);
        if (lastBlockLength != -1) {
          setLocatedBlocksFields(newLocatedBlocks, lastBlockLength);
          return;
        }

        // Getting last block length as -1 is a special case. When cluster
        // restarts, DNs may not report immediately. At this time partial block
        // locations will not be available with NN for getting the length. Lets
        // retry for 3 times to get the length.

        if (retriesForLastBlockLength-- <= 0) {
          throw new IOException("Could not obtain the last block locations.");
        }

        DFSClient.LOG.warn("Last block locations not available. "
            + "Datanodes might not have reported blocks completely."
            + " Will retry for " + retriesForLastBlockLength + " times");
        waitFor(conf.getRetryIntervalForGetLastBlockLength());
      }
    }
  }

  /**
   * Set locatedBlocks and related fields, using the passed lastBlockLength.
   * Should be called within infoLock.
   */
  private void setLocatedBlocksFields(LocatedBlocks locatedBlocksToSet, long lastBlockLength) {
    locatedBlocks = locatedBlocksToSet;
    lastBlockBeingWrittenLength = lastBlockLength;
    fileEncryptionInfo = locatedBlocks.getFileEncryptionInfo();

View on GitHub (pinned to 2add963021)

Solutions

1. Wait for DataNodes to finish block reporting (check NN UI for under-replicated blocks / last-contact, or run hdfs fsck on the file) and retry the open
2. Increase dfs.client.retry.interval.get-last-block-length.ms (default 4000) so the 3 built-in attempts span the reporting delay
3. Wrap the open in caller-side retry with backoff until the writer closes the file or DNs report
4. If the writer died leaving the block unfinalized, let NameNode lease recovery finalize the last block before reading

Example fix

// before
FSDataInputStream in = fs.open(path); // fails right after cluster restart

// after: retry open while DNs finish reporting the last block
FSDataInputStream in = null;
for (int i = 0; i < 5; i++) {
  try { in = fs.open(path); break; }
  catch (IOException e) {
    if (i == 4 || !e.getMessage().contains("last block locations")) throw e;
    Thread.sleep(4000L * (i + 1));
  }
}

static boolean isLastBlockLocationsUnavailable(IOException e) {
  return e.getMessage() != null && e.getMessage().contains("Could not obtain the last block locations");
}

FSDataInputStream in = null;
for (int i = 0; i < 5; i++) {
  try { in = fs.open(path); break; }
  catch (IOException e) {
    if (i == 4 || !isLastBlockLocationsUnavailable(e)) throw e;
    Thread.sleep(4000L * (i + 1)); // DNs still reporting after restart
  }
}

When it happens

Trigger: DFSClient.open() / FileSystem.open() on a file with isLastBlockComplete()==false when no listed DN has sent its incremental block report yet; classic right after a NameNode or DataNode restart, or in the window right after a write pipeline was created.

Common situations: Tailing log or job-output files that another process is still writing immediately after a cluster restart; opening files created seconds ago on clusters with slow block reporting; NN failover while a writer is active; the retry interval configured too small so 3 attempts finish before DNs report.

Related errors

• Bad status:
• Data node {nodeID} is attempting to report storage ID {datan
• ProcessReport from dead or unregistered node: {nodeID}
• Could not change property {property} from '{oldVal}' to '{ne
• "concat: target file " + target + " is under construction"

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/de96ccdae322b2d4. Report an issue: GitHub.