{"record":{"id":"de96ccdae322b2d4","repo":"apache/hadoop","slug":"could-not-obtain-the-last-block-locations","errorCode":null,"errorMessage":"Could not obtain the last block locations.","messagePattern":"Could not obtain the last block locations\\.","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSInputStream.java","lineNumber":254,"sourceCode":"        if (locatedBlocks == null || refreshLocatedBlocks) {\n          newLocatedBlocks = fetchAndCheckLocatedBlocks(locatedBlocks);\n        } else {\n          newLocatedBlocks = locatedBlocks;\n        }\n\n        long lastBlockLength = getLastBlockLength(newLocatedBlocks);\n        if (lastBlockLength != -1) {\n          setLocatedBlocksFields(newLocatedBlocks, lastBlockLength);\n          return;\n        }\n\n        // Getting last block length as -1 is a special case. When cluster\n        // restarts, DNs may not report immediately. At this time partial block\n        // locations will not be available with NN for getting the length. Lets\n        // retry for 3 times to get the length.\n\n        if (retriesForLastBlockLength-- <= 0) {\n          throw new IOException(\"Could not obtain the last block locations.\");\n        }\n\n        DFSClient.LOG.warn(\"Last block locations not available. \"\n            + \"Datanodes might not have reported blocks completely.\"\n            + \" Will retry for \" + retriesForLastBlockLength + \" times\");\n        waitFor(conf.getRetryIntervalForGetLastBlockLength());\n      }\n    }\n  }\n\n  /**\n   * Set locatedBlocks and related fields, using the passed lastBlockLength.\n   * Should be called within infoLock.\n   */\n  private void setLocatedBlocksFields(LocatedBlocks locatedBlocksToSet, long lastBlockLength) {\n    locatedBlocks = locatedBlocksToSet;\n    lastBlockBeingWrittenLength = lastBlockLength;\n    fileEncryptionInfo = locatedBlocks.getFileEncryptionInfo();","sourceCodeStart":236,"sourceCodeEnd":272,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSInputStream.java#L236-L272","documentation":"Thrown while opening a file whose last block is still under construction. DFSInputStream.openInfo asks the NameNode for located blocks, and when the last block's length comes back as -1 (no DataNode has reported the in-progress replica yet), the client retries up to retriesForLastBlockLength times (3 by default), sleeping dfs.client.retry.interval.get-last-block-length.ms between tries. This IOException means those retries were exhausted: the DNs holding the last block still have not reported it to the NN.","triggerScenarios":"DFSClient.open() / FileSystem.open() on a file with isLastBlockComplete()==false when no listed DN has sent its incremental block report yet; classic right after a NameNode or DataNode restart, or in the window right after a write pipeline was created.","commonSituations":"Tailing log or job-output files that another process is still writing immediately after a cluster restart; opening files created seconds ago on clusters with slow block reporting; NN failover while a writer is active; the retry interval configured too small so 3 attempts finish before DNs report.","solutions":["Wait for DataNodes to finish block reporting (check NN UI for under-replicated blocks / last-contact, or run hdfs fsck on the file) and retry the open","Increase dfs.client.retry.interval.get-last-block-length.ms (default 4000) so the 3 built-in attempts span the reporting delay","Wrap the open in caller-side retry with backoff until the writer closes the file or DNs report","If the writer died leaving the block unfinalized, let NameNode lease recovery finalize the last block before reading"],"exampleFix":"// before\nFSDataInputStream in = fs.open(path); // fails right after cluster restart\n\n// after: retry open while DNs finish reporting the last block\nFSDataInputStream in = null;\nfor (int i = 0; i < 5; i++) {\n  try { in = fs.open(path); break; }\n  catch (IOException e) {\n    if (i == 4 || !e.getMessage().contains(\"last block locations\")) throw e;\n    Thread.sleep(4000L * (i + 1));\n  }\n}","handlingStrategy":"retry","validationCode":null,"typeGuard":"static boolean isLastBlockLocationsUnavailable(IOException e) {\n  return e.getMessage() != null && e.getMessage().contains(\"Could not obtain the last block locations\");\n}","tryCatchPattern":"FSDataInputStream in = null;\nfor (int i = 0; i < 5; i++) {\n  try { in = fs.open(path); break; }\n  catch (IOException e) {\n    if (i == 4 || !isLastBlockLocationsUnavailable(e)) throw e;\n    Thread.sleep(4000L * (i + 1)); // DNs still reporting after restart\n  }\n}","preventionTips":["Do not open files mid-write right after cluster restarts; wait for block reports or writer close","Size dfs.client.retry.interval.get-last-block-length.ms to your cluster's block-report latency","Prefer reading finalized files; for files under construction, poll until closed before opening"],"tags":["hdfs","open","under-construction","block-report","cluster-restart"],"backgroundTag":"block-locations-unavailable","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-23T01:17:44.959Z"}