{"record":{"id":"306408c0e956bfe8","repo":"apache/hadoop","slug":"attempted-to-read-past-end-of-file","errorCode":null,"errorMessage":"Attempted to read past end of file","messagePattern":"Attempted to read past end of file","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSInputStream.java","lineNumber":612,"sourceCode":"        LocatedBlock blk = fetchBlockAt(curOff, remaining, true);\n        assert curOff >= blk.getStartOffset() : \"Block not found\";\n        blockRange.add(blk);\n        long bytesRead = blk.getStartOffset() + blk.getBlockSize() - curOff;\n        remaining -= bytesRead;\n        curOff += bytesRead;\n      }\n      return blockRange;\n    }\n  }\n\n  /**\n   * Open a DataInputStream to a DataNode so that it can be read from.\n   * We get block ID and the IDs of the destinations at startup, from the namenode.\n   */\n  private synchronized DatanodeInfo blockSeekTo(long target)\n      throws IOException {\n    if (target >= getFileLength()) {\n      throw new IOException(\"Attempted to read past end of file\");\n    }\n\n    maybeRegisterBlockRefresh();\n\n    // Will be getting a new BlockReader.\n    closeCurrentBlockReaders();\n\n    //\n    // Connect to best DataNode for desired Block, with potential offset\n    //\n    DatanodeInfo chosenNode;\n    int refetchToken = 1; // only need to get a new access token once\n    int refetchEncryptionKey = 1; // only need to get a new encryption key once\n\n    boolean connectFailedOnce = false;\n\n    while (true) {\n      //","sourceCodeStart":594,"sourceCodeEnd":630,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSInputStream.java#L594-L630","documentation":"blockSeekTo() refuses to open a DataNode reader when the target byte position is at or past the stream's current file length - sequential reads must never reach past EOF. Encountered when the reader's position exceeds the real data extent, most often because the cached length is stale after the file was truncated or the last block finalized shorter than expected.","triggerScenarios":"Sequential read() advancing pos to a value >= getFileLength() while locatedBlocks still describe a longer file; read retries after a concurrent truncate.","commonSituations":"Tail readers on files being truncated/rotated; long-lived streams whose file shrank underneath them; mixed read+truncate workloads.","solutions":["Treat this as EOF: stop reading and re-stat the file before continuing","Re-open the stream after the file shrinks to refresh block locations and length","Prevent concurrent truncation of files that have active readers"],"exampleFix":null,"handlingStrategy":"validation","validationCode":"if (in.getPos() >= fs.getFileStatus(path).getLen()) {\n  return -1; // at EOF; do not trigger blockSeekTo\n}\nint n = in.read(buf);","typeGuard":null,"tryCatchPattern":"try {\n  n = in.read(buf);\n} catch (IOException e) {\n  if (e.getMessage() != null && e.getMessage().equals(\"Attempted to read past end of file\")) {\n    n = -1; // file shrank underneath us: treat as EOF\n    reopenStream();\n  } else throw e;\n}","preventionTips":["In tail loops, stop when pos reaches the current length and re-stat before continuing","Never truncate files that have active sequential readers","Refresh or re-open streams whenever the file may have shrunk"],"tags":["hdfs","eof","read","truncate"],"backgroundTag":"end-of-file","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}