{"record":{"id":"7f53af5e5e3a65b3","repo":"apache/hadoop","slug":"attempted-to-read-past-end-of-file-7f53af","errorCode":null,"errorMessage":"Attempted to read past end of file","messagePattern":"Attempted to read past end of file","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSStripedInputStream.java","lineNumber":160,"sourceCode":"  protected ByteBuffer getCurStripeBuf() {\n    return curStripeBuf;\n  }\n\n  protected ByteBufferPool getBufferPool() {\n    return BUFFER_POOL;\n  }\n\n  protected ThreadPoolExecutor getStripedReadsThreadPool(){\n    return dfsClient.getStripedReadsThreadPool();\n  }\n  /**\n   * When seeking into a new block group, create blockReader for each internal\n   * block in the group.\n   */\n  @VisibleForTesting\n  synchronized void blockSeekTo(long target) throws IOException {\n    if (target >= getFileLength()) {\n      throw new IOException(\"Attempted to read past end of file\");\n    }\n\n    maybeRegisterBlockRefresh();\n\n    // Will be getting a new BlockReader.\n    closeCurrentBlockReaders();\n\n    // Compute desired striped block group\n    LocatedStripedBlock targetBlockGroup = getBlockGroupAt(target);\n    // Update current position\n    this.pos = target;\n    this.blockEnd = targetBlockGroup.getStartOffset() +\n        targetBlockGroup.getBlockSize() - 1;\n    currentLocatedBlock = targetBlockGroup;\n  }\n\n  @Override\n  public synchronized void close() throws IOException {","sourceCodeStart":142,"sourceCodeEnd":178,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSStripedInputStream.java#L142-L178","documentation":"DFSStripedInputStream.blockSeekTo(long target) - called by read paths to (re)position onto a block group of an erasure-coded (EC) file - throws IOException('Attempted to read past end of file') when target >= getFileLength(). For striped files this check sits at the top of the seek-to-block path, so any read/skip/seek continuation at or beyond EOF of an EC file surfaces here rather than as the replicated stream's EOFException variants.","triggerScenarios":"Continuing to read() a striped file after a previous read already returned -1 (loop not honoring EOF); positional pread(position, ...) with position >= file length on an EC file; skip() clamped arithmetic feeding an out-of-range target back into blockSeekTo; racing truncation shrinking the EC file between length check and read.","commonSituations":"Custom record readers / parsers run over erasure-coded datasets (e.g. EC-enabled zones in Hive/Spark warehouses) that ignore the -1 EOF sentinel; readers using a cached file length against files concurrently compacted/truncated; code tested only on replicated files where the equivalent mistake throws a different (caught) exception type.","solutions":["Break the read loop on the first read() == -1 and never call read again after it.","Before positional reads, validate position < fs.getFileStatus(path).getLen() (re-stat, do not use a cached length).","If truncation races are expected, catch this IOException at the top level, re-stat the file, and treat the shorter length as the source of truth.","Prefer pread/readFully with range checks computed from a fresh length over raw byte-at-a-time loops on EC files."],"exampleFix":"// before\nint n;\nwhile ((n = in.read(buf, 0, buf.length)) != 0) { // wrong sentinel: -1 keeps looping\n  sink.write(buf, 0, n);\n}\n\n// after\nint n;\nwhile ((n = in.read(buf, 0, buf.length)) != -1) {\n  sink.write(buf, 0, n);\n}","handlingStrategy":"validation","validationCode":"long fileLen = fs.getFileStatus(path).getLen(); // fresh\nlong remaining = fileLen - position;\nif (remaining <= 0) return DONE; // at/after EOF on an EC file - nothing to read\nif (length > remaining) length = (int) remaining;\nint n = stripedIn.read(position, buf, 0, length);","typeGuard":null,"tryCatchPattern":"try {\n  n = in.read(buf, 0, buf.length);\n} catch (IOException e) {\n  if (String.valueOf(e.getMessage()).contains(\"Attempted to read past end of file\")) {\n    long freshLen = fs.getFileStatus(path).getLen();\n    if (position >= freshLen) return DONE; // genuine EOF (file truncated): stop cleanly\n    in.seek(Math.min(position, freshLen - 1)); continue; // shrank mid-read: clamp and retry\n  }\n  throw e;\n}","preventionTips":["Treat read() == -1 as terminal - never read again after it.","Compute read ranges from a freshly stated file length, not a cached one.","Make truncation atomic (temp + rename) so EC files never shrink under readers.","Test readers against erasure-coded files, not just replicated ones - exception types differ."],"tags":["hdfs","hdfs-client","erasure-coding","striped","eof","read"],"backgroundTag":"read-past-eof","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}