{"record":{"id":"c1936f9f6aa94bd6","repo":"apache/hadoop","slug":"end-of-file-reached-before-reading-fully-c1936f","errorCode":null,"errorMessage":"End of file reached before reading fully.","messagePattern":"End of file reached before reading fully\\.","errorType":"exception","errorClass":"EOFException","httpStatus":null,"severity":"error","filePath":"hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSInputStream.java","lineNumber":1774,"sourceCode":"    throw new IOException(\"Mark/reset not supported\");\n  }\n\n  @Override\n  public int read(long position, final ByteBuffer buf) throws IOException {\n    if (!buf.hasRemaining()) {\n      return 0;\n    }\n    return pread(position, buf);\n  }\n\n  @Override\n  public void readFully(long position, final ByteBuffer buf)\n      throws IOException {\n    int nread = 0;\n    while (buf.hasRemaining()) {\n      int nbytes = read(position + nread, buf);\n      if (nbytes < 0) {\n        throw new EOFException(FSExceptionMessages.EOF_IN_READ_FULLY);\n      }\n      nread += nbytes;\n    }\n  }\n\n  /** Utility class to encapsulate data node info and its address. */\n  static final class DNAddrPair {\n    final DatanodeInfo info;\n    final InetSocketAddress addr;\n    final StorageType storageType;\n    final LocatedBlock block;\n\n    DNAddrPair(DatanodeInfo info, InetSocketAddress addr,\n        StorageType storageType, LocatedBlock block) {\n      this.info = info;\n      this.addr = addr;\n      this.storageType = storageType;\n      this.block = block;","sourceCodeStart":1756,"sourceCodeEnd":1792,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSInputStream.java#L1756-L1792","documentation":"DFSInputStream.readFully(long position, ByteBuffer buf) loops read(position + nread, buf) until the buffer is full; if a positional read returns -1 (EOF) before the buffer fills, it throws EOFException('End of file reached before reading fully.'). readFully is all-or-nothing by contract: the requested range must lie entirely within [0, fileLength). The position argument is absolute, not relative to the current stream position.","triggerScenarios":"readFully(pos, buf) where pos + buf.remaining() > file length; offsets computed from a stale file length obtained before another process truncated the file; racing append (reader stat'ed length L, file truncated to less before the read).","commonSituations":"Reading fixed-size footers/trailers at (fileLen - footerSize) when the file is shorter than footerSize; MapReduce/Spark input formats computing split boundaries against a cached file length; concurrent compaction or rewrite jobs truncating files under active readers.","solutions":["Validate up front: long remaining = fs.getFileStatus(path).getLen() - position; ensure remaining >= buf.remaining() before calling readFully.","If the file may be changing underneath you, re-stat the length on EOFException and decide whether to retry (file grew) or fail (file truncated).","If partial data is acceptable, replace readFully with a positional read() loop that tolerates -1 instead of demanding the full range.","If you control the writer, make truncation/rewrite atomic (write to temp path then rename) so readers never see a shrinking file."],"exampleFix":"// before\nByteBuffer footer = ByteBuffer.allocate(FOOTER_LEN);\nin.readFully(fileLenCached - FOOTER_LEN, footer); // throws if file shrank\n\n// after\nlong fileLen = fs.getFileStatus(path).getLen(); // fresh length\nByteBuffer footer = ByteBuffer.allocate(FOOTER_LEN);\nif (fileLen < FOOTER_LEN) throw new EOFException(path + \" too small for footer\");\nin.readFully(fileLen - FOOTER_LEN, footer);","handlingStrategy":"validation","validationCode":"long fileLen = fs.getFileStatus(path).getLen(); // fresh, not cached\nlong remaining = fileLen - position;\nif (remaining < 0) throw new IllegalArgumentException(\"position past EOF: \" + position);\nif (buf.remaining() > remaining) buf.limit(buf.position() + (int) remaining);\nin.readFully(position, buf);","typeGuard":null,"tryCatchPattern":"try {\n  in.readFully(position, buf);\n} catch (EOFException e) {\n  long freshLen = fs.getFileStatus(path).getLen(); // file may have been truncated\n  if (position + nread < freshLen) retryReadFully(position + nread, buf); // grew: continue\n  else throw new IllegalStateException(\"file truncated under reader: \" + path, e);\n}","preventionTips":["Always compute read ranges from a freshly stated file length.","Write-to-temp-then-rename so readers never observe truncation.","Use read() loops instead of readFully when partial reads are acceptable.","Check position + buf.remaining() <= len before every readFully call."],"tags":["hdfs","hdfs-client","eof","read-fully","positional-read"],"backgroundTag":"read-past-eof","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}