{"record":{"id":"0a21c1937741ae33","repo":"apache/hadoop","slug":"premature-eof-from-inputstream","errorCode":null,"errorMessage":"Premature EOF from inputStream","messagePattern":"Premature EOF from inputStream","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/IOUtils.java","lineNumber":216,"sourceCode":"  }\n\n  /**\n   * Reads len bytes in a loop.\n   *\n   * @param in InputStream to read from\n   * @param buf The buffer to fill\n   * @param off offset from the buffer\n   * @param len the length of bytes to read\n   * @throws IOException if it could not read requested number of bytes \n   * for any reason (including EOF)\n   */\n  public static void readFully(InputStream in, byte[] buf,\n      int off, int len) throws IOException {\n    int toRead = len;\n    while (toRead > 0) {\n      int ret = in.read(buf, off, toRead);\n      if (ret < 0) {\n        throw new IOException( \"Premature EOF from inputStream\");\n      }\n      toRead -= ret;\n      off += ret;\n    }\n  }\n  \n  /**\n   * Similar to readFully(). Skips bytes in a loop.\n   * @param in The InputStream to skip bytes from\n   * @param len number of bytes to skip.\n   * @throws IOException if it could not skip requested number of bytes \n   * for any reason (including EOF)\n   */\n  public static void skipFully(InputStream in, long len) throws IOException {\n    long amt = len;\n    while (amt > 0) {\n      long ret = in.skip(amt);\n      if (ret == 0) {","sourceCodeStart":198,"sourceCodeEnd":234,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/IOUtils.java#L198-L234","documentation":"Thrown by IOUtils.readFully(InputStream, byte[], int, int) when in.read() returns -1 (end of stream) before the requested 'len' bytes have been read. The method loops until the buffer is full; hitting EOF mid-loop means the stream is shorter than the caller promised, so the read is treated as a hard failure rather than returning a partial count.","triggerScenarios":"Any readFully() call on a stream that ends early: reading a fixed-length record or header from a truncated file, reading a checksummed block where the checksummed length exceeds the data length, or deserializing a Writable (readFields uses readFully) from bytes shorter than its serialized form.","commonSituations":"Truncated files from interrupted uploads, failed distcp transfers, or a still-open writer that hasn't flushed; HDFS files with a missing/corrupt final block; version skew where the writer's serialized layout is longer than the reader expects; a FSDataInputStream positioned past the intended segment.","solutions":["Verify the source length before reading: compare fs.getFileStatus(path).getLen() against the expected byte count.","If the file is truncated, re-fetch or regenerate it (re-run the distcp/upload, re-run the job that wrote it).","If both a writer and reader touch the file concurrently, ensure the writer closes/flushes (hflush/hsync) before the reader opens it.","If a length field drives the read, validate it against remaining bytes and fail with a clearer error before calling readFully."],"exampleFix":"// before: trusts a length read from the stream and fails opaquely at EOF\nbyte[] rec = new byte[len];\nIOUtils.readFully(in, rec, 0, len);\n\n// after: validate remaining bytes first, so truncation is detected with context\nlong remaining = fs.getFileStatus(path).getLen() - ((FSDataInputStream) in).getPos();\nif (remaining < len) {\n  throw new EOFException(\"File \" + path + \" truncated: need \" + len\n      + \" bytes, only \" + remaining + \" left\");\n}\nIOUtils.readFully(in, rec, 0, len);","handlingStrategy":"validation","validationCode":"long remaining = fileLen - ((FSDataInputStream) in).getPos();\nif (remaining < len) {\n  throw new EOFException(path + \" truncated: need \" + len + \", have \" + remaining);\n}\nIOUtils.readFully(in, buf, off, len);","typeGuard":null,"tryCatchPattern":"try {\n  IOUtils.readFully(in, buf, off, len);\n} catch (IOException e) {\n  if (e.getMessage() != null && e.getMessage().contains(\"Premature EOF\")) {\n    handleTruncatedFile(path); // refetch / regenerate / skip\n  } else { throw e; }\n}","preventionTips":["Compare fs.getFileStatus(path).getLen() against the expected size before parsing fixed-layout data.","Writers: call hflush/hsync and close cleanly so readers never see half-written files.","Validate transfers with checksums (distcp -check, crc) so truncation is caught at copy time, not parse time.","Prefer length-prefixed or self-describing formats so record bounds come from the data, not assumptions."],"tags":["io","eof","truncated-file","hadoop-common"],"backgroundTag":"truncated-input-stream","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-23T01:17:44.959Z"}