apache/hadoop · error · IOException

Got EOF but currentPos = ${currentPos} < filelength = ${file

Q: What causes "Got EOF but currentPos = ${currentPos} < filelength = ${fileLength}" in apache/hadoop?

Retry by reopening the stream and resuming from getPos() — ranged HTTP makes resumption cheap

Error message

Got EOF but currentPos = ${currentPos} < filelength = ${fileLength}

What it means

ByteRangeInputStream derives fileLength from the HTTP response (Content-Length = startOffset + stream length) and tracks currentPos in update(). When a read returns -1 (EOF) while currentPos is still below fileLength, the server delivered fewer bytes than it promised and this IOException is thrown. It indicates a truncated HTTP body — the connection ended mid-stream — rather than a clean end of file.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/web/ByteRangeInputStream.java:194

    final List<String> values = headers.get(key);
    if (values != null) {
      for(String v : values) {
        for(final StringTokenizer t = new StringTokenizer(v, ",");
            t.hasMoreTokens(); ) {
          if (value.equalsIgnoreCase(t.nextToken())) {
            return true;
          }
        }
      }
    }
    return false;
  }

  private int update(final int n) throws IOException {
    if (n != -1) {
      currentPos += n;
    } else if (fileLength != null && currentPos < fileLength) {
      throw new IOException("Got EOF but currentPos = " + currentPos
          + " < filelength = " + fileLength);
    }
    return n;
  }

  @Override
  public int read() throws IOException {
    final int b = getInputStream().read();
    update((b == -1) ? -1 : 1);
    return b;
  }

  @Override
  public int read(@Nonnull byte b[], int off, int len) throws IOException {
    return update(getInputStream().read(b, off, len));
  }

  /**

View on GitHub (pinned to 2add963021)

Solutions

Retry by reopening the stream and resuming from getPos() — ranged HTTP makes resumption cheap
Raise idle/read timeouts on proxies and load balancers in the path (they cut long transfers first)
If reproducible at the same offset on the same file, run `hdfs fsck <path>` to check for a corrupted/under-replicated block on the serving DataNode
Prefer native hdfs:// client over webhdfs:// for large reads when network reliability is poor

Example fix

// resume pattern after premature EOF
long pos = in.getPos();
try {
  while (in.read(buf, 0, buf.length) != -1) { /* consume */ }
} catch (IOException e) { // 'Got EOF but currentPos < filelength'
  in.close();
  in = fs.open(path);
  in.seek(pos);
}

Defensive patterns

Strategy: retry

Try / catch

long pos = in.getPos();
try {
  IOUtils.copyFully(in, out); // conceptual full read
} catch (IOException e) {
  if (e.getMessage() != null && e.getMessage().contains("Got EOF but currentPos")) {
    in.close();
    try (FSDataInputStream resume = fs.open(path)) {
      resume.seek(pos);
      IOUtils.copyFully(resume, out); // continue from last known position
    }
  } else { throw e; }
}

Prevention

For large WebHDFS reads, checkpoint progress (getPos) so truncated transfers can resume instead of restarting
Raise proxy/LB read timeouts to exceed large transfer durations
Check DN health and `hdfs fsck` when truncation repeats at the same offset

When it happens

Trigger: Reading a WebHDFS file when the HTTP connection is cut before all promised bytes arrive: network drop between client and DataNode/NameNode, proxy or firewall idle/read timeout killing the connection mid-transfer, or a DataNode dying mid-read. seek() then read() makes it visible immediately on the new ranged request.

Common situations: Large file reads over WebHDFS through firewalls/LBs with aggressive timeouts; flaky networks to remote clusters; DataNodes restarting or being decommissioned during the read.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/ea850a33ee130f63. Report an issue: GitHub.