{"record":{"id":"fef527ef098c15aa","repo":"apache/hadoop","slug":"end-of-stream-reached-draining-data-between-ranges","errorCode":null,"errorMessage":"End of stream reached draining data between ranges; expected %,d bytes; only drained %,d bytes before -1 returned (position=%,d)","messagePattern":"End of stream reached draining data between ranges; expected %,d bytes; only drained %,d bytes before -1 returned \\(position=%,d\\)","errorType":"exception","errorClass":"EOFException","httpStatus":null,"severity":"error","filePath":"hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AInputStream.java","lineNumber":1012,"sourceCode":"    byte[] drainBuffer;\n    int size = (int)Math.min(InternalConstants.DRAIN_BUFFER_SIZE, drainQuantity);\n    drainBuffer = new byte[size];\n    LOG.debug(\"Draining {} bytes from stream from offset {}; buffer size={}\",\n        drainQuantity, position, size);\n    try {\n      long remaining = drainQuantity;\n      while (remaining > 0) {\n        checkIfVectoredIOStopped();\n        readCount = objectContent.read(drainBuffer, 0, (int)Math.min(size, remaining));\n        LOG.debug(\"Drained {} bytes from stream\", readCount);\n        if (readCount < 0) {\n          // read request failed; often network issues.\n          // no attempt is made to recover at this point.\n          final String s = String.format(\n              \"End of stream reached draining data between ranges; expected %,d bytes;\"\n                  + \" only drained %,d bytes before -1 returned (position=%,d)\",\n              drainQuantity, drainBytes, position + drainBytes);\n          throw new EOFException(s);\n        }\n        drainBytes += readCount;\n        remaining -= readCount;\n      }\n    } finally {\n      getS3AStreamStatistics().readVectoredBytesDiscarded(drainBytes);\n      LOG.debug(\"{} bytes drained from stream \", drainBytes);\n    }\n  }\n\n  /**\n   * Read data from S3 for this range and populate the buffer.\n   * @param range range of data to read.\n   * @param buffer buffer to fill.\n   */\n  private void readSingleRange(FileRange range, ByteBuffer buffer) {\n    LOG.debug(\"Start reading {} from {} \", range, getPathStr());\n    if (range.getLength() == 0) {","sourceCodeStart":994,"sourceCodeEnd":1030,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AInputStream.java#L994-L1030","documentation":"Thrown during readVectored() on S3AInputStream. When requested ranges are close enough to be served from one HTTP stream, S3A drains (reads and discards) the gap between them; this EOFException means the S3 stream returned -1 before drainQuantity gap bytes were consumed. The source comment states no recovery is attempted and network issues are the usual cause.","triggerScenarios":"readVectored(List<ByteRange>, ...) where ranges get merged into one request and the inter-range gap must be drained, and the HTTP response body is cut short by a connection reset, proxy/NAT idle timeout, or the object being replaced by a shorter one mid-read.","commonSituations":"Vectored (columnar) reads from ORC/Hive/Spark over flaky links; corporate proxies or load balancers terminating long S3 GETs; NAT idle timeouts on slow transfers; S3 transient connection drops.","solutions":["Retry the readVectored operation on a fresh stream - this failure is not auto-recovered inside S3A","Tune fs.s3a.connection.timeout / keepalive settings and check proxy/NAT idle timeouts on the S3 network path","Reduce gap draining: lower fs.s3a.vectored.read.max.merged.size or raise fs.s3a.vectored.read.min.seek.size so distant ranges use separate requests instead of one long drain","If it recurs on the same object/ranges, verify the object length and that nothing is concurrently rewriting it"],"exampleFix":"// before\ntry (FSDataInputStream in = fs.open(path)) {\n  List<CompletableFuture<ByteBuffer>> f = in.readVectored(ranges, ByteBuffer::allocate);\n  f.forEach(cf -> cf.join()); // EOF while draining gap between ranges\n}\n\n// after - retry with backoff on a fresh stream\nfor (int attempt = 1; attempt <= 3; attempt++) {\n  try (FSDataInputStream in = fs.open(path)) {\n    in.readVectored(ranges, ByteBuffer::allocate).forEach(cf -> cf.join());\n    break;\n  } catch (CompletionException e) {\n    if (!(e.getCause() instanceof EOFException) || attempt == 3) throw e;\n    Thread.sleep(100L << attempt);\n  }\n}","handlingStrategy":"retry","validationCode":"long len = fs.getFileStatus(path).getLen();\nfor (ByteRange r : ranges) {\n  if (r.getOffset() < 0 || r.getOffset() + r.getLength() > len) {\n    throw new IllegalArgumentException(\"Range outside object: \" + r);\n  }\n}","typeGuard":null,"tryCatchPattern":"catch the EOFException surfacing from the vectored-read futures (often wrapped in CompletionException/ExecutionException), reopen the stream with fs.open(path), and retry the readVectored with capped exponential backoff; give up after a few attempts and surface the original error","preventionTips":["Validate requested ranges against the object length before readVectored","Keep requested ranges sorted and adjacent so drain gaps stay small (tune fs.s3a.vectored.read.min.seek.size / max.merged.size)","Monitor S3A stream statistics for vectored bytes discarded to spot excessive draining"],"tags":["s3a","hadoop-aws","vectored-read","eofexception","network","connection-reset"],"backgroundTag":"truncated-http-response","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}