{"record":{"id":"f2737e42a9420ba2","repo":"apache/hadoop","slug":"unexpected-end-of-stream-buffer-d-readsize","errorCode":null,"errorMessage":"Unexpected end of stream: buffer[%d], readSize = %d, numRemainingBytes = %d","messagePattern":"Unexpected end of stream: buffer\\[(.+?)\\], readSize = (.+?), numRemainingBytes = (.+?)","errorType":"exception","errorClass":"EOFException","httpStatus":null,"severity":"error","filePath":"hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/prefetch/S3ARemoteObjectReader.java","lineNumber":165,"sourceCode":"    }\n\n    ResponseInputStream<GetObjectResponse> inputStream =\n        remoteObject.openForRead(offset, readSize);\n    int numRemainingBytes = readSize;\n    byte[] bytes = new byte[READ_BUFFER_SIZE];\n\n    int numBytesToRead;\n    int numBytes;\n\n    try {\n      do {\n        numBytesToRead = Math.min(READ_BUFFER_SIZE, numRemainingBytes);\n        numBytes = inputStream.read(bytes, 0, numBytesToRead);\n        if (numBytes < 0) {\n          String message = String.format(\n              \"Unexpected end of stream: buffer[%d], readSize = %d, numRemainingBytes = %d\",\n              buffer.capacity(), readSize, numRemainingBytes);\n          throw new EOFException(message);\n        }\n\n        if (numBytes > 0) {\n          buffer.put(bytes, 0, numBytes);\n          numRemainingBytes -= numBytes;\n        }\n      }\n      while (!this.closed && (numRemainingBytes > 0));\n    } finally {\n      remoteObject.close(inputStream, numRemainingBytes);\n    }\n  }\n}\n","sourceCodeStart":147,"sourceCodeEnd":179,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/prefetch/S3ARemoteObjectReader.java#L147-L179","documentation":"S3ARemoteObjectReader.readToBuffer() reads exactly the remaining bytes of a fetched block from the remote/block stream; if inputStream.read() returns -1 before numRemainingBytes reaches 0, it throws EOFException(\"Unexpected end of stream: buffer[...], readSize = ..., numRemainingBytes = ...\"). The S3 (or cached) stream ended earlier than the content length promised - a truncated read, not normal EOF.","triggerScenarios":"Network connection to S3 cut mid-block (proxy idle timeout, connection reset); block cache data file shorter than expected (disk full, partial write, external deletion of cache dir contents); object deleted or truncated concurrently; retry policy gave up partway through a ranged GET.","commonSituations":"Long-running Spark/Hive jobs on flaky networks or aggressive proxies; nodes whose prefetch cache disk fills; S3 request throttling mid-transfer; antivirus/janitor processes cleaning the prefetch buffer directory while a job runs.","solutions":["Retry the failing operation - truncation from network resets is typically transient and S3A read retries usually resolve it; application-level task retry catches the rest.","Check disk space and health of the prefetch/block cache directory; clean or reconfigure (buffer dir location/size) if it is being truncated or evicted mid-read.","Review proxy/load-balancer idle timeouts on the path to S3 (keep-alive settings) if resets recur at consistent intervals.","If it persists on the same object/offset, verify the object was not replaced/deleted mid-read and that no external process touches the cache dir."],"exampleFix":"// application-level guard (e.g. in a Spark task or job runner)\ntry {\n  return readThrough(stream);\n} catch (EOFException e) {\n  // truncated read of prefetched block: retry the whole operation\n  if (attempt < MAX_RETRIES) {\n    stream.close();\n    stream = fs.open(path); // fresh stream, fresh block fetch\n    return readThrough(stream);\n  }\n  throw e;\n}","handlingStrategy":"retry","validationCode":null,"typeGuard":null,"tryCatchPattern":"for (int attempt = 1; attempt <= 3; attempt++) {\n  try (FSDataInputStream in = fs.open(path)) {\n    return readBlock(in, offset, len);\n  } catch (EOFException e) {\n    if (attempt == 3) throw e;\n    // truncated transfer: next attempt reopens and refetches\n  }\n}","preventionTips":["Retry the read at a higher level (task/job) - mid-stream truncation is usually transient.","Keep the prefetch cache directory on healthy, non-full disks and never clean it externally while jobs run.","Tune proxy/keep-alive timeouts on the S3 data path if resets recur at fixed intervals."],"tags":["hadoop-aws","prefetch","eof","network-truncation","cache-corruption","s3"],"backgroundTag":"premature-end-of-stream","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-23T01:17:44.959Z"}