{"record":{"id":"1af4f8046cfeac76","repo":"apache/hadoop","slug":"received-end-of-stream-result-before-all-requested","errorCode":null,"errorMessage":"Received end of stream result before all requestedBytes were received;EndOf stream signal received at offset: %d where as stream was suppose to end at: %d for resource: %s of size: %d","messagePattern":"Received end of stream result before all requestedBytes were received;EndOf stream signal received at offset: (.+?) where as stream was suppose to end at: (.+?) for resource: (.+?) of size: (.+?)","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"hadoop-cloud-storage-project/hadoop-gcp/src/main/java/org/apache/hadoop/fs/gs/GoogleCloudStorageClientReadChannel.java","lineNumber":272,"sourceCode":"          we should not treat it as an error scenario anymore.\n          */\n          if (bytesRead == 0) {\n            LOG.trace(\n                \"Read {} from storage-client's byte channel at position: {} with channel \"\n                    + \"ending at: {} for resourceId: {} of size: {}\",\n                bytesRead, currentPosition, contentChannelEnd, resourceId, objectSize);\n          }\n\n          if (bytesRead < 0) {\n            // Because we don't know decompressed object size for gzip-encoded objects,\n            // assume that this is an object end.\n            if (gzipEncoded) {\n              objectSize = currentPosition;\n              contentChannelEnd = currentPosition;\n            }\n\n            if (currentPosition != contentChannelEnd && currentPosition != objectSize) {\n              throw new IOException(\n                  String.format(\n                      \"Received end of stream result before all requestedBytes were received;\"\n                          + \"EndOf stream signal received at offset: %d where as stream was \"\n                          + \"suppose to end at: %d for resource: %s of size: %d\",\n                      currentPosition, contentChannelEnd, resourceId, objectSize));\n            }\n            // If we have reached an end of a contentChannel but not an end of an object.\n            // then close contentChannel and continue reading an object if necessary.\n            if (contentChannelEnd != objectSize && currentPosition == contentChannelEnd) {\n              closeContentChannel();\n              continue;\n            } else {\n              break;\n            }\n          }\n          totalBytesRead += bytesRead;\n          currentPosition += bytesRead;\n          contentChannelCurrentPosition += bytesRead;","sourceCodeStart":254,"sourceCodeEnd":290,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-cloud-storage-project/hadoop-gcp/src/main/java/org/apache/hadoop/fs/gs/GoogleCloudStorageClientReadChannel.java#L254-L290","documentation":"In ContentReadChannel.readContent, the underlying storage stream returned -1 (end of stream) before currentPosition reached either contentChannelEnd (the requested Range end) or objectSize. Because fewer bytes arrived than the channel requested, integrity cannot be guaranteed and it throws with the observed offset, expected end, resource, and size. gzip-encoded objects are exempt: their decompressed size is unknown, so EOS itself defines objectSize.","triggerScenarios":"Reading through GoogleCloudStorageClientReadChannel when the HTTP range response is truncated mid-body: connection reset by peer/proxy, GCS front end closing the response early, TCP idle timeouts, or object size metadata that no longer matches actual content (concurrent overwrite of a different generation/size).","commonSituations":"Long sequential scans (Spark, distcp) over lossy networks; MTU/middlebox issues truncating responses; reads racing a rewrite of the same object; aggressive connection-reaping proxies or NAT gateways.","solutions":["Retry the operation: reopen the channel, seek to the last good position, and continue — transient truncation is the common case.","Pin the generation when opening (the channel's resourceId carries contentGeneration) so concurrent overwrites cannot shift size/content under the reader.","Investigate the network path (proxy, middlebox, idle timeouts, keep-alive settings) if truncation recurs on specific links.","If reproducible on a single object, compare `gcloud storage stat` size with metadata and re-upload the object."],"exampleFix":"// before\nint n = ch.read(buf); // IOException: end of stream at X, expected Y\n\n// after: reopen and resume from last good position\nlong pos = ch.position();\ntry (SeekableByteChannel retry = gcs.open(itemInfo)) {\n  retry.position(pos);\n  int n = retry.read(buf);\n}","handlingStrategy":"retry","validationCode":null,"typeGuard":null,"tryCatchPattern":"catch (IOException e) {\n  if (e.getMessage() != null && e.getMessage().contains(\"end of stream\")) {\n    try (SeekableByteChannel ch2 = gcs.open(itemInfo)) {\n      ch2.position(lastGoodPosition);   // resume from last verified offset\n      return ch2.read(dst);\n    }\n  }\n  throw e;\n}","preventionTips":["Track the last verified offset in long-running readers so truncated streams can resume instead of restart.","Open readers against a pinned generation so concurrent overwrites cannot change size/content.","Alert on recurring truncation — it usually indicates a network/middlebox problem, not bad data."],"tags":["gcs","read","premature-eof","data-integrity","network"],"backgroundTag":"premature-end-of-stream","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-23T01:17:44.959Z"}