apache/hadoop · error · IOException

Received end of stream result before all requestedBytes were

Error message

Received end of stream result before all requestedBytes were received;EndOf stream signal received at offset: %d where as stream was suppose to end at: %d for resource: %s of size: %d

What it means

In ContentReadChannel.readContent, the underlying storage stream returned -1 (end of stream) before currentPosition reached either contentChannelEnd (the requested Range end) or objectSize. Because fewer bytes arrived than the channel requested, integrity cannot be guaranteed and it throws with the observed offset, expected end, resource, and size. gzip-encoded objects are exempt: their decompressed size is unknown, so EOS itself defines objectSize.

Source

Thrown at hadoop-cloud-storage-project/hadoop-gcp/src/main/java/org/apache/hadoop/fs/gs/GoogleCloudStorageClientReadChannel.java:272

          we should not treat it as an error scenario anymore.
          */
          if (bytesRead == 0) {
            LOG.trace(
                "Read {} from storage-client's byte channel at position: {} with channel "
                    + "ending at: {} for resourceId: {} of size: {}",
                bytesRead, currentPosition, contentChannelEnd, resourceId, objectSize);
          }

          if (bytesRead < 0) {
            // Because we don't know decompressed object size for gzip-encoded objects,
            // assume that this is an object end.
            if (gzipEncoded) {
              objectSize = currentPosition;
              contentChannelEnd = currentPosition;
            }

            if (currentPosition != contentChannelEnd && currentPosition != objectSize) {
              throw new IOException(
                  String.format(
                      "Received end of stream result before all requestedBytes were received;"
                          + "EndOf stream signal received at offset: %d where as stream was "
                          + "suppose to end at: %d for resource: %s of size: %d",
                      currentPosition, contentChannelEnd, resourceId, objectSize));
            }
            // If we have reached an end of a contentChannel but not an end of an object.
            // then close contentChannel and continue reading an object if necessary.
            if (contentChannelEnd != objectSize && currentPosition == contentChannelEnd) {
              closeContentChannel();
              continue;
            } else {
              break;
            }
          }
          totalBytesRead += bytesRead;
          currentPosition += bytesRead;
          contentChannelCurrentPosition += bytesRead;

View on GitHub (pinned to 2add963021)

Solutions

1. Retry the operation: reopen the channel, seek to the last good position, and continue — transient truncation is the common case.
2. Pin the generation when opening (the channel's resourceId carries contentGeneration) so concurrent overwrites cannot shift size/content under the reader.
3. Investigate the network path (proxy, middlebox, idle timeouts, keep-alive settings) if truncation recurs on specific links.
4. If reproducible on a single object, compare `gcloud storage stat` size with metadata and re-upload the object.

Example fix

// before
int n = ch.read(buf); // IOException: end of stream at X, expected Y

// after: reopen and resume from last good position
long pos = ch.position();
try (SeekableByteChannel retry = gcs.open(itemInfo)) {
  retry.position(pos);
  int n = retry.read(buf);
}

catch (IOException e) {
  if (e.getMessage() != null && e.getMessage().contains("end of stream")) {
    try (SeekableByteChannel ch2 = gcs.open(itemInfo)) {
      ch2.position(lastGoodPosition);   // resume from last verified offset
      return ch2.read(dst);
    }
  }
  throw e;
}

When it happens

Trigger: Reading through GoogleCloudStorageClientReadChannel when the HTTP range response is truncated mid-body: connection reset by peer/proxy, GCS front end closing the response early, TCP idle timeouts, or object size metadata that no longer matches actual content (concurrent overwrite of a different generation/size).

Common situations: Long sequential scans (Spark, distcp) over lossy networks; MTU/middlebox issues truncating responses; reads racing a rewrite of the same object; aggressive connection-reaping proxies or NAT gateways.

Related errors

• Exception occurred while closing channel '%s'
• Unable to update the boundaries/Range of contentChannel %s
• Item not found: %s
• Null IO stream from reopen of ({}) {}
• read failed of {}, inputStream is {}

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/1af4f8046cfeac76. Report an issue: GitHub.