apache/beam · error · S3ClientError

str(e)

Error message

str(e)

What it means

`Client.get_stream` opens a ranged download (`get_object` with a `bytes=N-` Range header) and wraps any failure of that boto3 call in a `messages.S3ClientError` carrying the AWS HTTP status code. The message is the raw `str(e)` from boto3, so the real cause (404, 403, throttling, connection reset) is embedded in the wrapped exception text. The stream is later consumed by `get_range`, which has its own recovery path.

Source

Thrown at sdks/python/apache_beam/io/aws/clients/s3/boto3_client.py:133

    if self._download_request and (
        start != self._download_pos or
        request.bucket != self._download_request.bucket or
        request.object != self._download_request.object):
      self._download_stream.close()
      self._download_stream = None

    # noinspection PyProtectedMember
    if not self._download_stream or self._download_stream._raw_stream.closed:
      try:
        self._download_stream = self.client.get_object(
            Bucket=request.bucket,
            Key=request.object,
            Range='bytes={}-'.format(start))['Body']
        self._download_request = request
        self._download_pos = start
      except Exception as e:
        raise messages.S3ClientError(str(e), get_http_error_code(e))

    return self._download_stream

  @retry.with_exponential_backoff()
  def get_range(self, request, start, end):
    r"""Retrieves an object's contents.

      Args:
        request: (GetRequest) request
        start: (int) start offset
        end: (int) end offset (exclusive)
      Returns:
        (bytes) The response message.
      """
    for i in range(2):
      try:
        stream = self.get_stream(request, start)
        data = stream.read(end - start)

View on GitHub (pinned to 12126d8942)

Solutions

1. Inspect the wrapped message for the boto3 error code (NoSuchKey, AccessDenied, SlowDown) and fix that root cause.
2. Confirm the object exists and the key/bucket in the request are correct before reading.
3. Check IAM permissions and credential freshness (s3:GetObject on bucket/key).
4. For 503/timeout errors, retry: `get_range` already retries with exponential backoff; add your own backoff for stream creation.
5. Reduce parallelism or request rate if seeing SlowDown/throttling.

Example fix

// before
stream = client.get_stream(request, start=0)
// after
try:
  stream = client.get_stream(request, start=0)
except messages.S3ClientError as e:
  if e.code in (500, 503):
    time.sleep(2); stream = client.get_stream(request, start=0)
  else:
    raise

import boto3
def can_read(bucket, key, s3=None):
  s3 = s3 or boto3.client('s3')
  try:
    s3.head_object(Bucket=bucket, Key=key)
    return True
  except s3.exceptions.ClientError:
    return False

def is_retryable(err):
  code = getattr(err, 'code', None)
  return isinstance(err, messages.S3ClientError) and code in (500, 502, 503, 504, None)

try:
  stream = client.get_stream(request, start)
except messages.S3ClientError as e:
  if getattr(e, 'code', None) in (500, 503):
    stream = retry.with_exponential_backoff()(client.get_stream)(request, start)
  else:
    raise

When it happens

Trigger: Calling `get_stream(request, start)` where the initial ranged GET fails: key missing (404), no s3:GetObject permission (403), invalid/expired credentials, S3 throttling (503 SlowDown), or network interruption while opening the download body.

Common situations: Reading large files that were deleted or overwritten mid-pipeline; IAM roles without read access to a bucket; long-running jobs whose temporary credentials expired; S3 throttling under heavy parallel reads.

Related errors

• str(e) (wrapped S3 client error message)
• error listing object keys: %v
• error retrieving page: %v
• error getting object %s: %v
• error getting metadata for object %s: %w

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/43dc0f698ae961c1. Report an issue: GitHub.