apache/beam · error · S3ClientError

str(e) (wrapped S3 client error message)

Error message

str(e) (wrapped S3 client error message)

What it means

`Client.get_object_metadata` wraps any exception raised by boto3's `head_object` call into a `messages.S3ClientError`, attaching the AWS HTTP status code extracted from the exception's `response` attribute. The message is simply `str(e)` of the underlying boto3 `ClientError` (or other exception), so the actual cause (404 NoSuchKey, 403 AccessDenied, connection failure, etc.) is in the wrapped text. This is a uniform error-translator so the rest of the S3 filesystem layer sees one exception type with an HTTP code.

Source

Thrown at sdks/python/apache_beam/io/aws/clients/s3/boto3_client.py:95

    self._download_request = None
    self._download_stream = None
    self._download_pos = 0

  def get_object_metadata(self, request):
    """Retrieves an object's metadata.

    Args:
      request: (GetRequest) input message

    Returns:
      (Object) The response message.
    """
    kwargs = {'Bucket': request.bucket, 'Key': request.object}

    try:
      boto_response = self.client.head_object(**kwargs)
    except Exception as e:
      raise messages.S3ClientError(str(e), get_http_error_code(e))

    item = messages.Item(
        boto_response['ETag'],
        request.object,
        boto_response['LastModified'],
        boto_response['ContentLength'],
        boto_response['ContentType'])

    return item

  def get_stream(self, request, start):
    """Opens a stream object starting at the given position.

    Args:
      request: (GetRequest) request
      start: (int) start offset
    Returns:
      (Stream) Boto3 stream object.

View on GitHub (pinned to 12126d8942)

Solutions

1. Read the wrapped `str(e)` message: it contains the boto3 ClientError code (e.g. 'NoSuchKey', 'AccessDenied') and fix the underlying cause accordingly.
2. Verify the object key and bucket in the request are correct (print request.bucket / request.object); a 404 usually means the path or concatenation logic is wrong.
3. Check AWS credentials and IAM permissions (s3:GetObject on the key) via `aws s3api head-object` with the same credentials.
4. Ensure boto3 has a region configured (AWS_REGION env var, ~/.aws/config, or pipeline options) to avoid endpoint errors.
5. For transient errors, rely on the retry wrapper or catch S3ClientError and re-issue with backoff.

Example fix

// before
metadata = client.get_object_metadata(messages.GetRequest(bucket='my-bucket', object='data/out.json'))
// after
try:
  metadata = client.get_object_metadata(messages.GetRequest(bucket='my-bucket', object='data/out.json'))
except messages.S3ClientError as e:
  if e.code == 404:
    metadata = None  # treat missing object as empty
  else:
    raise

import boto3
def object_exists(bucket, key, s3=None):
  s3 = s3 or boto3.client('s3')
  try:
    s3.head_object(Bucket=bucket, Key=key)
    return True
  except s3.exceptions.ClientError as e:
    if e.response['Error']['Code'] in ('404', 'NoSuchKey', 'NotFound'):
      return False
    raise

def is_not_found(err):
  return isinstance(err, messages.S3ClientError) and getattr(err, 'code', None) == 404

try:
  item = client.get_object_metadata(request)
except messages.S3ClientError as e:
  if getattr(e, 'code', None) == 404:
    item = None
  else:
    raise

When it happens

Trigger: Calling `get_object_metadata(request)` when the S3 HEAD request fails: the object key does not exist (404), the caller lacks s3:GetObject permission (403), the bucket does not exist or is in the wrong region, credentials are invalid/expired, or a network/timeout error occurs during `client.head_object`.

Common situations: File matching against a stale path after upstream step renamed keys; typos in bucket or prefix concatenation; IAM role missing GetObject; temporary AWS outage or throttling; boto3 not configured with region so endpoint resolution fails.

Related errors

• str(e)
• error listing object keys: %v
• error retrieving page: %v
• error getting object %s: %v
• error getting metadata for object %s: %w

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/3aac554e2d9f1078. Report an issue: GitHub.