apache/beam · error · S3ClientError

Tried to list nonexistent S3 path: s3://

Error message

Tried to list nonexistent S3 path: s3://%s/%s

What it means

`Client.list` raises this explicit `messages.S3ClientError` with HTTP code 404 when `list_objects_v2` succeeds but returns `KeyCount == 0`, i.e. the bucket+prefix matches no objects. The library treats an empty listing as 'the S3 path does not exist' to surface missing inputs early rather than silently returning zero files. Note it is also raised for prefixes that exist but are simply empty.

Solutions

1. Verify the exact s3://bucket/prefix with `aws s3 ls s3://bucket/prefix` using the same credentials — if empty, the path truly has no objects.
2. Fix the prefix string: check for missing trailing slash, wrong case, or wrong bucket/pipeline-option values.
3. Ensure the upstream job/stage that writes the data completed successfully before listing.
4. If an empty prefix is legitimate in your use case, catch `messages.S3ClientError` with code 404 and treat it as an empty result.
5. Check lifecycle policies that may have expired the objects.

Example fix

// before
listing = client.list(messages.ListRequest(bucket='my-bucket', prefix='data/2026-09-12/'))
// after
try:
  listing = client.list(messages.ListRequest(bucket='my-bucket', prefix='data/2026-09-12/'))
except messages.S3ClientError as e:
  if e.code == 404:
    listing = []  # no objects under prefix
  else:
    raise

import boto3
def prefix_has_objects(bucket, prefix, s3=None):
  s3 = s3 or boto3.client('s3')
  resp = s3.list_objects_v2(Bucket=bucket, Prefix=prefix, MaxKeys=1)
  return resp.get('KeyCount', 0) > 0

def is_missing_path(err):
  return isinstance(err, messages.S3ClientError) and getattr(err, 'code', None) == 404

try:
  listing = client.list(request)
except messages.S3ClientError as e:
  if getattr(e, 'code', None) == 404:
    listing = []  # empty/missing prefix is acceptable
  else:
    raise

When it happens

Trigger: Calling `list(request)` where the given `bucket`/`prefix` combination has zero keys: the path was never written, was deleted, the prefix string is wrong (e.g. missing/extra slash), or the job's output path from a previous stage is misconfigured.

Common situations: Downstream job pointing at an upstream output path that never ran or failed; typos in the s3 path or prefix (case sensitivity, missing '/'); environment-specific config pointing to a test bucket; deleted/ lifecycle-expired objects.

Understand the failure class

Background: 'Could not be found', 'does not exist', 'not found in database': the resource-not-found family when an ID, slug, key, or URI lookup comes back empty — this error's family across 20 libraries.

Related errors

• Basepath %r must be S3 path.
• Invalid path
• Must provide one of client or options
• Path %r must be S3 path.
• All parts but the last must be larger than

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/2695ae3fdc17360d. Report an issue: GitHub.

Appendix: source

Thrown at sdks/python/apache_beam/io/aws/clients/s3/boto3_client.py:185

    Args:
      request: (ListRequest) input message
    Returns:
      (ListResponse) The response message.
    """
    kwargs = {'Bucket': request.bucket, 'Prefix': request.prefix}

    if request.continuation_token is not None:
      kwargs['ContinuationToken'] = request.continuation_token

    try:
      boto_response = self.client.list_objects_v2(**kwargs)
    except Exception as e:
      raise messages.S3ClientError(str(e), get_http_error_code(e))

    if boto_response['KeyCount'] == 0:
      message = 'Tried to list nonexistent S3 path: s3://%s/%s' % (
          request.bucket, request.prefix)
      raise messages.S3ClientError(message, 404)

    items = [
        messages.Item(
            etag=content['ETag'],
            key=content['Key'],
            last_modified=content['LastModified'],
            size=content['Size']) for content in boto_response['Contents']
    ]

    try:
      next_token = boto_response['NextContinuationToken']
    except KeyError:
      next_token = None

    response = messages.ListResponse(items, next_token)
    return response

  def create_multipart_upload(self, request):

View on GitHub (pinned to 12126d8942)