apache/beam · error · S3ClientError

Tried to list nonexistent S3 path: s3://

Error message

Tried to list nonexistent S3 path: s3://%s/%s

What it means

FakeS3Client.list raises S3ClientError with 'Tried to list nonexistent S3 path: s3://bucket/prefix' (404) when no fake file under the bucket matches the requested prefix. It mimics a nonexistent-prefix listing failure, which real S3 would return as an empty list instead.

Solutions

1. Create fake files under the prefix before calling list
2. Compare the prefix string carefully (trailing slash, case) against seeded keys
3. Ensure the bucket matches the one used in create_file calls
4. If empty results are valid in real S3, handle S3ClientError 404 or use a different fake setup

Example fix

// before
results = fake_client.list(messages.ListRequest('mybucket', 'results/'))
// after
fake_client.create_file('mybucket', 'results/part-0000', b'a')
results = fake_client.list(messages.ListRequest('mybucket', 'results/'))

seeded = [k for (_, k) in fake_client.files if _ == bucket and k.startswith(prefix)]
assert seeded, f'no fake files under s3://{bucket}/{prefix}'

When it happens

Trigger: Calling list() with a bucket/prefix combination where no fake files were created whose keys start with that prefix, so matching_files ends up empty.

Common situations: Unit tests listing a prefix before seeding files; prefix strings with typos or missing/extra trailing slashes; bucket name mismatch between fixture seeding and the code under test.

Understand the failure class

Background: 'Could not be found', 'does not exist', 'not found in database': the resource-not-found family when an ID, slug, key, or URI lookup comes back empty — this error's family across 20 libraries.

Related errors

• Not Found
• The specified bucket does not exist
• Basepath %r must be S3 path.
• Checksum operation failed
• Delete operation failed

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/d072f0a812b72340. Report an issue: GitHub.

Appendix: source

Thrown at sdks/python/apache_beam/io/aws/clients/s3/fake_client.py:106

    """
    # TODO: Do we want to mock out a lack of credentials?
    file_ = self.get_file(request.bucket, request.object)
    return file_.get_metadata()

  def list(self, request):
    bucket = request.bucket
    prefix = request.prefix or ''
    matching_files = []

    for file_bucket, file_name in sorted(iter(self.files)):
      if bucket == file_bucket and file_name.startswith(prefix):
        file_object = self.get_file(file_bucket, file_name).get_metadata()
        matching_files.append(file_object)

    if not matching_files:
      message = 'Tried to list nonexistent S3 path: s3://%s/%s' % (
          bucket, prefix)
      raise messages.S3ClientError(message, 404)

    # Handle pagination.
    items_per_page = 5
    if not request.continuation_token:
      range_start = 0
    else:
      if request.continuation_token not in self.list_continuation_tokens:
        raise ValueError('Invalid page token.')
      range_start = self.list_continuation_tokens[request.continuation_token]
      del self.list_continuation_tokens[request.continuation_token]

    result = messages.ListResponse(
        items=matching_files[range_start:range_start + items_per_page])

    if range_start + items_per_page < len(matching_files):
      next_range_start = range_start + items_per_page
      next_continuation_token = '_page_token_%s_%s_%d' % (
          bucket, prefix, next_range_start)

View on GitHub (pinned to 12126d8942)