What causes "Not Found" in apache/beam?

Insert the fake file before reading: fake_client.create_file(bucket, obj, contents)

apache/beam · error · S3ClientError

Not Found

Error message

Not Found

What it means

FakeS3Client.get_file raises S3ClientError('Not Found', 404) when the requested (bucket, key) pair is not present in its in-memory file store. It is the test-double equivalent of a 404 NoSuchKey from real S3.

Solutions

Insert the fake file before reading: fake_client.create_file(bucket, obj, contents)
Assert on the correct bucket/key names matching what the code under test requests
Seed the FakeS3Client with all fixture files the pipeline will touch
Wrap reads in try/except S3ClientError with a 404 check if absence is expected

Example fix

// before
metadata = fake_client.get_object_metadata(messages.GetObjectMetadataRequest('mybucket', 'missing.txt'))
// after
fake_client.create_file('mybucket', 'missing.txt', b'data')
metadata = fake_client.get_object_metadata(messages.GetObjectMetadataRequest('mybucket', 'missing.txt'))

Defensive patterns

Strategy: try-catch

Validate before calling

if (bucket, obj) not in fake_client.files:
    fake_client.create_file(bucket, obj, b'fixture-data')

Try / catch

try:
    meta = fake_client.get_object_metadata(req)
except messages.S3ClientError as e:
    if e.code == 404:
        meta = None  # expected absence
    else:
        raise

Prevention

Seed all fixture files before the code under test runs
Use shared constants for bucket/key names between setup and assertions
Track delete_file calls in tests to avoid reads after deletes
Assert on fake_client.files contents when debugging

When it happens

Trigger: Calling get_object_metadata, list, get_range, or copy against a FakeS3Client whose internal files dict lacks (bucket, obj); e.g. a file was never inserted or was deleted by delete_file.

Common situations: Unit tests that forgot to call create_file/put before reading; test fixtures referencing a key that was renamed; delete_file called earlier in the same test than expected.

Understand the failure class

Background: 'Could not be found', 'does not exist', 'not found in database': the resource-not-found family when an ID, slug, key, or URI lookup comes back empty — this error's family across 20 libraries.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/cad3f37bfffad8cd. Report an issue: GitHub.

Appendix: source

Thrown at sdks/python/apache_beam/io/aws/clients/s3/fake_client.py:75

    self.files = {}
    self.list_continuation_tokens = {}
    self.multipart_uploads = {}

    # boto3 has different behavior when running some operations against a bucket
    # that exists vs. against one that doesn't. To emulate that behavior, the
    # mock client keeps a set of bucket names that it knows "exist".
    self.known_buckets = set()

  def add_file(self, f):
    self.files[(f.bucket, f.key)] = f
    if f.bucket not in self.known_buckets:
      self.known_buckets.add(f.bucket)

  def get_file(self, bucket, obj):
    try:
      return self.files[bucket, obj]
    except:
      raise messages.S3ClientError('Not Found', 404)

  def delete_file(self, bucket, obj):
    del self.files[(bucket, obj)]

  def get_object_metadata(self, request):
    r"""Retrieves an object's metadata.

    Args:
      request: (GetRequest) input message

    Returns:
      (Item) The response message.
    """
    # TODO: Do we want to mock out a lack of credentials?
    file_ = self.get_file(request.bucket, request.object)
    return file_.get_metadata()

  def list(self, request):

View on GitHub (pinned to 12126d8942)