{"record":{"id":"2695ae3fdc17360d","repo":"apache/beam","slug":"tried-to-list-nonexistent-s3-path-s3-s-s","errorCode":null,"errorMessage":"Tried to list nonexistent S3 path: s3://%s/%s","messagePattern":"Tried to list nonexistent S3 path: s3://(.+?)/(.+?)","errorType":"http","errorClass":"S3ClientError","httpStatus":404,"severity":"error","filePath":"sdks/python/apache_beam/io/aws/clients/s3/boto3_client.py","lineNumber":185,"sourceCode":"    Args:\n      request: (ListRequest) input message\n    Returns:\n      (ListResponse) The response message.\n    \"\"\"\n    kwargs = {'Bucket': request.bucket, 'Prefix': request.prefix}\n\n    if request.continuation_token is not None:\n      kwargs['ContinuationToken'] = request.continuation_token\n\n    try:\n      boto_response = self.client.list_objects_v2(**kwargs)\n    except Exception as e:\n      raise messages.S3ClientError(str(e), get_http_error_code(e))\n\n    if boto_response['KeyCount'] == 0:\n      message = 'Tried to list nonexistent S3 path: s3://%s/%s' % (\n          request.bucket, request.prefix)\n      raise messages.S3ClientError(message, 404)\n\n    items = [\n        messages.Item(\n            etag=content['ETag'],\n            key=content['Key'],\n            last_modified=content['LastModified'],\n            size=content['Size']) for content in boto_response['Contents']\n    ]\n\n    try:\n      next_token = boto_response['NextContinuationToken']\n    except KeyError:\n      next_token = None\n\n    response = messages.ListResponse(items, next_token)\n    return response\n\n  def create_multipart_upload(self, request):","sourceCodeStart":167,"sourceCodeEnd":203,"githubUrl":"https://github.com/apache/beam/blob/12126d8942aaf848030c478b4c6a28c6af861c66/sdks/python/apache_beam/io/aws/clients/s3/boto3_client.py#L167-L203","documentation":"`Client.list` raises this explicit `messages.S3ClientError` with HTTP code 404 when `list_objects_v2` succeeds but returns `KeyCount == 0`, i.e. the bucket+prefix matches no objects. The library treats an empty listing as 'the S3 path does not exist' to surface missing inputs early rather than silently returning zero files. Note it is also raised for prefixes that exist but are simply empty.","triggerScenarios":"Calling `list(request)` where the given `bucket`/`prefix` combination has zero keys: the path was never written, was deleted, the prefix string is wrong (e.g. missing/extra slash), or the job's output path from a previous stage is misconfigured.","commonSituations":"Downstream job pointing at an upstream output path that never ran or failed; typos in the s3 path or prefix (case sensitivity, missing '/'); environment-specific config pointing to a test bucket; deleted/ lifecycle-expired objects.","solutions":["Verify the exact s3://bucket/prefix with `aws s3 ls s3://bucket/prefix` using the same credentials — if empty, the path truly has no objects.","Fix the prefix string: check for missing trailing slash, wrong case, or wrong bucket/pipeline-option values.","Ensure the upstream job/stage that writes the data completed successfully before listing.","If an empty prefix is legitimate in your use case, catch `messages.S3ClientError` with code 404 and treat it as an empty result.","Check lifecycle policies that may have expired the objects."],"exampleFix":"// before\nlisting = client.list(messages.ListRequest(bucket='my-bucket', prefix='data/2026-09-12/'))\n// after\ntry:\n  listing = client.list(messages.ListRequest(bucket='my-bucket', prefix='data/2026-09-12/'))\nexcept messages.S3ClientError as e:\n  if e.code == 404:\n    listing = []  # no objects under prefix\n  else:\n    raise","handlingStrategy":"try-catch","validationCode":"import boto3\ndef prefix_has_objects(bucket, prefix, s3=None):\n  s3 = s3 or boto3.client('s3')\n  resp = s3.list_objects_v2(Bucket=bucket, Prefix=prefix, MaxKeys=1)\n  return resp.get('KeyCount', 0) > 0","typeGuard":"def is_missing_path(err):\n  return isinstance(err, messages.S3ClientError) and getattr(err, 'code', None) == 404","tryCatchPattern":"try:\n  listing = client.list(request)\nexcept messages.S3ClientError as e:\n  if getattr(e, 'code', None) == 404:\n    listing = []  # empty/missing prefix is acceptable\n  else:\n    raise","preventionTips":["Verify the s3://bucket/prefix exists with `aws s3 ls` before depending on it","Check upstream jobs completed and wrote the expected output path","Watch prefix string construction: trailing slash, case sensitivity, env-specific values","Confirm lifecycle rules are not expiring objects early","Treat empty prefixes explicitly in code instead of assuming existence"],"tags":["aws","s3","not-found","path","configuration"],"backgroundTag":"resource-not-found","analyzedSha":"12126d8942aaf848030c478b4c6a28c6af861c66","analyzedAt":"2026-09-13T01:50:10.254Z","contentChangedAt":"2026-09-13T01:50:10.254Z","schemaVersion":2},"datasetVersion":"2026-09-20T03:17:13.778Z"}