{"record":{"id":"1dd9be75876912cb","repo":"apache/beam","slug":"list-operation-failed","errorCode":null,"errorMessage":"List operation failed","messagePattern":"List operation failed","errorType":"exception","errorClass":"BeamIOError","httpStatus":null,"severity":"error","filePath":"sdks/python/apache_beam/io/aws/s3filesystem.py","lineNumber":136,"sourceCode":"\n    Listing is non-recursive, for filesystems that support directories.\n\n    Args:\n      dir_or_prefix: (string) A directory or location prefix (for filesystems\n        that don't have directories).\n\n    Returns:\n      Generator of ``FileMetadata`` objects.\n\n    Raises:\n      ``BeamIOError``: if listing fails, but not if no files were found.\n    \"\"\"\n    try:\n      for path, (size, updated) in s3io.S3IO(options=self._options).list_files(\n          dir_or_prefix, with_metadata=True):\n        yield FileMetadata(path, size, updated)\n    except Exception as e:  # pylint: disable=broad-except\n      raise BeamIOError(\"List operation failed\", {dir_or_prefix: e})\n\n  def _path_open(\n      self,\n      path,\n      mode,\n      mime_type='application/octet-stream',\n      compression_type=CompressionTypes.AUTO):\n    \"\"\"Helper functions to open a file in the provided mode.\n    \"\"\"\n    compression_type = FileSystem._get_compression_type(path, compression_type)\n    mime_type = CompressionTypes.mime_type(compression_type, mime_type)\n    raw_file = s3io.S3IO(options=self._options).open(\n        path, mode, mime_type=mime_type)\n    if compression_type == CompressionTypes.UNCOMPRESSED:\n      return raw_file\n    return CompressedFile(raw_file, compression_type=compression_type)\n\n  def create(","sourceCodeStart":118,"sourceCodeEnd":154,"githubUrl":"https://github.com/apache/beam/blob/12126d8942aaf848030c478b4c6a28c6af861c66/sdks/python/apache_beam/io/aws/s3filesystem.py#L118-L154","documentation":"S3FileSystem._list wraps s3io's list_files in a broad except and re-raises as BeamIOError('List operation failed', {dir_or_prefix: e}) when listing objects under an S3 dir/prefix fails for any reason — network errors, missing credentials, no such bucket/prefix, or permission problems.","triggerScenarios":"Calling S3FileSystem().match()/list on an s3:// prefix when the underlying S3IO.list_files raises: invalid AWS credentials, nonexistent bucket, access denied, throttling, or connectivity failure.","commonSituations":"Misconfigured AWS credentials/environment (no AWS_ACCESS_KEY_ID or instance profile); typo in bucket name; IAM policy lacking s3:ListBucket; transient network/timeout issues in CI without AWS access.","solutions":["Inspect exception_details in the BeamIOError (the per-path cause dict) to see the root error, then fix credentials, bucket name, or IAM permissions accordingly.","Verify AWS credentials are present and valid (aws sts get-caller-identity) and that the bucket/prefix exists.","Ensure the region/endpoint options passed via pipeline_options (--s3_endpoint, region) are correct for custom S3-compatible stores.","Add retry/backoff for transient S3 throttling before the filesystem layer; catch BeamIOError in user code and handle gracefully."],"exampleFix":"// before\ns3io.S3IO(options=self._options).list_files(dir_or_prefix, with_metadata=True)\n// after\ntry:\n  list(s3io.S3IO(options=self._options).list_files(dir_or_prefix, with_metadata=True))\nexcept Exception as e:\n  raise BeamIOError('List operation failed', {dir_or_prefix: e})  # inspect .exception_details for root cause","handlingStrategy":"try-catch","validationCode":"import os\ndef s3_env_ready() -> bool:\n    return bool(os.environ.get('AWS_ACCESS_KEY_ID') or os.environ.get('AWS_PROFILE') or os.environ.get('AWS_CONTAINER_CREDENTIALS_RELATIVE_URI'))","typeGuard":null,"tryCatchPattern":"try:\n    results = list(fs.match([f's3://{bucket}/{prefix}/*'])[0].metadata_list)\nexcept BeamIOError as e:\n    for path, cause in (e.exception_details or {}).items():\n        logging.error('S3 list failed for %s: %s', path, cause)\n    # check credentials/bucket/permissions, optionally retry with backoff","preventionTips":["Check AWS credentials and region before launching the pipeline","Verify bucket name and s3:ListBucket IAM permission","Inspect BeamIOError.exception_details for the root cause","Add retry/backoff for throttling in S3-compatible endpoints"],"tags":["s3","aws","beam-io-error","io","list"],"backgroundTag":"http-error-response","analyzedSha":"12126d8942aaf848030c478b4c6a28c6af861c66","analyzedAt":"2026-09-13T01:50:10.254Z","contentChangedAt":"2026-09-13T01:50:10.254Z","schemaVersion":2},"datasetVersion":"2026-09-14T16:17:12.679Z"}