apache/beam · error · BeamIOError

List operation failed

Error message

List operation failed

What it means

S3FileSystem._list wraps s3io's list_files in a broad except and re-raises as BeamIOError('List operation failed', {dir_or_prefix: e}) when listing objects under an S3 dir/prefix fails for any reason — network errors, missing credentials, no such bucket/prefix, or permission problems.

Source

Thrown at sdks/python/apache_beam/io/aws/s3filesystem.py:136

    Listing is non-recursive, for filesystems that support directories.

    Args:
      dir_or_prefix: (string) A directory or location prefix (for filesystems
        that don't have directories).

    Returns:
      Generator of ``FileMetadata`` objects.

    Raises:
      ``BeamIOError``: if listing fails, but not if no files were found.
    """
    try:
      for path, (size, updated) in s3io.S3IO(options=self._options).list_files(
          dir_or_prefix, with_metadata=True):
        yield FileMetadata(path, size, updated)
    except Exception as e:  # pylint: disable=broad-except
      raise BeamIOError("List operation failed", {dir_or_prefix: e})

  def _path_open(
      self,
      path,
      mode,
      mime_type='application/octet-stream',
      compression_type=CompressionTypes.AUTO):
    """Helper functions to open a file in the provided mode.
    """
    compression_type = FileSystem._get_compression_type(path, compression_type)
    mime_type = CompressionTypes.mime_type(compression_type, mime_type)
    raw_file = s3io.S3IO(options=self._options).open(
        path, mode, mime_type=mime_type)
    if compression_type == CompressionTypes.UNCOMPRESSED:
      return raw_file
    return CompressedFile(raw_file, compression_type=compression_type)

  def create(

View on GitHub (pinned to 12126d8942)

Solutions

1. Inspect exception_details in the BeamIOError (the per-path cause dict) to see the root error, then fix credentials, bucket name, or IAM permissions accordingly.
2. Verify AWS credentials are present and valid (aws sts get-caller-identity) and that the bucket/prefix exists.
3. Ensure the region/endpoint options passed via pipeline_options (--s3_endpoint, region) are correct for custom S3-compatible stores.
4. Add retry/backoff for transient S3 throttling before the filesystem layer; catch BeamIOError in user code and handle gracefully.

Example fix

// before
s3io.S3IO(options=self._options).list_files(dir_or_prefix, with_metadata=True)
// after
try:
  list(s3io.S3IO(options=self._options).list_files(dir_or_prefix, with_metadata=True))
except Exception as e:
  raise BeamIOError('List operation failed', {dir_or_prefix: e})  # inspect .exception_details for root cause

import os
def s3_env_ready() -> bool:
    return bool(os.environ.get('AWS_ACCESS_KEY_ID') or os.environ.get('AWS_PROFILE') or os.environ.get('AWS_CONTAINER_CREDENTIALS_RELATIVE_URI'))

try:
    results = list(fs.match([f's3://{bucket}/{prefix}/*'])[0].metadata_list)
except BeamIOError as e:
    for path, cause in (e.exception_details or {}).items():
        logging.error('S3 list failed for %s: %s', path, cause)
    # check credentials/bucket/permissions, optionally retry with backoff

When it happens

Trigger: Calling S3FileSystem().match()/list on an s3:// prefix when the underlying S3IO.list_files raises: invalid AWS credentials, nonexistent bucket, access denied, throttling, or connectivity failure.

Common situations: Misconfigured AWS credentials/environment (no AWS_ACCESS_KEY_ID or instance profile); typo in bucket name; IAM policy lacking s3:ListBucket; transient network/timeout issues in CI without AWS access.

Understand the failure class

Background: "API error: {status}" and "HTTP 401/403/404/429/5xx" errors: non-2xx HTTP responses explained — this error's family across 27 libraries.

Related errors

• Basepath %r must be S3 path.
• Path %r must be S3 path.
• Invalid path: %s
• Unable to copy unequal number of sources and destinations
• Rename operation failed

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/1dd9be75876912cb. Report an issue: GitHub.