apache/beam · error · BeamIOError

exists() operation failed

Error message

exists() operation failed

What it means

Raised by S3FileSystem.exists when the underlying S3IO.exists(path) call raises any exception; it is wrapped as BeamIOError with the path as key. This indicates the existence probe itself failed (not that the path is absent).

Source

Thrown at sdks/python/apache_beam/io/aws/s3filesystem.py:234

    src_dest_pairs = list(zip(source_file_names, destination_file_names))
    results = s3io.S3IO(options=self._options).rename_files(src_dest_pairs)
    exceptions = {(src, dest): error
                  for (src, dest, error) in results if error is not None}
    if exceptions:
      raise BeamIOError("Rename operation failed", exceptions)

  def exists(self, path):
    """Check if the provided path exists on the FileSystem.

    Args:
      path: string path that needs to be checked.

    Returns: boolean flag indicating if path exists
    """
    try:
      return s3io.S3IO(options=self._options).exists(path)
    except Exception as e:  # pylint: disable=broad-except
      raise BeamIOError("exists() operation failed", {path: e})

  def size(self, path):
    """Get size of path on the FileSystem.

    Args:
      path: string path in question.

    Returns: int size of path according to the FileSystem.

    Raises:
      ``BeamIOError``: if path doesn't exist.
    """
    try:
      return s3io.S3IO(options=self._options).size(path)
    except Exception as e:  # pylint: disable=broad-except
      raise BeamIOError("size() operation failed", {path: e})

  def last_updated(self, path):

View on GitHub (pinned to 12126d8942)

Solutions

1. Inspect the wrapped exception details for the path to see the root cause
2. Validate the path matches s3://<bucket>/<object> before calling exists
3. Check AWS credentials/region configuration (env vars, ~/.aws, instance profile)
4. Retry on transient network errors

Example fix

// before
if fs.exists(user_path): ...
// after
try:
    if fs.exists(user_path): ...
except BeamIOError as e:
    logging.error('exists check failed: %s', e.exception_details)

import re
if not re.match(r'^s3://[^/]+/.+', path):
    raise ValueError(f'invalid s3 path: {path}')

try:
    exists = fs.exists(path)
except BeamIOError as e:
    log.error('exists failed for %s: %s', path, e.exception_details)
    exists = False

When it happens

Trigger: Calling fs.exists('s3://bucket/key') when boto3 client misconfiguration, invalid credentials, network failure, or an invalid path format raises inside S3IO.exists.

Common situations: Expired or missing AWS credentials; wrong region/endpoint config; network outage in CI; malformed path string passed instead of a proper s3:// URL.

Understand the failure class

Background: "API request failed": what wrapped HTTP errors from external APIs mean and how to find the real cause — this error's family across 29 libraries.

Related errors

• Basepath %r must be S3 path.
• Path %r must be S3 path.
• Invalid path: %s
• List operation failed
• Rename operation failed

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/d72a4bf6bfa1b692. Report an issue: GitHub.