apache/beam · error · BeamIOError

size() operation failed

Error message

size() operation failed

What it means

Raised by S3FileSystem.size when S3IO.size(path) raises any exception, wrapped as BeamIOError keyed by path. Per the docs it is also the signal that the path does not exist, since size requires an existing object.

Solutions

1. Call fs.exists(path) (and handle file-not-found) before requesting size
2. Check the exact path/key spelling including prefix and extension
3. Validate credentials and region configuration
4. Catch BeamIOError and treat missing-path cases as expected in pipelines

Example fix

// before
size = fs.size(path)  # 404 for missing object
// after
try:
    size = fs.size(path)
except BeamIOError as e:
    if not fs.exists(path):
        raise FileNotFoundError(path) from e
    raise

if not fs.exists(path):
    raise FileNotFoundError(path)

try:
    size = fs.size(path)
except BeamIOError as e:
    if not fs.exists(path):
        raise FileNotFoundError(path) from e
    raise

When it happens

Trigger: Calling fs.size('s3://bucket/key') on a missing object (S3 raises 404/ClientError), invalid path format, or credential/network failure.

Common situations: Computing sizes of files deleted by compaction/GC before the size step; key typo or wrong prefix; unauthenticated environments.

Understand the failure class

Background: "File not found" and ENOENT errors: why libraries can't find a file that should exist — this error's family across 50 libraries.

Related errors

• Basepath %r must be S3 path.
• Checksum operation failed
• Delete operation failed
• ENOENT
• error listing object keys

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/d6d41b6d23d20d9d. Report an issue: GitHub.

Appendix: source

Thrown at sdks/python/apache_beam/io/aws/s3filesystem.py:250

      return s3io.S3IO(options=self._options).exists(path)
    except Exception as e:  # pylint: disable=broad-except
      raise BeamIOError("exists() operation failed", {path: e})

  def size(self, path):
    """Get size of path on the FileSystem.

    Args:
      path: string path in question.

    Returns: int size of path according to the FileSystem.

    Raises:
      ``BeamIOError``: if path doesn't exist.
    """
    try:
      return s3io.S3IO(options=self._options).size(path)
    except Exception as e:  # pylint: disable=broad-except
      raise BeamIOError("size() operation failed", {path: e})

  def last_updated(self, path):
    """Get UNIX Epoch time in seconds on the FileSystem.

    Args:
      path: string path of file.

    Returns: float UNIX Epoch time

    Raises:
      ``BeamIOError``: if path doesn't exist.
    """
    try:
      return s3io.S3IO(options=self._options).last_updated(path)
    except Exception as e:  # pylint: disable=broad-except
      raise BeamIOError("last_updated operation failed", {path: e})

  def checksum(self, path):

View on GitHub (pinned to 12126d8942)