What causes "Size operation failed" in apache/beam?

Call fs.exists(path) first or check the wrapped exception to distinguish missing-file from other failures

apache/beam · error · BeamIOError

Size operation failed

Error message

Size operation failed

What it means

BlobStorageFileSystem.size() returns the byte size of a file via BlobStorageIO.size() and re-raises any exception as BeamIOError('Size operation failed', {path: e}). Per the docstring it is also the expected error if the path does not exist, since the underlying call raises when the blob is missing.

Source

Thrown at sdks/python/apache_beam/io/azure/blobstoragefilesystem.py:247

      return self._blobstorageIO().exists(path)
    except Exception as e:  # pylint: disable=broad-except
      raise BeamIOError("Exists operation failed", {path: e})

  def size(self, path):
    """Get size in bytes of a file on the FileSystem.

    Args:
      path: string filepath of file.

    Returns: int size of file according to the FileSystem.

    Raises:
      ``BeamIOError``: if path doesn't exist.
    """
    try:
      return self._blobstorageIO().size(path)
    except Exception as e:  # pylint: disable=broad-except
      raise BeamIOError("Size operation failed", {path: e})

  def last_updated(self, path):
    """Get UNIX Epoch time in seconds on the FileSystem.

    Args:
      path: string path of file.

    Returns: float UNIX Epoch time

    Raises:
      ``BeamIOError``: if path doesn't exist.
    """
    try:
      return self._blobstorageIO().last_updated(path)
    except Exception as e:  # pylint: disable=broad-except
      raise BeamIOError("Last updated operation failed", {path: e})

  def checksum(self, path):

View on GitHub (pinned to 12126d8942)

Solutions

Call fs.exists(path) first or check the wrapped exception to distinguish missing-file from other failures
Verify the path with blobstorageio.parse_azfs_path to catch malformed paths early
Confirm credentials have read access to the container
Retry on transient Azure errors (throttling/network) with backoff

Example fix

// before
size = fs.size(path)
// after
try:
    size = fs.size(path)
except BeamIOError as e:
    if not fs.exists(path):
        raise FileNotFoundError(path) from e
    raise

Defensive patterns

Strategy: try-catch

Validate before calling

if not fs.exists(path):
    raise FileNotFoundError(f"no such blob: {path}")

Try / catch

try:
    size = fs.size(path)
except BeamIOError as e:
    details = e.exception_details.get(path)
    logging.error("size failed for %s: %s", path, details)
    raise

Prevention

Call exists() before size() when the file may legitimately be absent
Ensure read permissions on the container
Retry transient Azure errors (throttling, timeouts) with exponential backoff
Verify path format with parse_azfs_path before calls

When it happens

Trigger: Calling fs.size(path) on an azfs:// path for a blob that does not exist, or when the underlying Azure SDK call fails (auth, network, permissions).

Common situations: Stat'ing a file produced by an earlier pipeline stage that has not been written yet; path built with the wrong container or prefix; credentials lacking read permission on the container; transient Azure errors.

Understand the failure class

Background: "failed to read file", EACCES, ENOENT and "could not read <path>" errors: when a program can't read a file from disk — this error's family across 49 libraries.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/0ce4b853e907ffe3. Report an issue: GitHub.