apache/beam · error · BeamIOError

Metadata operation failed

Error message

Metadata operation failed

What it means

BlobStorageFileSystem.metadata() fetches size and last_updated via BlobStorageIO._status() and re-raises any exception as BeamIOError('Metadata operation failed', {path: e}). It fails if the blob does not exist or the Azure SDK call fails, and also if the returned status dict lacks the expected 'size'/'last_updated' keys.

Source

Thrown at sdks/python/apache_beam/io/azure/blobstoragefilesystem.py:299

  def metadata(self, path):
    """Fetch metadata fields of a file on the FileSystem.

    Args:
      path: string path of a file.

    Returns:
      :class:`~apache_beam.io.filesystem.FileMetadata`.

    Raises:
      ``BeamIOError``: if path isn't a file or doesn't exist.
    """
    try:
      file_metadata = self._blobstorageIO()._status(path)
      return FileMetadata(
          path, file_metadata['size'], file_metadata['last_updated'])
    except Exception as e:  # pylint: disable=broad-except
      raise BeamIOError("Metadata operation failed", {path: e})

  def delete(self, paths):
    """Deletes files or directories at the provided paths.
    Directories will be deleted recursively.

    Args:
      paths: list of paths that give the file objects to be deleted

    Raises:
      ``BeamIOError``: if any of the delete operations fail
    """
    results = self._blobstorageIO().delete_paths(paths)
    # Retrieve exceptions.
    exceptions = {
        path: error
        for (path, error) in results.items() if error is not None
    }

View on GitHub (pinned to 12126d8942)

Solutions

1. Check fs.exists(path) before fetching metadata
2. Inspect BeamIOError.exception_details[path] for the underlying KeyError/Azure exception
3. Pin compatible azure-storage-blob versions so _status returns the expected fields
4. Verify credentials and container read permissions

Example fix

// before
meta = fs.metadata(path)
// after
try:
    meta = fs.metadata(path)
except BeamIOError as e:
    logging.error(f"metadata failed for {path}: {e.exception_details}")
    meta = None

if not fs.exists(path):
    raise FileNotFoundError(f"no such blob: {path}")

try:
    meta = fs.metadata(path)
except BeamIOError as e:
    logging.error("metadata failed for %s: %s", path, e.exception_details)
    meta = None  # handle missing metadata explicitly downstream

When it happens

Trigger: Calling fs.metadata(path) on a missing blob, with bad credentials, or when _status returns a dict without 'size' or 'last_updated' (KeyError is also caught and wrapped).

Common situations: Listing-derived paths that were deleted before metadata fetch; service changes/SDK version drift making _status fields absent; auth misconfiguration; using it on directory-style prefixes.

Understand the failure class

Background: "failed to read file", EACCES, ENOENT and "could not read <path>" errors: when a program can't read a file from disk — this error's family across 49 libraries.

Related errors

• Exists operation failed
• Size operation failed
• Last updated operation failed
• Checksum operation failed
• Unable to rename unequal number of sources and destinations.

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/fa60165421da44d8. Report an issue: GitHub.