apache/beam · error · BeamIOError

Metadata operation failed

Error message

Metadata operation failed

What it means

GcsFileSystem.metadata raises BeamIOError('Metadata operation failed') when fetching file metadata via gcsio._status(path) throws. The original exception is preserved in exception_details keyed by the path.

Solutions

1. Validate the path format and existence (parse_gcs_path / exists) before calling metadata.
2. Inspect exception_details for the underlying NotFound/permission error.
3. Fix the gs:// path or point to an existing object.
4. Catch BeamIOError to handle missing objects gracefully.

Example fix

// before
meta = fs.metadata(path)
// after
try:
  meta = fs.metadata(path)
except BeamIOError:
  meta = None  # object missing or unreadable

if not fs.exists(path):
    raise FileNotFoundError(path)

def valid_gcs_path(p):
    return isinstance(p, str) and p.startswith('gs://') and p.count('/') >= 3

try:
    meta = fs.metadata(path)
except BeamIOError as e:
    logging.warning('metadata failed: %s', e.exception_details.get(path))
    meta = None

When it happens

Trigger: Calling GcsFileSystem.metadata(path) where the path is malformed, the object does not exist, or the GCS API call fails (permissions, network).

Common situations: Querying size/updated of an object that was never written or already deleted; wrong bucket name; path missing gs:// scheme.

Understand the failure class

Background: "File not found" and ENOENT errors: why libraries can't find a file that should exist — this error's family across 50 libraries.

Related errors

• Checksum operation failed
• Copy operation failed
• Delete operation failed
• dynamic: wrapped StorageException via…
• empty chunk

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/74ff934316c49c7a. Report an issue: GitHub.

Appendix: source

Thrown at sdks/python/apache_beam/io/gcp/gcsfilesystem.py:343

      raise BeamIOError("Checksum operation failed", {path: e})

  def metadata(self, path):
    """Fetch metadata fields of a file on the FileSystem.

    Args:
      path: string path of a file.

    Returns:
      :class:`~apache_beam.io.filesystem.FileMetadata`.

    Raises:
      ``BeamIOError``: if path isn't a file or doesn't exist.
    """
    try:
      file_metadata = self._gcsIO()._status(path)
      return FileMetadata(path, file_metadata['size'], file_metadata['updated'])
    except Exception as e:  # pylint: disable=broad-except
      raise BeamIOError("Metadata operation failed", {path: e})

  def delete(self, paths):
    """Deletes files or directories at the provided paths.
    Directories will be deleted recursively.

    Args:
      paths: list of paths that give the file objects to be deleted
    """

    exceptions = {}

    for path in paths:
      if path.endswith('/'):
        self._gcsIO().delete(path, recursive=True)
        continue
      else:
        path_to_use = path
      match_result = self.match([path_to_use])[0]

View on GitHub (pinned to 12126d8942)