apache/beam · error · BeamIOError

Metadata operation failed

Error message

Metadata operation failed

What it means

Raised by S3FileSystem.metadata when the underlying S3IO._status(path) call raises; wrapped as BeamIOError keyed by path. This fetches size and last_updated into a FileMetadata, so it fails whenever the object status cannot be retrieved.

Solutions

1. Call fs.exists(path) first and handle missing paths explicitly
2. Inspect BeamIOError.exception_details for the root exception
3. Fix AWS credentials/region/IAM configuration if the failure is systemic
4. Catch BeamIOError around metadata calls in incremental pipelines

Example fix

// before
md = fs.metadata(path)
// after
try:
    md = fs.metadata(path)
except BeamIOError:
    md = None  # object vanished or unreadable

if not fs.exists(path):
    return None

try:
    md = fs.metadata(path)
except BeamIOError as e:
    log.warning('metadata unavailable for %s: %s', path, e.exception_details)
    md = None

When it happens

Trigger: Calling fs.metadata('s3://bucket/key') on a missing object, invalid path format, or with broken credentials/network causing the status request to raise.

Common situations: Listing-based pipelines assuming a file still exists; typo'd keys; environments without AWS credentials configured; IAM policies lacking GetObject.

Understand the failure class

Background: "File not found" and ENOENT errors: why libraries can't find a file that should exist — this error's family across 50 libraries.

Related errors

• last_updated operation failed
• Basepath %r must be S3 path.
• Checksum operation failed
• Delete operation failed
• ENOENT

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/0994a775eb50a9db. Report an issue: GitHub.

Appendix: source

Thrown at sdks/python/apache_beam/io/aws/s3filesystem.py:302

  def metadata(self, path):
    """Fetch metadata fields of a file on the FileSystem.

    Args:
      path: string path of a file.

    Returns:
      :class:`~apache_beam.io.filesystem.FileMetadata`.

    Raises:
      ``BeamIOError``: if path isn't a file or doesn't exist.
    """
    try:
      file_metadata = s3io.S3IO(options=self._options)._status(path)
      return FileMetadata(
          path, file_metadata['size'], file_metadata['last_updated'])
    except Exception as e:  # pylint: disable=broad-except
      raise BeamIOError("Metadata operation failed", {path: e})

  def delete(self, paths):
    """Deletes files or directories at the provided paths.
    Directories will be deleted recursively.

    Args:
      paths: list of paths that give the file objects to be deleted
    """
    results = s3io.S3IO(options=self._options).delete_paths(paths)
    exceptions = {
        path: error
        for (path, error) in results.items() if error is not None
    }
    if exceptions:
      raise BeamIOError("Delete operation failed", exceptions)

  def report_lineage(self, path, lineage):
    try:

View on GitHub (pinned to 12126d8942)