{"record":{"id":"0994a775eb50a9db","repo":"apache/beam","slug":"metadata-operation-failed","errorCode":null,"errorMessage":"Metadata operation failed","messagePattern":"Metadata operation failed","errorType":"exception","errorClass":"BeamIOError","httpStatus":null,"severity":"error","filePath":"sdks/python/apache_beam/io/aws/s3filesystem.py","lineNumber":302,"sourceCode":"\n  def metadata(self, path):\n    \"\"\"Fetch metadata fields of a file on the FileSystem.\n\n    Args:\n      path: string path of a file.\n\n    Returns:\n      :class:`~apache_beam.io.filesystem.FileMetadata`.\n\n    Raises:\n      ``BeamIOError``: if path isn't a file or doesn't exist.\n    \"\"\"\n    try:\n      file_metadata = s3io.S3IO(options=self._options)._status(path)\n      return FileMetadata(\n          path, file_metadata['size'], file_metadata['last_updated'])\n    except Exception as e:  # pylint: disable=broad-except\n      raise BeamIOError(\"Metadata operation failed\", {path: e})\n\n  def delete(self, paths):\n    \"\"\"Deletes files or directories at the provided paths.\n    Directories will be deleted recursively.\n\n    Args:\n      paths: list of paths that give the file objects to be deleted\n    \"\"\"\n    results = s3io.S3IO(options=self._options).delete_paths(paths)\n    exceptions = {\n        path: error\n        for (path, error) in results.items() if error is not None\n    }\n    if exceptions:\n      raise BeamIOError(\"Delete operation failed\", exceptions)\n\n  def report_lineage(self, path, lineage):\n    try:","sourceCodeStart":284,"sourceCodeEnd":320,"githubUrl":"https://github.com/apache/beam/blob/12126d8942aaf848030c478b4c6a28c6af861c66/sdks/python/apache_beam/io/aws/s3filesystem.py#L284-L320","documentation":"Raised by S3FileSystem.metadata when the underlying S3IO._status(path) call raises; wrapped as BeamIOError keyed by path. This fetches size and last_updated into a FileMetadata, so it fails whenever the object status cannot be retrieved.","triggerScenarios":"Calling fs.metadata('s3://bucket/key') on a missing object, invalid path format, or with broken credentials/network causing the status request to raise.","commonSituations":"Listing-based pipelines assuming a file still exists; typo'd keys; environments without AWS credentials configured; IAM policies lacking GetObject.","solutions":["Call fs.exists(path) first and handle missing paths explicitly","Inspect BeamIOError.exception_details for the root exception","Fix AWS credentials/region/IAM configuration if the failure is systemic","Catch BeamIOError around metadata calls in incremental pipelines"],"exampleFix":"// before\nmd = fs.metadata(path)\n// after\ntry:\n    md = fs.metadata(path)\nexcept BeamIOError:\n    md = None  # object vanished or unreadable","handlingStrategy":"try-catch","validationCode":"if not fs.exists(path):\n    return None","typeGuard":null,"tryCatchPattern":"try:\n    md = fs.metadata(path)\nexcept BeamIOError as e:\n    log.warning('metadata unavailable for %s: %s', path, e.exception_details)\n    md = None","preventionTips":["Expect objects to disappear between listing and metadata fetch","Keep AWS credentials and IAM GetObject permissions configured","Validate s3:// path format up front"],"tags":["s3","io","aws","file-metadata"],"backgroundTag":"file-not-found","analyzedSha":"12126d8942aaf848030c478b4c6a28c6af861c66","analyzedAt":"2026-09-13T01:50:10.254Z","contentChangedAt":"2026-09-13T01:50:10.254Z","schemaVersion":2},"datasetVersion":"2026-09-20T03:17:13.778Z"}