{"record":{"id":"7f289cb6c4ba408f","repo":"apache/beam","slug":"list-operation-failed-localfilesystem","errorCode":null,"errorMessage":"List operation failed","messagePattern":"List operation failed","errorType":"exception","errorClass":"BeamIOError","httpStatus":null,"severity":"error","filePath":"sdks/python/apache_beam/io/localfilesystem.py","lineNumber":129,"sourceCode":"      ``BeamIOError``: if listing fails, but not if no files were found.\n    \"\"\"\n    if not self.exists(dir_or_prefix):\n      return\n\n    def list_files(root):\n      for dirpath, _, files in os.walk(root):\n        for filename in files:\n          yield self.join(dirpath, filename)\n\n    try:\n      for f in list_files(dir_or_prefix):\n        try:\n          yield FileMetadata(f, os.path.getsize(f), os.path.getmtime(f))\n        except OSError:\n          # Files may disappear, such as when listing /tmp.\n          pass\n    except Exception as e:  # pylint: disable=broad-except\n      raise BeamIOError(\"List operation failed\", {dir_or_prefix: e})\n\n  def _path_open(\n      self,\n      path,\n      mode,\n      mime_type='application/octet-stream',\n      compression_type=CompressionTypes.AUTO):\n    \"\"\"Helper functions to open a file in the provided mode.\n    \"\"\"\n    compression_type = FileSystem._get_compression_type(path, compression_type)\n    raw_file = io.open(path, mode)\n    if compression_type == CompressionTypes.UNCOMPRESSED:\n      return raw_file\n    else:\n      return CompressedFile(raw_file, compression_type=compression_type)\n\n  def create(\n      self,","sourceCodeStart":111,"sourceCodeEnd":147,"githubUrl":"https://github.com/apache/beam/blob/12126d8942aaf848030c478b4c6a28c6af861c66/sdks/python/apache_beam/io/localfilesystem.py#L111-L147","documentation":"LocalFileSystem._list walks the directory tree matching a glob/directory prefix and yields FileMetadata entries; unexpected errors during the walk are collected and re-raised as BeamIOError('List operation failed', {dir_or_prefix: original_error}). The exception details dict maps the requested path to the underlying error. Disappearing files are tolerated (OSError on stat is skipped), so this error reflects a failure of the listing itself, not transient file removal.","triggerScenarios":"Calling LocalFileSystem.match/._list (directly or via beam.io.filesystems.FileSystems.match) with a directory or glob whose traversal fails: permission denied on a subdirectory, path is not a directory/glob that resolves oddly, or an OS error during os.walk/scandir.","commonSituations":"Listing /home/otheruser/** without read permission; glob patterns with bad syntax that reach the OS layer; NFS/network mounts flaking during traversal; pointing match() at a nonexistent prefix with strict expectations.","solutions":["Read BeamIOError.exception_details[dir_or_prefix] to find the underlying cause (permission, missing path, etc.).","Check read/execute permissions on every directory the glob traverses; narrow the pattern to accessible subtrees.","Verify the glob/prefix is syntactically valid and rooted at an existing directory.","Catch BeamIOError around FileSystems.match and fall back to a plain os.path.exists check if you only need existence.","Retry once if the filesystem is a network mount; transient NFS/EFS errors often clear."],"exampleFix":"// before\nresult = FileSystems.match(['file:///data/2026/**/*.json'])\n// after\nfrom apache_beam.io.filesystem import BeamIOError\ntry:\n    result = FileSystems.match(['file:///data/2026/**/*.json'])\nexcept BeamIOError as e:\n    print(e.exception_details)  # inspect root cause per path","handlingStrategy":"try-catch","validationCode":"import os, glob\nassert os.path.isdir(os.path.dirname(base_pattern) or '.'), \"base dir missing\"","typeGuard":null,"tryCatchPattern":"from apache_beam.io.filesystem import BeamIOError\ntry:\n    results = FileSystems.match([pattern])\nexcept BeamIOError as e:\n    for path, err in e.exception_details.items():\n        logging.warning('list failed for %s: %s', path, err)","preventionTips":["Check read/execute permissions on all directories a glob traverses","Validate glob patterns against a known-good base directory","Retry on network-mounted filesystems (NFS/EFS) where traversal is transiently flaky"],"tags":["python","apache-beam","filesystem","glob","listing"],"backgroundTag":"file-read-failed","analyzedSha":"12126d8942aaf848030c478b4c6a28c6af861c66","analyzedAt":"2026-09-13T01:50:10.254Z","contentChangedAt":"2026-09-13T01:50:10.254Z","schemaVersion":2},"datasetVersion":"2026-09-14T11:17:12.474Z"}