apache/beam · error · BeamIOError

List operation failed

Error message

List operation failed

What it means

LocalFileSystem._list walks the directory tree matching a glob/directory prefix and yields FileMetadata entries; unexpected errors during the walk are collected and re-raised as BeamIOError('List operation failed', {dir_or_prefix: original_error}). The exception details dict maps the requested path to the underlying error. Disappearing files are tolerated (OSError on stat is skipped), so this error reflects a failure of the listing itself, not transient file removal.

Source

Thrown at sdks/python/apache_beam/io/localfilesystem.py:129

      ``BeamIOError``: if listing fails, but not if no files were found.
    """
    if not self.exists(dir_or_prefix):
      return

    def list_files(root):
      for dirpath, _, files in os.walk(root):
        for filename in files:
          yield self.join(dirpath, filename)

    try:
      for f in list_files(dir_or_prefix):
        try:
          yield FileMetadata(f, os.path.getsize(f), os.path.getmtime(f))
        except OSError:
          # Files may disappear, such as when listing /tmp.
          pass
    except Exception as e:  # pylint: disable=broad-except
      raise BeamIOError("List operation failed", {dir_or_prefix: e})

  def _path_open(
      self,
      path,
      mode,
      mime_type='application/octet-stream',
      compression_type=CompressionTypes.AUTO):
    """Helper functions to open a file in the provided mode.
    """
    compression_type = FileSystem._get_compression_type(path, compression_type)
    raw_file = io.open(path, mode)
    if compression_type == CompressionTypes.UNCOMPRESSED:
      return raw_file
    else:
      return CompressedFile(raw_file, compression_type=compression_type)

  def create(
      self,

View on GitHub (pinned to 12126d8942)

Solutions

1. Read BeamIOError.exception_details[dir_or_prefix] to find the underlying cause (permission, missing path, etc.).
2. Check read/execute permissions on every directory the glob traverses; narrow the pattern to accessible subtrees.
3. Verify the glob/prefix is syntactically valid and rooted at an existing directory.
4. Catch BeamIOError around FileSystems.match and fall back to a plain os.path.exists check if you only need existence.
5. Retry once if the filesystem is a network mount; transient NFS/EFS errors often clear.

Example fix

// before
result = FileSystems.match(['file:///data/2026/**/*.json'])
// after
from apache_beam.io.filesystem import BeamIOError
try:
    result = FileSystems.match(['file:///data/2026/**/*.json'])
except BeamIOError as e:
    print(e.exception_details)  # inspect root cause per path

import os, glob
assert os.path.isdir(os.path.dirname(base_pattern) or '.'), "base dir missing"

from apache_beam.io.filesystem import BeamIOError
try:
    results = FileSystems.match([pattern])
except BeamIOError as e:
    for path, err in e.exception_details.items():
        logging.warning('list failed for %s: %s', path, err)

When it happens

Trigger: Calling LocalFileSystem.match/._list (directly or via beam.io.filesystems.FileSystems.match) with a directory or glob whose traversal fails: permission denied on a subdirectory, path is not a directory/glob that resolves oddly, or an OS error during os.walk/scandir.

Common situations: Listing /home/otheruser/** without read permission; glob patterns with bad syntax that reach the OS layer; NFS/network mounts flaking during traversal; pointing match() at a nonexistent prefix with strict expectations.

Understand the failure class

Background: "failed to read file", EACCES, ENOENT and "could not read <path>" errors: when a program can't read a file from disk — this error's family across 49 libraries.

Related errors

• Encountered a type that is not currently supported by RowCod
• Match operation failed
• err (re-raised OSError from os.makedirs)
• err (re-raised OSError during copy)
• Copy operation failed

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/7f289cb6c4ba408f. Report an issue: GitHub.