apache/beam · error · BeamIOError

Match operation failed

Error message

Match operation failed

What it means

FileSystems.match() evaluates each glob pattern in its own thread and collects exceptions per pattern; if any pattern failed it re-raises them all as a single BeamIOError('Match operation failed', exceptions) whose .eventual_detail / exception payload maps the failing pattern to its underlying error. It is an aggregate failure: one bad pattern fails the whole batch.

Source

Thrown at sdks/python/apache_beam/io/filesystem.py:777

      metadata_list = []
      for file_metadata in self.match_files(file_metadatas, pattern):
        if limit is not None and len(metadata_list) >= limit:
          break
        metadata_list.append(file_metadata)

      return MatchResult(pattern, metadata_list)

    exceptions = {}
    result = []
    for pattern, limit in zip(patterns, limits):
      try:
        result.append(_match(pattern, limit))
      except Exception as e:  # pylint: disable=broad-except
        exceptions[pattern] = e

    if exceptions:
      raise BeamIOError("Match operation failed", exceptions)
    return result

  @abc.abstractmethod
  def create(
      self,
      path,
      mime_type='application/octet-stream',
      compression_type=CompressionTypes.AUTO) -> BinaryIO:
    """Returns a write channel for the given file path.

    Args:
      path: string path of the file object to be written to the system
      mime_type: MIME type to specify the type of content in the file object
      compression_type: Type of compression to be used for this object

    Returns: file handle with a close function for the user to use
    """
    raise NotImplementedError

View on GitHub (pinned to 12126d8942)

Solutions

  1. Inspect the BeamIOError's exception detail dict to find which specific pattern failed and why.
  2. Fix the failing pattern's path/scheme and ensure the matching filesystem dependency is installed (e.g. apache-beam[gcp]).
  3. Match patterns in smaller batches or one at a time so one bad pattern doesn't fail the whole call.
  4. Retry transient failures for cloud-backed filesystems with backoff.

Example fix

// before
try:
    matches = FileSystems.match(patterns)
except BeamIOError as e:
    pass  # which pattern failed?
// after
try:
    matches = FileSystems.match(patterns)
except BeamIOError as e:
    for pattern, cause in e.eventual_detail.items():
        logging.error('pattern %s failed: %s', pattern, cause)
Defensive patterns

Strategy: try-catch

Validate before calling

from apache_beam.io.filesystems import FileSystems
# pre-check each pattern resolves to a known filesystem scheme
for p in patterns:
    assert '://' in p or p.startswith('/'), f'pattern lacks a scheme: {p}'

Try / catch

try:
    matches = FileSystems.match(patterns)
except BeamIOError as e:
    for pattern, cause in e.eventual_detail.items():
        logging.warning('match failed for %s: %s', pattern, cause)
    good = [p for p in patterns if p not in e.eventual_detail]
    matches = FileSystems.match(good) if good else []

Prevention

When it happens

Trigger: Calling FileSystems.match(['gs://bucket/data/*.json', ...]) where any pattern resolves to a filesystem with no matching scheme, the pattern is malformed, or the underlying store raises (e.g. permission error listing a bucket).

Common situations: Typos in bucket names or paths that don't include a scheme the installed plugins support; missing GCS/HDFS extras so 'gs://' patterns cannot be resolved; transient cloud-store errors during bulk globbing.

Understand the failure class

Background: "File not found" and ENOENT errors: why libraries can't find a file that should exist — this error's family across 50 libraries.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/4af924c2fc5ecc6d. Report an issue: GitHub.