apache/beam · error · BeamIOError

Match operation failed

Error message

Match operation failed

What it means

FileSystems.match() evaluates each glob pattern in its own thread and collects exceptions per pattern; if any pattern failed it re-raises them all as a single BeamIOError('Match operation failed', exceptions) whose .eventual_detail / exception payload maps the failing pattern to its underlying error. It is an aggregate failure: one bad pattern fails the whole batch.

Solutions

1. Inspect the BeamIOError's exception detail dict to find which specific pattern failed and why.
2. Fix the failing pattern's path/scheme and ensure the matching filesystem dependency is installed (e.g. apache-beam[gcp]).
3. Match patterns in smaller batches or one at a time so one bad pattern doesn't fail the whole call.
4. Retry transient failures for cloud-backed filesystems with backoff.

Example fix

// before
try:
    matches = FileSystems.match(patterns)
except BeamIOError as e:
    pass  # which pattern failed?
// after
try:
    matches = FileSystems.match(patterns)
except BeamIOError as e:
    for pattern, cause in e.eventual_detail.items():
        logging.error('pattern %s failed: %s', pattern, cause)

from apache_beam.io.filesystems import FileSystems
# pre-check each pattern resolves to a known filesystem scheme
for p in patterns:
    assert '://' in p or p.startswith('/'), f'pattern lacks a scheme: {p}'

try:
    matches = FileSystems.match(patterns)
except BeamIOError as e:
    for pattern, cause in e.eventual_detail.items():
        logging.warning('match failed for %s: %s', pattern, cause)
    good = [p for p in patterns if p not in e.eventual_detail]
    matches = FileSystems.match(good) if good else []

When it happens

Trigger: Calling FileSystems.match(['gs://bucket/data/*.json', ...]) where any pattern resolves to a filesystem with no matching scheme, the pattern is malformed, or the underlying store raises (e.g. permission error listing a bucket).

Common situations: Typos in bucket names or paths that don't include a scheme the installed plugins support; missing GCS/HDFS extras so 'gs://' patterns cannot be resolved; transient cloud-store errors during bulk globbing.

Understand the failure class

Background: "File not found" and ENOENT errors: why libraries can't find a file that should exist — this error's family across 50 libraries.

Related errors

• Found more than one filesystem for path
• List operation failed
• Unable to get filesystem from specified path, please use…
• Unable to get the Filesystem
• A schema is required to write non-schema'd data.

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/4af924c2fc5ecc6d. Report an issue: GitHub.

Appendix: source

Thrown at sdks/python/apache_beam/io/filesystem.py:777

      metadata_list = []
      for file_metadata in self.match_files(file_metadatas, pattern):
        if limit is not None and len(metadata_list) >= limit:
          break
        metadata_list.append(file_metadata)

      return MatchResult(pattern, metadata_list)

    exceptions = {}
    result = []
    for pattern, limit in zip(patterns, limits):
      try:
        result.append(_match(pattern, limit))
      except Exception as e:  # pylint: disable=broad-except
        exceptions[pattern] = e

    if exceptions:
      raise BeamIOError("Match operation failed", exceptions)
    return result

  @abc.abstractmethod
  def create(
      self,
      path,
      mime_type='application/octet-stream',
      compression_type=CompressionTypes.AUTO) -> BinaryIO:
    """Returns a write channel for the given file path.

    Args:
      path: string path of the file object to be written to the system
      mime_type: MIME type to specify the type of content in the file object
      compression_type: Type of compression to be used for this object

    Returns: file handle with a close function for the user to use
    """
    raise NotImplementedError

View on GitHub (pinned to 12126d8942)