{"record":{"id":"4af924c2fc5ecc6d","repo":"apache/beam","slug":"match-operation-failed","errorCode":null,"errorMessage":"Match operation failed","messagePattern":"Match operation failed","errorType":"exception","errorClass":"BeamIOError","httpStatus":null,"severity":"error","filePath":"sdks/python/apache_beam/io/filesystem.py","lineNumber":777,"sourceCode":"\n      metadata_list = []\n      for file_metadata in self.match_files(file_metadatas, pattern):\n        if limit is not None and len(metadata_list) >= limit:\n          break\n        metadata_list.append(file_metadata)\n\n      return MatchResult(pattern, metadata_list)\n\n    exceptions = {}\n    result = []\n    for pattern, limit in zip(patterns, limits):\n      try:\n        result.append(_match(pattern, limit))\n      except Exception as e:  # pylint: disable=broad-except\n        exceptions[pattern] = e\n\n    if exceptions:\n      raise BeamIOError(\"Match operation failed\", exceptions)\n    return result\n\n  @abc.abstractmethod\n  def create(\n      self,\n      path,\n      mime_type='application/octet-stream',\n      compression_type=CompressionTypes.AUTO) -> BinaryIO:\n    \"\"\"Returns a write channel for the given file path.\n\n    Args:\n      path: string path of the file object to be written to the system\n      mime_type: MIME type to specify the type of content in the file object\n      compression_type: Type of compression to be used for this object\n\n    Returns: file handle with a close function for the user to use\n    \"\"\"\n    raise NotImplementedError","sourceCodeStart":759,"sourceCodeEnd":795,"githubUrl":"https://github.com/apache/beam/blob/12126d8942aaf848030c478b4c6a28c6af861c66/sdks/python/apache_beam/io/filesystem.py#L759-L795","documentation":"FileSystems.match() evaluates each glob pattern in its own thread and collects exceptions per pattern; if any pattern failed it re-raises them all as a single BeamIOError('Match operation failed', exceptions) whose .eventual_detail / exception payload maps the failing pattern to its underlying error. It is an aggregate failure: one bad pattern fails the whole batch.","triggerScenarios":"Calling FileSystems.match(['gs://bucket/data/*.json', ...]) where any pattern resolves to a filesystem with no matching scheme, the pattern is malformed, or the underlying store raises (e.g. permission error listing a bucket).","commonSituations":"Typos in bucket names or paths that don't include a scheme the installed plugins support; missing GCS/HDFS extras so 'gs://' patterns cannot be resolved; transient cloud-store errors during bulk globbing.","solutions":["Inspect the BeamIOError's exception detail dict to find which specific pattern failed and why.","Fix the failing pattern's path/scheme and ensure the matching filesystem dependency is installed (e.g. apache-beam[gcp]).","Match patterns in smaller batches or one at a time so one bad pattern doesn't fail the whole call.","Retry transient failures for cloud-backed filesystems with backoff."],"exampleFix":"// before\ntry:\n    matches = FileSystems.match(patterns)\nexcept BeamIOError as e:\n    pass  # which pattern failed?\n// after\ntry:\n    matches = FileSystems.match(patterns)\nexcept BeamIOError as e:\n    for pattern, cause in e.eventual_detail.items():\n        logging.error('pattern %s failed: %s', pattern, cause)","handlingStrategy":"try-catch","validationCode":"from apache_beam.io.filesystems import FileSystems\n# pre-check each pattern resolves to a known filesystem scheme\nfor p in patterns:\n    assert '://' in p or p.startswith('/'), f'pattern lacks a scheme: {p}'","typeGuard":null,"tryCatchPattern":"try:\n    matches = FileSystems.match(patterns)\nexcept BeamIOError as e:\n    for pattern, cause in e.eventual_detail.items():\n        logging.warning('match failed for %s: %s', pattern, cause)\n    good = [p for p in patterns if p not in e.eventual_detail]\n    matches = FileSystems.match(good) if good else []","preventionTips":["Test each new glob pattern interactively (single-pattern match) before batching.","Keep patterns scheme-qualified and ensure the matching extra ([gcp], [aws]) is installed.","Log e.eventual_detail to identify the offending pattern instead of failing the whole batch.","Use MatchActionResult/limit options deliberately and retry transient cloud errors."],"tags":["python","io","glob","filesystem","batch-failure"],"backgroundTag":"file-not-found","analyzedSha":"12126d8942aaf848030c478b4c6a28c6af861c66","analyzedAt":"2026-09-13T01:50:10.254Z","contentChangedAt":"2026-09-13T01:50:10.254Z","schemaVersion":2},"datasetVersion":"2026-09-14T16:17:12.679Z"}