apache/beam · error · BeamIOError

Directories are not allowed in ReadMatches transform.Found

Error message

Directories are not allowed in ReadMatches transform.Found %s.

What it means

ReadMatches.process raises BeamIOError when a matched metadata path is a directory (ends with '/' or backslash) and skip_directories is False. ReadMatches can only turn file blobs into ReadableFile objects, so directories cannot be read. Matching patterns like 'dir/*' that match directories directly trigger this.

Solutions

Enable skip_directories=True (default) on ReadMatches so directory entries are silently skipped
Refine the glob to exclude directories or match only the file extension (e.g. '*.json')
Pre-filter matches with a filter transform checking not path.endswith('/')

Example fix

// before
p | MatchAll('gs://b/data/*') | ReadMatches(skip_directories=False)
// after
p | MatchAll('gs://b/data/*') | ReadMatches(skip_directories=True)  # default; skips dirs

Defensive patterns

Strategy: validation

Validate before calling

bad = [m.path for m in matches if m.path.endswith('/') or m.path.endswith('\\')]
if bad:
    logging.warning('Directory entries will fail ReadMatches: %s', bad)

Try / catch

from apache_beam.io.fileio import ReadMatches
try:
    files = matches | ReadMatches()
except Exception as e:
    if 'Directories are not allowed' in str(e):
        matches = matches | beam.Filter(lambda m: not (m.path.endswith('/') or m.path.endswith('\\')))
        files = matches | ReadMatches()
    else:
        raise

Prevention

Keep skip_directories=True (the default)
Glob on file extensions ('data/*.json') rather than bare prefixes to exclude subdirectories
Filter out directory-looking paths before ReadMatches when the backend lists placeholders

When it happens

Trigger: Using a match pattern that includes directories (e.g. 'gs://b/data/*' with subdirectories present) while skip_directories=False; explicitly configuring skip_directories=False and feeding directory entries.

Common situations: Nested directory trees under a globbed prefix; cloud-storage listings that include directory placeholders; users setting skip_directories=False expecting nested traversal instead of an error.

Understand the failure class

Background: "is not a compatible type" / "cannot merge" errors: when a value's type doesn't match what the library requires — this error's family across 65 libraries.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/46cf4eadcbbbf8f5. Report an issue: GitHub.

Appendix: source

Thrown at sdks/python/apache_beam/io/fileio.py:253

class _ReadMatchesFn(beam.DoFn):
  def __init__(self, compression, skip_directories):
    self._compression = compression
    self._skip_directories = skip_directories

  def process(
      self,
      file_metadata: Union[str, filesystem.FileMetadata],
  ) -> Iterable[ReadableFile]:
    metadata = (
        filesystem.FileMetadata(file_metadata, 0) if isinstance(
            file_metadata, str) else file_metadata)

    if ((metadata.path.endswith('/') or metadata.path.endswith('\\')) and
        self._skip_directories):
      return
    elif metadata.path.endswith('/') or metadata.path.endswith('\\'):
      raise BeamIOError(
          'Directories are not allowed in ReadMatches transform.'
          'Found %s.' % metadata.path)

    # TODO: Mime type? Other arguments? Maybe arguments passed in to transform?
    yield ReadableFile(metadata, self._compression)


class _PollClock(object):
  """Shares one clock reading per poll round, so the start gate and the poll
  budget judge the ``start_timestamp`` boundary consistently."""
  def __init__(self):
    self.last_poll_micros: Optional[int] = None


class _WatchWindowTermination(TerminationCondition):
  """Stops after the polls that fall in the ``[start, stop)`` window.

  ``max_polls`` is the ``PeriodicImpulse`` tick count

View on GitHub (pinned to 12126d8942)