apache/beam · error · ValueError

Found more than one filesystem for path %s

Error message

Found more than one filesystem for path %s

What it means

FileSystems.get_filesystem requires exactly one FileSystem subclass whose scheme() matches the path's scheme; when two or more implementations claim the same scheme it raises ValueError('Found more than one filesystem for path %s') because Beam cannot decide which to instantiate. This is an ambiguous-registration condition, not a per-request I/O failure.

Source

Thrown at sdks/python/apache_beam/io/filesystems.py:151

      path_scheme = FileSystems.get_scheme(path)
      systems = [
          fs for fs in FileSystem.get_all_subclasses()
          if fs.scheme() == path_scheme
      ]
      if len(systems) == 0:
        raise ValueError(
            'Unable to get filesystem from specified path, please use the '
            'correct path or ensure the required dependency is installed, '
            'e.g., pip install apache-beam[gcp]. Path specified: %s' % path)
      elif len(systems) == 1:
        # Pipeline options could come either from the Pipeline itself (using
        # direct runner), or via RuntimeValueProvider (other runners).
        options = (
            FileSystems._pipeline_options or
            RuntimeValueProvider.runtime_options)
        return systems[0](pipeline_options=options)
      else:
        raise ValueError('Found more than one filesystem for path %s' % path)
    except ValueError:
      raise
    except Exception as e:
      raise BeamIOError('Unable to get the Filesystem', {path: e})

  @staticmethod
  def join(basepath, *paths):
    # type: (str, *str) -> str

    """Join two or more pathname components for the filesystem

    Args:
      basepath: string path of the first component of the path
      paths: path components to be added

    Returns: full path after combining all the passed components
    """
    filesystem = FileSystems.get_filesystem(basepath)

View on GitHub (pinned to 12126d8942)

Solutions

  1. Remove or de-register the duplicate FileSystem subclass so exactly one implementation remains for the scheme.
  2. Make the custom filesystem's scheme() return a unique scheme string instead of colliding with a built-in one.
  3. In tests, scope mock registration so it doesn't run alongside real imports of the built-in filesystem module.
  4. Avoid importing vendored copies of apache_beam.io filesystem modules; use the installed package's classes.

Example fix

// before
class MyGcsFileSystem(FileSystem):
  @classmethod
  def scheme(cls):
    return 'gs'  # collides with built-in GCSFileSystem
// after
class MyGcsFileSystem(FileSystem):
  @classmethod
  def scheme(cls):
    return 'mygs'  # unique scheme; use mygs://bucket/... paths
Defensive patterns

Strategy: validation

Validate before calling

from apache_beam.io import FileSystem
scheme = urlparse(path).scheme
matches = [fs for fs in FileSystem.get_all_subclasses() if fs.scheme() == scheme]
assert len(matches) <= 1, f'ambiguous filesystems for scheme {scheme}: {matches}'

Try / catch

try:
    fs = FileSystems.get_filesystem(path)
except ValueError as e:
    if 'Found more than one filesystem' in str(e):
        # pick the intended implementation explicitly instead of ambiguity
        from apache_beam.io.gcp.gcsfilesystem import GCSFileSystem
        fs = GCSFileSystem(pipeline_options=None)
    else:
        raise

Prevention

When it happens

Trigger: Importing a custom/third-party FileSystem whose scheme() returns 'gs' (or 's3', 'file', ...) when a built-in implementation for the same scheme is also registered — both subclasses exist and len(systems) > 1.

Common situations: Bundling a vendored copy of a Beam filesystem class alongside the real one; test fixtures registering a mock filesystem for a real scheme and leaking the registration; two IO plugins both claiming the same URI scheme.

Understand the failure class

Background: "Invalid value" and "allowed values are" config errors: what your library rejected and how to fix it — this error's family across 41 libraries.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/2a740296e296c9d9. Report an issue: GitHub.