apache/beam · error · ValueError

Unable to get filesystem from specified path, please use the

Error message

Unable to get filesystem from specified path, please use the correct path or ensure the required dependency is installed, e.g., pip install apache-beam[gcp]. Path specified: %s

What it means

FileSystems.get_filesystem(path) resolves the URL scheme (e.g. gs://, s3://, hdfs://) against all registered FileSystem subclasses; if none matches the path's scheme it raises ValueError telling the user to check the path or install the required dependency such as apache-beam[gcp]. Beam cannot route I/O for a scheme it has no implementation for.

Source

Thrown at sdks/python/apache_beam/io/filesystems.py:139

    match_result = FileSystems.URI_SCHEMA_PATTERN.match(path.strip())
    if match_result is None:
      return None
    return match_result.groupdict()['scheme']

  @staticmethod
  def get_filesystem(path):
    # type: (str) -> FileSystem

    """Get the correct filesystem for the specified path
    """
    try:
      path_scheme = FileSystems.get_scheme(path)
      systems = [
          fs for fs in FileSystem.get_all_subclasses()
          if fs.scheme() == path_scheme
      ]
      if len(systems) == 0:
        raise ValueError(
            'Unable to get filesystem from specified path, please use the '
            'correct path or ensure the required dependency is installed, '
            'e.g., pip install apache-beam[gcp]. Path specified: %s' % path)
      elif len(systems) == 1:
        # Pipeline options could come either from the Pipeline itself (using
        # direct runner), or via RuntimeValueProvider (other runners).
        options = (
            FileSystems._pipeline_options or
            RuntimeValueProvider.runtime_options)
        return systems[0](pipeline_options=options)
      else:
        raise ValueError('Found more than one filesystem for path %s' % path)
    except ValueError:
      raise
    except Exception as e:
      raise BeamIOError('Unable to get the Filesystem', {path: e})

  @staticmethod

View on GitHub (pinned to 12126d8942)

Solutions

Install the extra that provides the filesystem: pip install 'apache-beam[gcp]' (or [aws], [azure], as appropriate).
Fix the path so it uses a supported scheme, e.g. 'gs://bucket/path' not 'gcs://bucket/path'.
Verify the scheme is registered: check FileSystem.get_all_subclasses() and their .scheme() values.
For local files, use plain filesystem paths ('/tmp/x' or 'file:///tmp/x') rather than remote schemes.
If you wrote a custom FileSystem, ensure its module is imported (subclass registration happens at import time) and its scheme() matches the path prefix.

Example fix

// before
pip install apache-beam
FileSystems.get_filesystem('gs://bucket/data.csv')  # ValueError
// after
pip install 'apache-beam[gcp]'
FileSystems.get_filesystem('gs://bucket/data.csv')  # -> GCSFileSystem

Defensive patterns

Strategy: fallback

Validate before calling

from urllib.parse import urlparse
from apache_beam.io import FileSystem
scheme = urlparse(path).scheme
known = {fs.scheme() for fs in FileSystem.get_all_subclasses()}
if scheme not in known:
    raise ValueError(f'no filesystem for scheme {scheme!r}; install the matching apache-beam extra')

Try / catch

try:
    fs = FileSystems.get_filesystem(path)
except ValueError as e:
    if 'Unable to get filesystem' in str(e):
        raise RuntimeError(
            f'{e}; install the right extra: pip install apache-beam[gcp|aws]'
        ) from e
    raise

Prevention

Install the extras you use: apache-beam[gcp], [aws], [azure].
Use canonical schemes: gs:// (not gcs://), s3://, hdfs://.
Verify scheme support early in job setup by calling FileSystems.get_filesystem on a sample path.
If using a custom FileSystem, import its module before first use (registration is at import time).

When it happens

Trigger: Calling FileSystems.get_filesystem('gs://bucket/obj') without the GCP extras installed; using a misspelled or missing scheme (e.g. 'gcs://bucket' instead of 'gs://', or a bare '/local/path' with no scheme where local_filesystem isn't registered).

Common situations: Installing plain apache-beam instead of apache-beam[gcp] and then reading GCS paths; forgetting extras like apache-beam[aws] for S3; typos in path schemes in pipeline options; custom filesystem plugin not imported/registered before use.

Understand the failure class

Background: "X is not installed. Please install it with pip install Y": missing optional dependency errors — ImportError/ValueError raised when a library's optional extra was never installed — this error's family across 22 libraries.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/a398cf9e07f626ec. Report an issue: GitHub.