apache/beam · error · ValueError

Path %r must be GCS path.

Error message

Path %r must be GCS path.

What it means

GCSFileSystem.split only understands paths carrying the 'gs://' prefix; the path passed in does not start with GCS_PREFIX, so it cannot be decomposed into a GCS head/tail pair. This is a path-validation guard for filesystem-agnostic code that dispatched to GCSFileSystem by mistake.

Solutions

1. Prefix the path with 'gs://' before splitting.
2. Route the path through FileSystems.get_filesystem / FileSystems.match so the correct scheme handler runs.
3. Fix the pipeline option (e.g. --temp_location) to a gs:// URL when using GCS.

Example fix

# before
GCSFileSystem.split('gcs://bucket/dir/file')
# after
GCSFileSystem.split('gs://bucket/dir/file')

def assert_gcs_path(path):
    if not isinstance(path, str) or not path.startswith('gs://'):
        raise ValueError('Path must be GCS path: %r' % path)
    return path.strip()

def is_gcs_path(path):
    return isinstance(path, str) and path.strip().startswith('gs://')

try:
    directory, name = fs.split(path)
except ValueError as e:
    logging.error('split() got non-GCS path: %s', e)
    raise

When it happens

Trigger: Calling GCSFileSystem.split('s3://bucket/key') or split('/local/path/file'); passing an unnormalized path from user config into Beam's GCS match/copy machinery.

Common situations: Mixing local and GCS paths in pipeline options (e.g. a local temp_location fed to a GCSFileSystem-based code path); typos like 'gcs://bucket' or missing prefix.

Understand the failure class

Background: "Invalid URL" errors: why new URL(), URI.parse, and reqwest::Url reject your string — missing scheme, whitespace, and bad path format — this error's family across 39 libraries.

Related errors

• Basepath %r must be GCS path.
• Destination %r must be GCS path.
• Invalid path
• cache_root GCS bucket path is invalid.
• combine_fn must be specified.

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/04b6f50679a7951e. Report an issue: GitHub.

Appendix: source

Thrown at sdks/python/apache_beam/io/gcp/gcsfilesystem.py:90

      path = path.rstrip('/') + '/' + p.lstrip('/')
    return path

  def split(self, path):
    """Splits the given path into two parts.

    Splits the path into a pair (head, tail) such that tail contains the last
    component of the path and head contains everything up to that.

    Head will include the GCS prefix ('gs://').

    Args:
      path: path as a string
    Returns:
      a pair of path components as strings.
    """
    path = path.strip()
    if not path.startswith(GCSFileSystem.GCS_PREFIX):
      raise ValueError('Path %r must be GCS path.' % path)

    prefix_len = len(GCSFileSystem.GCS_PREFIX)
    last_sep = path[prefix_len:].rfind('/')
    if last_sep >= 0:
      last_sep += prefix_len

    if last_sep > 0:
      return (path[:last_sep], path[last_sep + 1:])
    elif last_sep < 0:
      return (path, '')
    else:
      raise ValueError('Invalid path: %s' % path)

  def mkdirs(self, path):
    """Recursively create directories for the provided path.

    Args:
      path: string path of the directory structure that should be created

View on GitHub (pinned to 12126d8942)