apache/beam · error · ValueError

GCS path must be in the form gs://

Error message

GCS path must be in the form gs://<bucket>/<object>. Encountered {gcs_path!r}

What it means

parse_gcs_path raises ValueError when the given string is not a valid GCS path of the form gs://<bucket>/<object>. It requires a non-empty bucket and, unless object_optional=True, a non-empty object name.

Solutions

1. Ensure the path matches gs://<bucket>/<object> with a non-empty bucket and object.
2. Add or fix the 'gs://' scheme prefix.
3. Pass object_optional=True only when bucket-only paths are intentional.
4. Use FullMatchGlob patterns or apache_beam.io.filesystems.FileSystems to handle mixed local/GCS paths.

Example fix

// before
parse_gcs_path('gs://my-bucket')
// after
parse_gcs_path('gs://my-bucket/path/to/object', object_optional=True)

import re
if not re.match(r'^gs://[^/]+/.+$', path):
    raise ValueError(f'invalid gcs path: {path!r}')

def is_gcs_path(p):
    return isinstance(p, str) and re.match(r'^gs://[^/]+/.+$', p) is not None

try:
    bucket, obj = parse_gcs_path(path)
except ValueError as e:
    logging.error('bad path: %s', e)
    return

When it happens

Trigger: Passing paths like 'gs://bucket' (no object, object_optional=False), 'gs:///obj', local paths like '/tmp/file', or paths with a different scheme to parse_gcs_path or its callers (open, delete, copy, exists).

Common situations: Mixing local file paths with GCS paths in a pipeline; forgetting the object part of the URI; missing 'gs://' prefix after string formatting; Windows-style paths.

Understand the failure class

Background: "Invalid URL" errors: why new URL(), URI.parse, and reqwest::Url reject your string — missing scheme, whitespace, and bad path format — this error's family across 39 libraries.

Related errors

• Can't get filename from root path in the bucket
• Can't resolve the sibling of a root path
• Could not find file
• Error constructing default value for gcpTempLocation…
• Expected a valid 'gs://' path but was given

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/87edf926b61680dc. Report an issue: GitHub.

Appendix: source

Thrown at sdks/python/apache_beam/io/gcp/gcsio.py:69

from apache_beam.options.pipeline_options import GoogleCloudOptions
from apache_beam.options.pipeline_options import PipelineOptions

__all__ = ['GcsIO', 'create_storage_client']

_LOGGER = logging.getLogger(__name__)

DEFAULT_READ_BUFFER_SIZE = 16 * 1024 * 1024

# Maximum number of operations permitted in GcsIO.copy_batch() and
# GcsIO.delete_batch().
MAX_BATCH_OPERATION_SIZE = 100


def parse_gcs_path(gcs_path, object_optional=False):
  """Return the bucket and object names of the given gs:// path."""
  match = re.match('^gs://([^/]+)/(.*)$', gcs_path)
  if match is None or (match.group(2) == '' and not object_optional):
    raise ValueError(
        'GCS path must be in the form gs://<bucket>/<object>. '
        f'Encountered {gcs_path!r}')
  return match.group(1), match.group(2)


def default_gcs_bucket_name(project, region):
  from hashlib import md5
  return 'dataflow-staging-%s-%s' % (
      region, md5(project.encode('utf8')).hexdigest())


def _get_project_number(project_id, credentials=None):
  """Resolves a project ID to its project number using Cloud Resource Manager API."""
  from google.cloud import resourcemanager_v3
  client = resourcemanager_v3.ProjectsClient(credentials=credentials)
  project_info = client.get_project(name=f"projects/{project_id}")
  # project_info.name is of the form "projects/PROJECT_NUMBER"
  return int(project_info.name.split('/')[-1])

View on GitHub (pinned to 12126d8942)