apache/beam · error · ValueError

Invalid GCS bucket provided!

Error message

Invalid GCS bucket provided!

What it means

assert_bucket_exists verifies a GCS cache bucket via the Storage client; if the client raises a NotFound ClientError, the bucket does not exist and a ValueError('Invalid GCS bucket provided!') is raised so caching is not silently pointed at a missing bucket.

Solutions

1. Create the bucket (gsutil mb or console) or correct the bucket name in the cache configuration
2. Verify the bucket exists: gsutil ls -b gs://<bucket>
3. Check credentials/project: ensure GOOGLE_APPLICATION_CREDENTIALS can see the bucket's project
4. Confirm the configured path is gcs://<existing-bucket>/... and not a subpath typo

Example fix

// before
ib.options.cache_destination = 'gs://my-typoed-bucket/cache'
// after
# ensure it exists first: gsutil mb -l us-central1 gs://my-bucket
ib.options.cache_destination = 'gs://my-bucket/cache'

from google.cloud import storage
def bucket_exists(name):
    try:
        storage.Client().get_bucket(name.split('gs://')[1].split('/')[0])
        return True
    except Exception:
        return False
assert bucket_exists(ib.options.cache_destination), 'bucket missing'

def is_valid_gcs_path(path):
    return isinstance(path, str) and path.startswith('gs://') and len(path.split('/')[2]) > 2

try:
    configure_cache('gs://my-bucket/cache')
except ValueError as e:
    if 'Invalid GCS bucket' in str(e):
        create_bucket_or_fix_name()

When it happens

Trigger: Configuring interactive Beam/GCS caching with a bucket name that does not exist, is misspelled, or lives in a project the credentials cannot see; the bucket was deleted after configuration.

Common situations: Setting cache_destination / staging_location gcs:// paths in notebooks; typos in bucket names; wrong GOOGLE_APPLICATION_CREDENTIALS pointing at a project without the bucket.

Understand the failure class

Background: 'Could not be found', 'does not exist', 'not found in database': the resource-not-found family when an ID, slug, key, or URI lookup comes back empty — this error's family across 20 libraries.

Related errors

• BigQuery temp location expected a valid 'gs://' path, but…
• bucket must be non-empty
• cache_root GCS bucket path is invalid.
• Error constructing default value for gcpTempLocation…
• failed to create GCS client

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/688e1bca4907b09b. Report an issue: GitHub.

Appendix: source

Thrown at sdks/python/apache_beam/runners/interactive/utils.py:457

def assert_bucket_exists(bucket_name: str) -> None:
  """Asserts whether the specified GCS bucket with the name
  bucket_name exists.

    Logs an error and raises a ValueError if the bucket does not exist.

    Logs a warning if the bucket cannot be verified to exist.
  """
  try:
    from google.cloud.exceptions import ClientError
    from google.cloud.exceptions import NotFound

    from apache_beam.io.gcp.gcsio import create_storage_client
    storage_client = create_storage_client(PipelineOptions())
    storage_client.get_bucket(bucket_name)
  except ClientError as e:
    if isinstance(e, NotFound):
      _LOGGER.error('%s bucket does not exist!', bucket_name)
      raise ValueError('Invalid GCS bucket provided!')
    else:
      _LOGGER.warning(
          'ClientError - unable to verify whether bucket %s exists',
          bucket_name)
  except ImportError:
    _LOGGER.warning(
        'ImportError - unable to verify whether bucket %s exists', bucket_name)


def detect_pipeline_runner(pipeline):
  if isinstance(pipeline, Pipeline):
    from apache_beam.runners.interactive.interactive_runner import InteractiveRunner
    if isinstance(pipeline.runner, InteractiveRunner):
      pipeline_runner = pipeline.runner._underlying_runner
    else:
      pipeline_runner = pipeline.runner
  else:
    pipeline_runner = None

View on GitHub (pinned to 12126d8942)