apache/beam · error · ValueError

Bucket gs:// is not owned by project .

Error message

Bucket gs://{bucket.name} is not owned by project {project_id}.

What it means

_validate_bucket_project raises ValueError when a GCS bucket is owned by a different project than the Beam pipeline project. During default bucket creation, Beam compares the bucket's project number with the current project number and fails if they differ.

Solutions

1. Use a different, project-specific bucket name (set via options like staging/temp location) so a foreign-named bucket isn't reused.
2. Create a fresh bucket owned by the current project and configure it explicitly.
3. Verify the bucket's owner project with `gsutil bucket get` or the storage API.
4. Delete/reclaim the old bucket if it is no longer needed by the other project.

Example fix

// before
python -m apache_beam... --project new-proj  # reuses old bucket owned by old-proj
// after
--temp_location gs://new-proj-dataflow-temp/temp --staging_location gs://new-proj-dataflow-temp/staging

from google.cloud import storage
client = storage.Client(project=project_id)
b = client.get_bucket(bucket_name)
info = client._connection.api_request('GET', f'/b/{b.name}?projection=noAcl')
# compare info['projectNumber'] with your project's number before relying on the bucket

def bucket_owned_by(bucket_info, project_number):
    return bucket_info.get('projectNumber') == project_number

try:
    bucket = get_or_create_default_gcs_bucket(options)
except ValueError as e:
    logging.error('bucket project mismatch: %s', e)
    # fall back to an explicitly configured bucket

When it happens

Trigger: Running get_or_create_default_gcs_bucket when a bucket with the default name already exists but belongs to another project (e.g. after changing the GCP project ID or reusing a bucket name across projects).

Common situations: Reusing an old Dataflow staging bucket name in a new project; project ID renamed/recreated so project numbers differ; shared bucket naming conventions across teams.

Understand the failure class

Background: Permission denied / not authorized / 403 Forbidden: access-control rejections when the caller lacks the required role, grant, or ownership — this error's family across 18 libraries.

Related errors

• Could not upload to GCS path
• Could not upload to GCS path
• failed to read from
• failed to write manifest
• Unable create default bucket.

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/69cf319284043719. Report an issue: GitHub.

Appendix: source

Thrown at sdks/python/apache_beam/io/gcp/gcsio.py:119

  if bucket_project_number is None:
    _LOGGER.warning(
        'Bucket gs://%s does not have a project number. Skipping ownership validation.',
        bucket.name)
    return

  try:
    project_number = _get_project_number(project_id, credentials=credentials)
  except Exception as e:
    _LOGGER.warning(
        'Failed to resolve project number for project %s: %s. '
        'Skipping bucket ownership validation.',
        project_id,
        e)
    return

  if bucket_project_number != project_number:
    raise ValueError(
        f'Bucket gs://{bucket.name} is not owned by project {project_id}.')


def get_or_create_default_gcs_bucket(options):
  """Create a default GCS bucket for this project."""
  if getattr(options, 'dataflow_kms_key', None):
    _LOGGER.warning(
        'Cannot create a default bucket when --dataflow_kms_key is set.')
    return None

  project = getattr(options, 'project', None)
  region = getattr(options, 'region', None)
  if not project or not region:
    return None

  bucket_name = default_gcs_bucket_name(project, region)
  gcs = GcsIO(pipeline_options=options)
  bucket = gcs.get_bucket(bucket_name)

View on GitHub (pinned to 12126d8942)