apache/beam · error · IOError

Could not upload to GCS path

Error message

Could not upload to GCS path %s: %s. Please verify that credentials are valid and that you have write access to the specified path.

What it means

_upload_to_gcs uploads a local file to a GCS bucket via the google-cloud-storage client before a Cloud Build. If the upload raises Forbidden or NotFound, the library converts it to IOError with guidance to check credentials and write access; any other exception is re-raised unchanged.

Solutions

1. Verify the GCS path and bucket name exist (gsutil ls <path> or gcloud storage ls)
2. Set up valid credentials: gcloud auth application-default login, or a service account key with Storage write access
3. Grant roles/storage.objectCreator on the bucket to the identity in use
4. Confirm storage.googleapis.com API access/network if errors persist

Example fix

// before
builder = SdkContainerImageBuilder(..., gcs_location='gs://wrong-bucket/sdk.tar')
// after
gcloud auth application-default login
gsutil mb gs://my-staging-bucket
builder = SdkContainerImageBuilder(..., gcs_location='gs://my-staging-bucket/sdk.tar')

from google.cloud import storage
from google.api_core.exceptions import Forbidden, NotFound
client = storage.Client()
try:
    bucket = client.get_bucket('my-staging-bucket')
    bucket.test_iam_permissions(['storage.objects.create'])
except (Forbidden, NotFound) as e:
    raise SystemExit(f'GCS staging path unusable before pipeline start: {e}')

def gcs_path_writable(client, gcs_location: str) -> bool:
    from urllib.parse import urlparse
    p = urlparse(gcs_location)
    if p.scheme != 'gs' or not p.netloc:
        return False
    try:
        b = client.get_bucket(p.netloc)
        return bool(b.test_iam_permissions(['storage.objects.create']))
    except Exception:
        return False

try:
    builder.build_container_image(...)
except IOError as e:
    if 'Could not upload to GCS path' in str(e):
        logger.error('Check credentials/write access: %s', e)
        raise SystemExit(7)
    raise

When it happens

Trigger: _invoke_docker_build_and_push calls _upload_to_gcs with a gcs_location whose bucket doesn't exist (NotFound) or whose credentials lack storage.objects.create on it (Forbidden) during blob.upload_from_filename.

Common situations: Application Default Credentials not set up (gcloud auth application-default login); service account lacking Storage Object Creator role; bucket name typo or bucket in another project; staging location path misconfigured in pipeline options.

Understand the failure class

Background: Permission denied / not authorized / 403 Forbidden: access-control rejections when the caller lacks the required role, grant, or ownership — this error's family across 18 libraries.

Related errors

• Could not upload to GCS path
• failed to write manifest
• Basepath %r must be GCS path.
• Bucket gs:// is not owned by project .
• cache_root GCS bucket path is invalid.

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/88e89b7f6729d43b. Report an issue: GitHub.

Appendix: source

Thrown at sdks/python/apache_beam/runners/portability/sdk_container_builder.py:320

        (time.time() - now))
    _LOGGER.info(
        "Python SDK container built and pushed as %s." % container_image_name)

  def _upload_to_gcs(self, local_file_path, gcs_location):
    bucket_name, blob_name = self._get_gcs_bucket_and_name(gcs_location)
    _LOGGER.info('Starting GCS upload to %s...', gcs_location)
    from google.cloud import storage
    from google.cloud.exceptions import Forbidden
    from google.cloud.exceptions import NotFound
    try:
      bucket = self._storage_client.get_bucket(bucket_name)
      blob = bucket.get_blob(blob_name)
      if not blob:
        blob = storage.Blob(name=blob_name, bucket=bucket)
      blob.upload_from_filename(local_file_path)
    except Exception as e:
      if isinstance(e, (Forbidden, NotFound)):
        raise IOError((
            'Could not upload to GCS path %s: %s. Please verify '
            'that credentials are valid and that you have write '
            'access to the specified path.') % (gcs_location, e.message))
      raise
    _LOGGER.info('Completed GCS upload to %s.', gcs_location)

  def _get_cloud_build_id_and_log_url(self, metadata):
    # google-cloud-build 3.35+
    if getattr(metadata, 'build', None):
      build = metadata.build
      return (build.id, build.log_url)
    # Fallback for older clients that use additionalProperties.
    id = None
    log_url = None
    additional_props = getattr(metadata, 'additionalProperties', None)
    if additional_props:
      for item in additional_props:
        if item.key == 'build':

View on GitHub (pinned to 12126d8942)