apache/beam · error · IOError

Could not upload to GCS path

Error message

Could not upload to GCS path %s: %s. Please verify that credentials are valid, that the specified path exists, and that you have write access to it.

What it means

IOError raised by Job.stage_file when a GCS upload fails with googleapiclient Forbidden or NotFound. The message advises checking credentials, path existence, and write access. Other exception types are re-raised unchanged.

Solutions

1. Verify the GCS path exists (gsutil ls <path>) and create the bucket/prefix if missing
2. Check credentials: gcloud auth application-default login or correct GOOGLE_APPLICATION_CREDENTIALS for the service account
3. Grant the caller storage.objects.create on the bucket (e.g. roles/storage.objectCreator)
4. Confirm the staging_location/temp_location point to the correct project and region

Example fix

# before
options.view_as(GoogleCloudOptions).staging_location = 'gs://wrong-bucket/staging'
# after
assert gcs_path_exists('gs://my-bucket/staging')  # or create it
options.view_as(GoogleCloudOptions).staging_location = 'gs://my-bucket/staging'

from google.cloud import storage
c = storage.Client()
bucket = c.bucket('my-bucket')
if not bucket.exists():
    raise SystemExit('bucket missing')
# also verify write via a test object if needed

try:
    job = Job(pipeline, options)
except IOError as e:
    if 'Could not upload to GCS' in str(e):
        print('Check credentials and bucket write access:', e)
    raise

When it happens

Trigger: stage_file() (called via stage_file_with_retry) uploading a staged file to a GCS path when GCS returns 403 Forbidden (bad credentials or no write permission) or 404 NotFound (bucket/prefix does not exist).

Common situations: Service account lacking roles/storage.objectCreator on the bucket; typo'd bucket name; bucket in another project or deleted; running with stale/expired Application Default Credentials.

Understand the failure class

Background: Permission denied / not authorized / 403 Forbidden: access-control rejections when the caller lacks the required role, grant, or ownership — this error's family across 18 libraries.

Related errors

• Could not upload to GCS path
• failed to write manifest
• Bucket gs:// is not owned by project .
• could not resolve to a temp directory for import batch files
• Failed to cancel job

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/2138f33302ce5e55. Report an issue: GitHub.

Appendix: source

Thrown at sdks/python/apache_beam/runners/dataflow/internal/apiclient.py:720

      from google.cloud.storage.fileio import BlobWriter
      bucket = self._storage_client.get_bucket(bucket_name)
      blob = bucket.get_blob(blob_name)
      if not blob:
        blob = Blob(blob_name, bucket)
      with BlobWriter(blob) as f:
        f.write(stream.read())
      _LOGGER.info(
          'Completed GCS upload to %s in %s seconds.',
          gcs_location,
          int(time.time() - start_time))
      return
    except Exception as e:
      reportable_errors = [
          Forbidden,
          NotFound,
      ]
      if type(e) in reportable_errors:
        raise IOError((
            'Could not upload to GCS path %s: %s. Please verify '
            'that credentials are valid, that the specified path '
            'exists, and that you have write access to it.') %
                      (gcs_or_local_path, e))
      raise

  @retry.with_exponential_backoff(
      retry_filter=retry.retry_on_server_errors_and_timeout_filter)
  def stage_file_with_retry(
      self,
      gcs_or_local_path,
      file_name,
      stream_or_path,
      mime_type='application/octet-stream',
      total_size=None):

    if isinstance(stream_or_path, str):
      path = stream_or_path

View on GitHub (pinned to 12126d8942)