{"record":{"id":"2138f33302ce5e55","repo":"apache/beam","slug":"could-not-upload-to-gcs-path-s-s-please-verify-that","errorCode":null,"errorMessage":"Could not upload to GCS path %s: %s. Please verify that credentials are valid, that the specified path exists, and that you have write access to it.","messagePattern":"Could not upload to GCS path (.+?): (.+?)\\. Please verify that credentials are valid, that the specified path exists, and that you have write access to it\\.","errorType":"exception","errorClass":"IOError","httpStatus":null,"severity":"error","filePath":"sdks/python/apache_beam/runners/dataflow/internal/apiclient.py","lineNumber":720,"sourceCode":"      from google.cloud.storage.fileio import BlobWriter\n      bucket = self._storage_client.get_bucket(bucket_name)\n      blob = bucket.get_blob(blob_name)\n      if not blob:\n        blob = Blob(blob_name, bucket)\n      with BlobWriter(blob) as f:\n        f.write(stream.read())\n      _LOGGER.info(\n          'Completed GCS upload to %s in %s seconds.',\n          gcs_location,\n          int(time.time() - start_time))\n      return\n    except Exception as e:\n      reportable_errors = [\n          Forbidden,\n          NotFound,\n      ]\n      if type(e) in reportable_errors:\n        raise IOError((\n            'Could not upload to GCS path %s: %s. Please verify '\n            'that credentials are valid, that the specified path '\n            'exists, and that you have write access to it.') %\n                      (gcs_or_local_path, e))\n      raise\n\n  @retry.with_exponential_backoff(\n      retry_filter=retry.retry_on_server_errors_and_timeout_filter)\n  def stage_file_with_retry(\n      self,\n      gcs_or_local_path,\n      file_name,\n      stream_or_path,\n      mime_type='application/octet-stream',\n      total_size=None):\n\n    if isinstance(stream_or_path, str):\n      path = stream_or_path","sourceCodeStart":702,"sourceCodeEnd":738,"githubUrl":"https://github.com/apache/beam/blob/12126d8942aaf848030c478b4c6a28c6af861c66/sdks/python/apache_beam/runners/dataflow/internal/apiclient.py#L702-L738","documentation":"IOError raised by Job.stage_file when a GCS upload fails with googleapiclient Forbidden or NotFound. The message advises checking credentials, path existence, and write access. Other exception types are re-raised unchanged.","triggerScenarios":"stage_file() (called via stage_file_with_retry) uploading a staged file to a GCS path when GCS returns 403 Forbidden (bad credentials or no write permission) or 404 NotFound (bucket/prefix does not exist).","commonSituations":"Service account lacking roles/storage.objectCreator on the bucket; typo'd bucket name; bucket in another project or deleted; running with stale/expired Application Default Credentials.","solutions":["Verify the GCS path exists (gsutil ls <path>) and create the bucket/prefix if missing","Check credentials: gcloud auth application-default login or correct GOOGLE_APPLICATION_CREDENTIALS for the service account","Grant the caller storage.objects.create on the bucket (e.g. roles/storage.objectCreator)","Confirm the staging_location/temp_location point to the correct project and region"],"exampleFix":"# before\noptions.view_as(GoogleCloudOptions).staging_location = 'gs://wrong-bucket/staging'\n# after\nassert gcs_path_exists('gs://my-bucket/staging')  # or create it\noptions.view_as(GoogleCloudOptions).staging_location = 'gs://my-bucket/staging'","handlingStrategy":"validation","validationCode":"from google.cloud import storage\nc = storage.Client()\nbucket = c.bucket('my-bucket')\nif not bucket.exists():\n    raise SystemExit('bucket missing')\n# also verify write via a test object if needed","typeGuard":null,"tryCatchPattern":"try:\n    job = Job(pipeline, options)\nexcept IOError as e:\n    if 'Could not upload to GCS' in str(e):\n        print('Check credentials and bucket write access:', e)\n    raise","preventionTips":["Verify service-account storage permissions (storage.objects.create) on the staging bucket","Confirm buckets exist with gsutil ls before launching","Keep Application Default Credentials fresh (gcloud auth application-default login)","Pin staging/temp buckets to the same project/region as the job"],"tags":["dataflow","gcs","permissions","upload","credentials"],"backgroundTag":"permission-denied","analyzedSha":"12126d8942aaf848030c478b4c6a28c6af861c66","analyzedAt":"2026-09-13T01:50:10.254Z","contentChangedAt":"2026-09-13T01:50:10.254Z","schemaVersion":2},"datasetVersion":"2026-09-20T03:17:13.778Z"}