apache/beam · error · ValueError

Invalid GCS location: %r. Writing to BigQuery with FILE_LOAD

Error message

Invalid GCS location: %r.
Writing to BigQuery with FILE_LOADS method requires a GCS location to be provided to write files to be loaded into BigQuery. Please provide a GCS bucket, or pass method="STREAMING_INSERTS" to WriteToBigQuery.

What it means

After resolving custom_gcs_temp_location, BigQueryBatchFileLoads verifies that a statically-provided location is a valid GCS URI ('gs://...'). If the value was provided but does not point at GCS, verify() raises this ValueError at graph-construction time.

Source

Thrown at sdks/python/apache_beam/io/gcp/bigquery_file_loads.py:1022

    # thus we will need temporary tables for atomicity.
    self.dynamic_destinations = bool(callable(destination))

    self.additional_bq_parameters = additional_bq_parameters or {}
    self.table_side_inputs = table_side_inputs or ()
    self.schema_side_inputs = schema_side_inputs or ()

    self.is_streaming_pipeline = is_streaming_pipeline
    self.load_job_project_id = load_job_project_id
    self._validate = validate
    if self._validate:
      self.verify()

  def verify(self):
    if (isinstance(self._custom_gcs_temp_location.get(), vp.StaticValueProvider)
        and not self._custom_gcs_temp_location.get().startswith('gs://')):
      # Only fail if the custom location is provided, and it is not a GCS
      # location.
      raise ValueError(
          'Invalid GCS location: %r.\n'
          'Writing to BigQuery with FILE_LOADS method requires a '
          'GCS location to be provided to write files to be '
          'loaded into BigQuery. Please provide a GCS bucket, or '
          'pass method="STREAMING_INSERTS" to WriteToBigQuery.' %
          self._custom_gcs_temp_location.get())
    if self.is_streaming_pipeline and not self.triggering_frequency:
      raise ValueError(
          'triggering_frequency must be specified to use file'
          'loads in streaming')
    elif not self.is_streaming_pipeline and self.triggering_frequency:
      raise ValueError(
          'triggering_frequency can only be used with file'
          'loads in streaming')
    if not self.is_streaming_pipeline and self.with_auto_sharding:
      return ValueError(
          'with_auto_sharding can only be used with file loads in streaming.')

View on GitHub (pinned to 12126d8942)

Solutions

Set custom_gcs_temp_location to a full 'gs://bucket/path' URI.
Use --temp_location gs://bucket/path instead and omit custom_gcs_temp_location.
Switch to method='STREAMING_INSERTS' if GCS staging is impossible.

Example fix

// before
beam.io.WriteToBigQuery(table, custom_gcs_temp_location='/tmp/staging')
// after
beam.io.WriteToBigQuery(table, custom_gcs_temp_location='gs://my-bucket/staging')

Defensive patterns

Strategy: validation

Validate before calling

loc = custom_gcs_temp_location
if loc is not None and not str(loc).startswith('gs://'):
    raise ValueError(f'custom_gcs_temp_location must be a gs:// URI, got {loc!r}')

Type guard

def is_gcs_uri(v) -> bool:
    return isinstance(v, str) and v.startswith('gs://')

Try / catch

try:
    transform = beam.io.WriteToBigQuery(table, custom_gcs_temp_location=loc)
except ValueError as e:
    if 'Invalid GCS location' in str(e):
        fix_temp_location()

Prevention

Validate bucket URIs against a gs:// prefix regex before constructing transforms
Avoid local filesystem paths for staging
Use pipeline-level --temp_location as a single source of truth

When it happens

Trigger: Passing custom_gcs_temp_location='/tmp/staging' or 's3://bucket' (not starting with gs://) as a StaticValueProvider to WriteToBigQuery with FILE_LOADS.

Common situations: Using a local temp directory in tests, copying an S3 path from another pipeline, or forgetting the 'gs://' scheme on a bucket name.

Understand the failure class

Background: "Invalid URL" errors: why new URL(), URI.parse, and reqwest::Url reject your string — missing scheme, whitespace, and bad path format — this error's family across 39 libraries.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/6cb6c4934a7acc22. Report an issue: GitHub.