apache/beam · error · ValueError

Invalid GCS location: %r. Writing to BigQuery with FILE_LOAD

Error message

Invalid GCS location: %r.
Writing to BigQuery with FILE_LOADS method requires a GCS location to be provided to write files to be loaded into BigQuery. Please provide a GCS bucket through custom_gcs_temp_location in the constructor of WriteToBigQuery or the fallback option --temp_location, or pass method="STREAMING_INSERTS" to WriteToBigQuery.

What it means

The BigQuery FILE_LOADS write method stages rows as files in GCS before loading them into BigQuery. Before writing, Beam validates that the resolved GCS base path (custom_gcs_temp_location, else temp_location) exists and starts with 'gs://'. If not, it raises this ValueError early in the pipeline rather than failing later at execution time.

Source

Thrown at sdks/python/apache_beam/io/gcp/bigquery_file_loads.py:132

      step_id=step_name,
      job_type=job_type,
      random=_bq_uuid())


def file_prefix_generator(
    with_validation=True, pipeline_gcs_location=None, temp_location=None):
  def _generate_file_prefix(unused_elm):
    # If a gcs location is provided to the pipeline, then we shall use that.
    # Otherwise, we shall use the temp_location from pipeline options.
    gcs_base = pipeline_gcs_location.get()
    if not gcs_base:
      gcs_base = temp_location

    # This will fail at pipeline execution time, but will fail early, as this
    # step doesn't have any dependencies (and thus will be one of the first
    # stages to be run).
    if with_validation and (not gcs_base or not gcs_base.startswith('gs://')):
      raise ValueError(
          'Invalid GCS location: %r.\n'
          'Writing to BigQuery with FILE_LOADS method requires a'
          ' GCS location to be provided to write files to be loaded'
          ' into BigQuery. Please provide a GCS bucket through'
          ' custom_gcs_temp_location in the constructor of WriteToBigQuery'
          ' or the fallback option --temp_location, or pass'
          ' method="STREAMING_INSERTS" to WriteToBigQuery.' % gcs_base)

    prefix_uuid = _bq_uuid()
    return fs.FileSystems.join(gcs_base, 'bq_load', prefix_uuid)

  return _generate_file_prefix


def _make_new_file_writer(
    file_prefix,
    destination,
    file_format,

View on GitHub (pinned to 12126d8942)

Solutions

1. Pass custom_gcs_temp_location='gs://my-bucket/temp' to WriteToBigQuery.
2. Provide --temp_location gs://my-bucket/temp in PipelineOptions.
3. Switch to method='STREAMING_INSERTS' if a GCS staging bucket is not available.

Example fix

// before
result | beam.io.WriteToBigQuery('proj:dataset.table')
// after
result | beam.io.WriteToBigQuery('proj:dataset.table',
    custom_gcs_temp_location='gs://my-bucket/temp')

loc = custom_gcs_temp_location or temp_location
if not loc or not str(loc).startswith('gs://'):
    raise ValueError('Need a gs:// staging location for FILE_LOADS')

def is_gcs_uri(loc) -> bool:
    return isinstance(loc, str) and loc.startswith('gs://')

try:
    result | beam.io.WriteToBigQuery(table, method=WriteToBigQuery.Method.FILE_LOADS)
except ValueError as e:
    if 'Invalid GCS location' in str(e):
        rerun_with_gcs_location()

When it happens

Trigger: Calling WriteToBigQuery with method=FILE_LOADS (the default for batch) where neither custom_gcs_temp_location nor the pipeline's --temp_location is set, or the resolved location is not a 'gs://' URI.

Common situations: Running batch BigQuery loads locally without --temp_location, passing a local path like '/tmp' instead of a GCS bucket, or migrating code that previously used STREAMING_INSERTS to FILE_LOADS.

Understand the failure class

Background: "Invalid URL" errors: why new URL(), URI.parse, and reqwest::Url reject your string — missing scheme, whitespace, and bad path format — this error's family across 39 libraries.

Related errors

• ReadFromBigQuery requires a GCS location to be provided. Nei
• Error constructing default value for gcpTempLocation: tempLo
• BigQuery temp location expected a valid 'gs://' path, but wa
• project must be specified either in ReadBigQueryChangeHistor
• Invalid GCS location: %r. Writing to BigQuery with FILE_LOAD

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/5bae01145a7cb063. Report an issue: GitHub.