apache/beam · error · ValueError

ReadFromBigQuery requires a GCS location to be provided. Nei

Error message

ReadFromBigQuery requires a GCS location to be provided. Neither gcs_location in the constructor nor the fallback option --temp_location is set.

What it means

ReadFromBigQuery (batch extract via FILE_LOADS/Avro export) exports query results to files in GCS before reading them back. bigquery_export_destination_uri builds that export URI from gcs_location, falling back to --temp_location; if neither is set it raises this ValueError because there is nowhere to write the export files.

Source

Thrown at sdks/python/apache_beam/io/gcp/bigquery_read_internal.py:88

  """Returns the fully qualified Google Cloud Storage URI where the
  extracted table should be written.
  """
  file_pattern = 'bigquery-table-dump-*.json'

  gcs_location = None
  if gcs_location_vp is not None:
    if isinstance(gcs_location_vp, ValueProvider):
      gcs_location = gcs_location_vp.get()
    else:
      gcs_location = gcs_location_vp

  if gcs_location is not None:
    gcs_base = gcs_location
  elif temp_location is not None:
    gcs_base = temp_location
    _LOGGER.debug("gcs_location is empty, using temp_location instead")
  else:
    raise ValueError(
        'ReadFromBigQuery requires a GCS location to be provided. Neither '
        'gcs_location in the constructor nor the fallback option '
        '--temp_location is set.')

  if not unique_id:
    unique_id = uuid.uuid4().hex

  if directory_only:
    return FileSystems.join(gcs_base, unique_id)
  else:
    return FileSystems.join(gcs_base, unique_id, file_pattern)


class _PassThroughThenCleanup(PTransform):
  """A PTransform that invokes a DoFn after the input PCollection has been
    processed.

    DoFn should have arguments (element, side_input, cleanup_signal).

View on GitHub (pinned to 12126d8942)

Solutions

1. Pass gcs_location='gs://my-bucket/temp' to ReadFromBigQuery.
2. Add --temp_location gs://my-bucket/temp to your PipelineOptions.
3. If gcs_location is a runtime ValueProvider, ensure the value is actually supplied at job submission.

Example fix

// before
beam.io.ReadFromBigQuery(query='SELECT * FROM t', project='p')
// after
beam.io.ReadFromBigQuery(query='SELECT * FROM t', project='p', gcs_location='gs://my-bucket/temp')

opts = pipeline.options.view_as(beam.options.pipeline_options.GoogleCloudOptions)
if gcs_location is None and not opts.temp_location:
    raise ValueError('ReadFromBigQuery needs gcs_location or --temp_location')

def has_gcs_read_location(gcs_location, temp_location) -> bool:
    loc = gcs_location or temp_location
    return isinstance(loc, str) and loc.startswith('gs://')

When it happens

Trigger: Using ReadFromBigQuery (method=EXPORT) with gcs_location=None and no --temp_location in PipelineOptions; also hit by _export_files and file_path_to_remove when resolving the URI at runtime with an empty runtime ValueProvider.

Common situations: Running locally without --temp_location; setting gcs_location as a RuntimeValueProvider but never supplying the runtime value; forgetting GCS requirements when migrating from STREAMING_INSERTS reads.

Understand the failure class

Background: "is required", "must be set", "missing required field": configuration validation errors across open-source libraries — this error's family across 36 libraries.

Related errors

• Invalid GCS location: %r. Writing to BigQuery with FILE_LOAD
• Error constructing default value for gcpTempLocation: tempLo
• BigQuery temp location expected a valid 'gs://' path, but wa
• project must be specified either in ReadBigQueryChangeHistor
• Invalid GCS location: %r. Writing to BigQuery with FILE_LOAD

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/3dd8f9612bbb64c3. Report an issue: GitHub.