apache/beam · error · ValueError

project must be specified either in ReadBigQueryChangeHistor

Error message

project must be specified either in ReadBigQueryChangeHistory or in pipeline options (--project)

What it means

ReadBigQueryChangeHistory needs a GCP project ID to call the BigQuery change history API. It first takes the project passed to its constructor; if None, it falls back to --project in GoogleCloudOptions. If both are absent, it raises this ValueError because the API call cannot be scoped to a project.

Source

Thrown at sdks/python/apache_beam/io/gcp/bigquery_change_history.py:1246

    self._project = project
    self._temp_dataset = temp_dataset
    self._location = location
    self._change_type_column = change_type_column
    self._change_timestamp_column = change_timestamp_column
    self._columns = columns
    self._row_filter = row_filter
    self._batch_arrow_read = batch_arrow_read
    self._max_split_rounds = max_split_rounds
    self._reshuffle_decompress = reshuffle_decompress

  def expand(self, pbegin: beam.pvalue.PBegin) -> beam.PCollection:
    project = self._project
    if project is None:
      project = pbegin.pipeline.options.view_as(
          beam.options.pipeline_options.GoogleCloudOptions).project

    if project is None:
      raise ValueError(
          'project must be specified either in ReadBigQueryChangeHistory '
          'or in pipeline options (--project)')

    start_time = Timestamp(self._start_time or time.time())
    stop_time = (
        Timestamp(self._stop_time)
        if self._stop_time is not None else MAX_TIMESTAMP)
    buffer = Duration(seconds=self._buffer_sec)
    poll_interval = Duration(seconds=self._poll_interval_sec)

    temp_dataset = self._temp_dataset
    if temp_dataset is None:
      temp_dataset = f'beam_ch_temp_{uuid.uuid4().hex[:12]}'

    _LOGGER.info(
        '[ReadBigQueryChangeHistory] expand: table=%s, project=%s, '
        'change_function=%s, poll_interval=%d sec, buffer=%d sec, '
        'temp_dataset=%s, start_time=%s, stop_time=%s',

View on GitHub (pinned to 12126d8942)

Solutions

1. Pass project='my-project' to the ReadBigQueryChangeHistory constructor.
2. Set --project (GoogleCloudOptions.project) in your PipelineOptions when creating the pipeline.
3. Set the default project via 'gcloud config set project my-project' or the GOOGLE_CLOUD_PROJECT environment variable so options resolution picks it up.

Example fix

// before
beam.io.gcp.bigquery_change_history.ReadBigQueryChangeHistory(query='SELECT 1')
// after
beam.io.gcp.bigquery_change_history.ReadBigQueryChangeHistory(query='SELECT 1', project='my-gcp-project')

opts = pipeline.options.view_as(beam.options.pipeline_options.GoogleCloudOptions)
if transform._project is None and not opts.project:
    raise ValueError('Set project on ReadBigQueryChangeHistory or via --project')

def has_project(t) -> bool:
    return bool(getattr(t, '_project', None))

When it happens

Trigger: Constructing ReadBigQueryChangeHistory with project=None (or omitted) and launching the pipeline without the --project flag in GoogleCloudOptions.

Common situations: Running a pipeline on a machine with no gcloud default project configured, using a Dataflow template without the project parameter, or constructing the transform programmatically and forgetting to set GoogleCloudOptions.project.

Understand the failure class

Background: "is required", "must be set", "missing required field": configuration validation errors across open-source libraries — this error's family across 36 libraries.

Related errors

• Invalid GCS location: %r. Writing to BigQuery with FILE_LOAD
• triggering_frequency must be specified to use fileloads in s
• ReadFromBigQuery requires a GCS location to be provided. Nei
• table does not exist and create disposition is 'CreateNever'
• Unrecognized value for stable unique names:

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/440bdf7729b7d490. Report an issue: GitHub.