apache/beam · error · ValueError

temp_dataset has to be either str or DatasetReference

Error message

temp_dataset has to be either str or DatasetReference

What it means

ReadFromBigQuery's temp_dataset parameter names the BigQuery dataset used for temporary tables; it may be a dataset id string or a DatasetReference. _get_temp_dataset_id narrows the value: None, DatasetReference, or str are accepted; any other type raises this ValueError.

Solutions

1. Pass a plain string: temp_dataset='my_temp_dataset'.
2. Pass apache_beam.io.gcp.internal.clients.bigquery.DatasetReference(projectId='p', datasetId='d').
3. Convert dict values to DatasetReference before constructing the transform.

Example fix

// before
beam.io.ReadFromBigQuery(..., temp_dataset={'projectId': 'p', 'datasetId': 'd'})
// after
from apache_beam.io.gcp.internal.clients.bigquery import DatasetReference
beam.io.ReadFromBigQuery(..., temp_dataset=DatasetReference(projectId='p', datasetId='d'))

if temp_dataset is not None and not isinstance(temp_dataset, (str, DatasetReference)):
    raise TypeError('temp_dataset must be str or DatasetReference')

def is_valid_temp_dataset(v) -> bool:
    from apache_beam.io.gcp.internal.clients.bigquery import DatasetReference
    return v is None or isinstance(v, (str, DatasetReference))

When it happens

Trigger: Passing temp_dataset as a dict, a TableName, or other non-str/non-DatasetReference object to ReadFromBigQuery (or its internal _CustomBigQuerySource/_CustomBQSession transformer).

Common situations: Constructing a dict like {'projectId':..., 'datasetId':...} instead of a DatasetReference; passing a table reference rather than a dataset reference.

Understand the failure class

Background: Type mismatch errors: IllegalArgumentException, TypeError and type guards across 150 open-source libraries — this error's family across 150 libraries.

Related errors

• custom_gcs_temp_location must be str or ValueProvider
• Does not support converting unknown type value: " +…
• Error converting field :
• is not primitive type.
• Problem converting field

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/8cd87c838de21009. Report an issue: GitHub.

Appendix: source

Thrown at sdks/python/apache_beam/io/gcp/bigquery_read_internal.py:238

  def display_data(self):
    return {
        'use_json_exports': str(self.use_json_exports),
        'gcs_location': str(self.gcs_location),
        'bigquery_job_labels': json.dumps(self.bigquery_job_labels),
        'kms_key': str(self.kms_key),
        'project': str(self.project),
        'temp_dataset': str(self.temp_dataset)
    }

  def _get_temp_dataset_id(self):
    if self.temp_dataset is None:
      return None
    elif isinstance(self.temp_dataset, DatasetReference):
      return self.temp_dataset.datasetId
    elif isinstance(self.temp_dataset, str):
      return self.temp_dataset
    else:
      raise ValueError("temp_dataset has to be either str or DatasetReference")

  def _get_temp_dataset_project(self):
    """Returns the project ID for temporary dataset operations.
    
    If temp_dataset is a DatasetReference, returns its projectId.
    Otherwise, returns the pipeline project for billing.
    """
    if isinstance(self.temp_dataset, DatasetReference):
      return self.temp_dataset.projectId
    else:
      return self._get_project()

  def start_bundle(self):
    self.bq = bigquery_tools.BigQueryWrapper(
        temp_dataset_id=self._get_temp_dataset_id(),
        client=bigquery_tools.BigQueryWrapper._bigquery_client(self.options))

  def process(self,

View on GitHub (pinned to 12126d8942)