apache/beam · error · ValueError

Both a BigQuery temp_dataset_id and a temp_table_ref were…

Error message

Both a BigQuery temp_dataset_id and a temp_table_ref were specified. Please specify only one of these.

What it means

Raised in BigQueryWrapper.__init__ when both temp_dataset_id and temp_table_ref are supplied. A temp_table_ref already implies its dataset, so specifying both is ambiguous and Beam rejects the combination.

Solutions

Remove temp_dataset_id and keep only temp_table_ref (the ref carries the dataset).
Or remove temp_table_ref and keep only temp_dataset_id if you just want a custom temp dataset.
Audit config/option plumbing so only one temp-location setting is populated.
Let Beam create its own temp dataset by passing neither argument.

Example fix

// before
BigQueryWrapper(temp_dataset_id='my_tmp_ds', temp_table_ref=ref)

// after
BigQueryWrapper(temp_table_ref=ref)

Defensive patterns

Strategy: validation

Validate before calling

if temp_dataset_id and temp_table_ref:
    raise ValueError('pass either temp_dataset_id or temp_table_ref, not both')

Type guard

def temp_options_are_consistent(opts):
    return not (opts.get('temp_dataset_id') and opts.get('temp_table_ref'))

Try / catch

try:
    wrapper = BigQueryWrapper(temp_dataset_id=ds, temp_table_ref=ref)
except ValueError as e:
    if 'only one of these' in str(e):
        wrapper = BigQueryWrapper(temp_table_ref=ref)  # ref wins
    else:
        raise

Prevention

Choose one temp-location mechanism per pipeline and document it
When migrating from temp_dataset_id to temp_table_ref, delete the old option
Assert mutual exclusivity in pipeline option parsing

When it happens

Trigger: Constructing BigQueryWrapper(temp_dataset_id='x', temp_table_ref=<TableReference>) directly, or via pipeline options where both --temp_dataset_id-style custom options and a temp table reference end up set.

Common situations: Migrating code from the temp_dataset_id option to temp_table_ref and leaving both configured; wiring wrapper kwargs from two config sources that both define temp locations.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/76591e4a3e65f5c1. Report an issue: GitHub.

Appendix: source

Thrown at sdks/python/apache_beam/io/gcp/bigquery_tools.py:385

  def __init__(self, client=None, temp_dataset_id=None, temp_table_ref=None):
    self.client = client or BigQueryWrapper._bigquery_client(PipelineOptions())
    self.gcp_bq_client = client or gcp_bigquery.Client(
        client_info=ClientInfo(
            user_agent="apache-beam-%s" % apache_beam.__version__))

    self._unique_row_id = 0
    # For testing scenarios where we pass in a client we do not want a
    # randomized prefix for row IDs.
    self._row_id_prefix = '' if client else uuid.uuid4()
    self._latency_histogram_metric = Metrics.histogram(
        self.__class__,
        'latency_histogram_ms',
        LinearBucket(0, 20, 3000),
        BigQueryWrapper.HISTOGRAM_METRIC_LOGGER)

    if temp_dataset_id is not None and temp_table_ref is not None:
      raise ValueError(
          'Both a BigQuery temp_dataset_id and a temp_table_ref were specified.'
          ' Please specify only one of these.')

    if temp_dataset_id and temp_dataset_id.startswith(self.TEMP_DATASET):
      raise ValueError(
          'User provided temp dataset ID cannot start with %r' %
          self.TEMP_DATASET)

    if temp_table_ref is not None:
      self.temp_table_ref = temp_table_ref
      self.temp_dataset_id = temp_table_ref.datasetId
    else:
      self.temp_table_ref = None
      self._temporary_table_suffix = uuid.uuid4().hex
      self.temp_dataset_id = temp_dataset_id or self._get_temp_dataset()

    self.created_temp_dataset = False

View on GitHub (pinned to 12126d8942)