apache/beam · error · ValueError

User provided temp dataset ID cannot start with %r

Error message

User provided temp dataset ID cannot start with %r

What it means

Raised in BigQueryWrapper.__init__ when a user-supplied temp_dataset_id starts with the reserved prefix Beam uses for its own auto-created temp datasets (self.TEMP_DATASET, e.g. 'beam_temp_dataset_'). Beam reserves that namespace to avoid collisions with user datasets.

Solutions

Rename the dataset so it does not start with the reserved prefix (e.g. 'my_tmp_dataset').
Strip or replace the 'beam_temp_dataset_' prefix before constructing the wrapper.
Pass neither temp_dataset_id nor temp_table_ref and let Beam allocate a unique temp dataset.
If you truly need that dataset, create it outside Beam's temp mechanism and reference it as a regular dataset.

Example fix

// before
BigQueryWrapper(temp_dataset_id='beam_temp_dataset_20260912')

// after
BigQueryWrapper(temp_dataset_id='my_tmp_dataset_20260912')

Defensive patterns

Strategy: validation

Validate before calling

PREFIX = 'beam_temp_dataset_'
if temp_dataset_id and temp_dataset_id.startswith(PREFIX):
    raise ValueError(f"dataset id {temp_dataset_id!r} uses Beam's reserved prefix")

Type guard

def is_safe_temp_dataset_id(ds_id, reserved_prefix):
    return bool(ds_id) and not ds_id.startswith(reserved_prefix)

Try / catch

try:
    wrapper = BigQueryWrapper(temp_dataset_id=ds_id)
except ValueError as e:
    if 'cannot start with' in str(e):
        ds_id = 'my_' + ds_id  # or regenerate the name
        wrapper = BigQueryWrapper(temp_dataset_id=ds_id)
    else:
        raise

Prevention

Prefix custom temp datasets with your own project/team tag, not 'beam_temp_dataset_'
Never copy Beam-generated temp dataset IDs from logs into permanent config
Generate unique names with a timestamp or UUID suffix under your own prefix

When it happens

Trigger: Calling BigQueryWrapper(temp_dataset_id='beam_temp_dataset_xyz') — any dataset ID whose string starts with the reserved TEMP_DATASET prefix.

Common situations: Naming a custom temp dataset to 'blend in' with Beam's convention; copying a previously Beam-generated temp dataset ID from logs into config; template-generated options that reuse an old temp dataset name.

Understand the failure class

Background: "Must be a positive integer", "Invalid value", "Unsupported": the invalid-argument-value error family, when a library rejects the value you pass — this error's family across 35 libraries.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/768874afb9d90d6e. Report an issue: GitHub.

Appendix: source

Thrown at sdks/python/apache_beam/io/gcp/bigquery_tools.py:390

            user_agent="apache-beam-%s" % apache_beam.__version__))

    self._unique_row_id = 0
    # For testing scenarios where we pass in a client we do not want a
    # randomized prefix for row IDs.
    self._row_id_prefix = '' if client else uuid.uuid4()
    self._latency_histogram_metric = Metrics.histogram(
        self.__class__,
        'latency_histogram_ms',
        LinearBucket(0, 20, 3000),
        BigQueryWrapper.HISTOGRAM_METRIC_LOGGER)

    if temp_dataset_id is not None and temp_table_ref is not None:
      raise ValueError(
          'Both a BigQuery temp_dataset_id and a temp_table_ref were specified.'
          ' Please specify only one of these.')

    if temp_dataset_id and temp_dataset_id.startswith(self.TEMP_DATASET):
      raise ValueError(
          'User provided temp dataset ID cannot start with %r' %
          self.TEMP_DATASET)

    if temp_table_ref is not None:
      self.temp_table_ref = temp_table_ref
      self.temp_dataset_id = temp_table_ref.datasetId
    else:
      self.temp_table_ref = None
      self._temporary_table_suffix = uuid.uuid4().hex
      self.temp_dataset_id = temp_dataset_id or self._get_temp_dataset()

    self.created_temp_dataset = False

  @property
  def unique_row_id(self):
    """Returns a unique row ID (str) used to avoid multiple insertions.

    If the row ID is provided, BigQuery will make a best effort to not insert

View on GitHub (pinned to 12126d8942)