apache/beam · error · RuntimeError

Dataset %s:%s already exists so cannot be used as temporary.

Error message

Dataset %s:%s already exists so cannot be used as temporary.

What it means

Raised by `create_temporary_dataset` when the target BigQuery dataset already exists, is not a user-configured dataset, and the client did not create it as a temp dataset. Beam refuses to treat a pre-existing, unexpected dataset as temporary because deleting it at pipeline end would destroy foreign data. It is a safety check protecting against accidentally wiping someone else's dataset.

Source

Thrown at sdks/python/apache_beam/io/gcp/bigquery_tools.py:970

  @retry.with_exponential_backoff(
      num_retries=MAX_RETRIES,
      retry_filter=retry.retry_on_server_errors_and_timeout_filter)
  def create_temporary_dataset(
      self, project_id, location, labels=None, kms_key=None):
    self.get_or_create_dataset(
        project_id,
        self.temp_dataset_id,
        location=location,
        default_table_expiration_ms=_DEFAULT_TABLE_EXPIRATION_MS,
        labels=labels,
        kms_key=kms_key)

    if (project_id is not None and not self.is_user_configured_dataset() and
        not self.created_temp_dataset):
      # Unittests don't pass projectIds so they can be run without error
      # User configured datasets are allowed to pre-exist.
      raise RuntimeError(
          'Dataset %s:%s already exists so cannot be used as temporary.' %
          (project_id, self.temp_dataset_id))

  @retry.with_exponential_backoff(
      num_retries=MAX_RETRIES,
      retry_filter=retry.retry_on_server_errors_and_timeout_filter)
  def clean_up_temporary_dataset(self, project_id):
    temp_table = self._get_temp_table(project_id)
    try:
      self.client.datasets.Get(
          bigquery.BigqueryDatasetsGetRequest(
              projectId=project_id, datasetId=temp_table.datasetId))
    except HttpError as exn:
      if exn.status_code == 404:
        _LOGGER.warning(
            'Dataset %s:%s does not exist', project_id, temp_table.datasetId)
        return
      else:

View on GitHub (pinned to 12126d8942)

Solutions

Delete the stale temporary dataset (via `bq rm -r -f <project>:<dataset>` or console) and rerun the pipeline.
Investigate why previous runs didn't clean up (crash/OOM/kill) and ensure the cleanup path runs; check for orphaned datasets named beam_temp_dataset_*.
If you actually want to reuse an existing dataset, pass it as an explicitly user-configured dataset instead of relying on temp dataset logic.
Schedule periodic cleanup of orphaned temp datasets in the project.

Example fix

// shell before rerun
bq rm -r -f my-project:beam_temp_dataset_1a2b3c4d_5e6f_7890
// after
python pipeline.py --temp_dataset=...  # fresh run creates and cleans its own dataset

Defensive patterns

Strategy: try-catch

Validate before calling

from google.cloud import bigquery
client = bigquery.Client(project)
def temp_dataset_stale(dataset_id):
    try:
        client.get_dataset(dataset_id)
        return True
    except Exception:
        return False

Try / catch

try:
    setup_pipeline()
except RuntimeError as e:
    if 'already exists so cannot be used as temporary' in str(e):
        delete_dataset_stale_and_retry()
    else:
        raise

Prevention

Ensure cleanup runs in finally blocks or pipeline teardown so temp datasets are deleted.
Periodically sweep projects for orphaned beam_temp_dataset_* datasets.
Avoid hard-killing pipelines mid-run without cleanup.

When it happens

Trigger: Calling `_setup_temporary_dataset`/`create_temporary_dataset` when the dataset (e.g. 'beam_temp_dataset_<uuid>') already exists in the project from a prior run that crashed before cleanup, and `temp_dataset_id` is auto-generated.

Common situations: Rerunning a pipeline after a killed/crashed run left orphaned temp datasets; sharing a GCP project where stale temp datasets accumulate; unit-test vs production project misconfiguration.

Understand the failure class

Background: "already exists" / EEXIST / FileAlreadyExistsException: what the 'file already exists' error means and how to fix it — this error's family across 37 libraries.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/bcdb0fb23a7b0fb3. Report an issue: GitHub.