{"record":{"id":"bcdb0fb23a7b0fb3","repo":"apache/beam","slug":"dataset-s-s-already-exists-so-cannot-be-used-as-temporary","errorCode":null,"errorMessage":"Dataset %s:%s already exists so cannot be used as temporary.","messagePattern":"Dataset (.+?):(.+?) already exists so cannot be used as temporary\\.","errorType":"exception","errorClass":"RuntimeError","httpStatus":null,"severity":"error","filePath":"sdks/python/apache_beam/io/gcp/bigquery_tools.py","lineNumber":970,"sourceCode":"\n  @retry.with_exponential_backoff(\n      num_retries=MAX_RETRIES,\n      retry_filter=retry.retry_on_server_errors_and_timeout_filter)\n  def create_temporary_dataset(\n      self, project_id, location, labels=None, kms_key=None):\n    self.get_or_create_dataset(\n        project_id,\n        self.temp_dataset_id,\n        location=location,\n        default_table_expiration_ms=_DEFAULT_TABLE_EXPIRATION_MS,\n        labels=labels,\n        kms_key=kms_key)\n\n    if (project_id is not None and not self.is_user_configured_dataset() and\n        not self.created_temp_dataset):\n      # Unittests don't pass projectIds so they can be run without error\n      # User configured datasets are allowed to pre-exist.\n      raise RuntimeError(\n          'Dataset %s:%s already exists so cannot be used as temporary.' %\n          (project_id, self.temp_dataset_id))\n\n  @retry.with_exponential_backoff(\n      num_retries=MAX_RETRIES,\n      retry_filter=retry.retry_on_server_errors_and_timeout_filter)\n  def clean_up_temporary_dataset(self, project_id):\n    temp_table = self._get_temp_table(project_id)\n    try:\n      self.client.datasets.Get(\n          bigquery.BigqueryDatasetsGetRequest(\n              projectId=project_id, datasetId=temp_table.datasetId))\n    except HttpError as exn:\n      if exn.status_code == 404:\n        _LOGGER.warning(\n            'Dataset %s:%s does not exist', project_id, temp_table.datasetId)\n        return\n      else:","sourceCodeStart":952,"sourceCodeEnd":988,"githubUrl":"https://github.com/apache/beam/blob/12126d8942aaf848030c478b4c6a28c6af861c66/sdks/python/apache_beam/io/gcp/bigquery_tools.py#L952-L988","documentation":"Raised by `create_temporary_dataset` when the target BigQuery dataset already exists, is not a user-configured dataset, and the client did not create it as a temp dataset. Beam refuses to treat a pre-existing, unexpected dataset as temporary because deleting it at pipeline end would destroy foreign data. It is a safety check protecting against accidentally wiping someone else's dataset.","triggerScenarios":"Calling `_setup_temporary_dataset`/`create_temporary_dataset` when the dataset (e.g. 'beam_temp_dataset_<uuid>') already exists in the project from a prior run that crashed before cleanup, and `temp_dataset_id` is auto-generated.","commonSituations":"Rerunning a pipeline after a killed/crashed run left orphaned temp datasets; sharing a GCP project where stale temp datasets accumulate; unit-test vs production project misconfiguration.","solutions":["Delete the stale temporary dataset (via `bq rm -r -f <project>:<dataset>` or console) and rerun the pipeline.","Investigate why previous runs didn't clean up (crash/OOM/kill) and ensure the cleanup path runs; check for orphaned datasets named beam_temp_dataset_*.","If you actually want to reuse an existing dataset, pass it as an explicitly user-configured dataset instead of relying on temp dataset logic.","Schedule periodic cleanup of orphaned temp datasets in the project."],"exampleFix":"// shell before rerun\nbq rm -r -f my-project:beam_temp_dataset_1a2b3c4d_5e6f_7890\n// after\npython pipeline.py --temp_dataset=...  # fresh run creates and cleans its own dataset","handlingStrategy":"try-catch","validationCode":"from google.cloud import bigquery\nclient = bigquery.Client(project)\ndef temp_dataset_stale(dataset_id):\n    try:\n        client.get_dataset(dataset_id)\n        return True\n    except Exception:\n        return False","typeGuard":null,"tryCatchPattern":"try:\n    setup_pipeline()\nexcept RuntimeError as e:\n    if 'already exists so cannot be used as temporary' in str(e):\n        delete_dataset_stale_and_retry()\n    else:\n        raise","preventionTips":["Ensure cleanup runs in finally blocks or pipeline teardown so temp datasets are deleted.","Periodically sweep projects for orphaned beam_temp_dataset_* datasets.","Avoid hard-killing pipelines mid-run without cleanup."],"tags":["bigquery","python","gcp","temporary-resources"],"backgroundTag":"file-already-exists","analyzedSha":"12126d8942aaf848030c478b4c6a28c6af861c66","analyzedAt":"2026-09-13T01:50:10.254Z","contentChangedAt":"2026-09-13T01:50:10.254Z","schemaVersion":2},"datasetVersion":"2026-09-14T21:17:11.552Z"}