apache/beam · error · ValueError

Dataset {} does not exist in your project. You have to creat

Error message

Dataset {} does not exist in your project. You have to create table first.

What it means

_get_dataset looks up the configured BigQuery dataset and re-raises google.cloud.exceptions.NotFound as a ValueError when the dataset does not exist in the project. The client intentionally does not auto-create datasets; the message tells you to create the dataset (and table) beforehand.

Source

Thrown at sdks/python/apache_beam/testing/load_tests/load_test_metrics_utils.py:510

    try:
      self._bq_table = self._client.get_table(table_ref)
    except NotFound:
      table = bigquery.Table(table_ref, schema=bq_schemas)
      self._bq_table = self._client.create_table(table)

  def _update_schema(self):
    table_schema = self._bq_table.schema
    if self.schema and len(table_schema) != self.schema:
      self._bq_table.schema = self._prepare_schema()
      self._bq_table = self._client.update_table(self._bq_table, ["schema"])

  def _get_dataset(self, dataset_name):
    bq_dataset_ref = self._client.dataset(dataset_name)
    try:
      bq_dataset = self._client.get_dataset(bq_dataset_ref)
    except NotFound:
      raise ValueError(
          'Dataset {} does not exist in your project. '
          'You have to create table first.'.format(dataset_name))
    return bq_dataset

  def save(self, results):
    # update schema if needed
    self._update_schema()
    return self._client.insert_rows(self._bq_table, results)


class InfluxDBMetricsPublisherOptions(object):
  def __init__(
      self,
      measurement: str,
      db_name: str,
      hostname: str,
      user: Optional[str] = None,
      password: Optional[str] = None):

View on GitHub (pinned to 12126d8942)

Solutions

Create the dataset in the target project, e.g. `bq mk <dataset>`
Verify the dataset name spelling in your configuration
Check which project the BigQuery client authenticates to and pass the correct project if needed

Example fix

// before
client = BigQueryClient(schemas, dataset='load_test_metrics')  # dataset missing
// after: run `bq mk load_test_metrics` first, then construct the client

Defensive patterns

Strategy: try-catch

Validate before calling

from google.cloud import bigquery
client = bigquery.Client()
dataset_ok = dataset_name in [d.dataset_id for d in client.list_datasets()]

Try / catch

try:
    publisher = BigQueryClient(schemas, dataset_name, namespace)
except ValueError as e:
    if 'does not exist' in str(e):
        subprocess.run(['bq', 'mk', dataset_name], check=True)
        publisher = BigQueryClient(schemas, dataset_name, namespace)

Prevention

Provision the dataset before launching load tests
Verify the project your credentials default to (gcloud config get-value project)
Pin dataset names in test configuration

When it happens

Trigger: Constructing BigQueryClient pointed at a dataset name that does not exist in the GCP project; typo'd dataset name; wrong project configured on the BigQuery client credentials.

Common situations: Running load tests before provisioning the metrics dataset; dataset exists in another project than the one the credentials default to; dataset renamed or deleted.

Understand the failure class

Background: 'Could not be found', 'does not exist', 'not found in database': the resource-not-found family when an ID, slug, key, or URI lookup comes back empty — this error's family across 20 libraries.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/c3631b6336e71b90. Report an issue: GitHub.