apache/beam · error · RuntimeError

Table %s:%s.%s not found but create disposition is CREATE_NE

Error message

Table %s:%s.%s not found but create disposition is CREATE_NEVER.

What it means

Raised by `get_or_create_table` when `get_table` returns HTTP 404 (table missing) while `create_disposition` is `CREATE_NEVER`. Because CREATE_NEVER forbids creating the table, Beam propagates a RuntimeError instead of silently creating one. The write can never proceed under this configuration.

Source

Thrown at sdks/python/apache_beam/io/gcp/bigquery_tools.py:1179

    Returns:
      A bigquery.Table instance if table was found or created.

    Raises:
      `RuntimeError`: For various mismatches between the state of the table and
        the create/write dispositions passed in. For example if the table is not
        empty and WRITE_EMPTY was specified then an error will be raised since
        the table was expected to be empty.
    """
    from apache_beam.io.gcp.bigquery import BigQueryDisposition

    found_table = None
    try:
      found_table = self.get_table(project_id, dataset_id, table_id)
    except HttpError as exn:
      if exn.status_code == 404:
        if create_disposition == BigQueryDisposition.CREATE_NEVER:
          raise RuntimeError(
              'Table %s:%s.%s not found but create disposition is CREATE_NEVER.'
              % (project_id, dataset_id, table_id))
      else:
        raise

    # If table exists already then handle the semantics for WRITE_EMPTY and
    # WRITE_TRUNCATE write dispositions.
    if found_table and write_disposition in (
        BigQueryDisposition.WRITE_EMPTY, BigQueryDisposition.WRITE_TRUNCATE):
      # Delete the table and recreate it (later) if WRITE_TRUNCATE was
      # specified.
      if write_disposition == BigQueryDisposition.WRITE_TRUNCATE:
        self._delete_table(project_id, dataset_id, table_id)
      elif (write_disposition == BigQueryDisposition.WRITE_EMPTY and
            not self._is_table_empty(project_id, dataset_id, table_id)):
        raise RuntimeError(
            'Table %s:%s.%s is not empty but write disposition is WRITE_EMPTY.'
            % (project_id, dataset_id, table_id))

View on GitHub (pinned to 12126d8942)

Solutions

Create the table before the pipeline runs, or change `create_disposition` to `CREATE_IF_NEEDED` and supply a schema.
Verify the project_id, dataset_id, and table_id spelling against the actual BigQuery resource (bq show <project>:<dataset>.<table>).
Check pipeline options/environment — you may be pointed at the wrong project (dev vs prod).
If the table is created by an upstream job, add a dependency/precedence guarantee so it exists before this write starts.

Example fix

// before
table_info = bq_tools.get_or_create_table(project, dataset, table, schema, BigQueryDisposition.CREATE_NEVER, BigQueryDisposition.WRITE_APPEND)
// after
table_info = bq_tools.get_or_create_table(project, dataset, table, schema, BigQueryDisposition.CREATE_IF_NEEDED, BigQueryDisposition.WRITE_APPEND)

Defensive patterns

Strategy: validation

Validate before calling

from google.api_core.exceptions import NotFound
from google.cloud import bigquery
client = bigquery.Client()
def table_exists(project, dataset, table):
    try:
        client.get_table(f'{project}.{dataset}.{table}')
        return True
    except NotFound:
        return False
# require table_exists(...) true before using CREATE_NEVER

Try / catch

try:
    get_or_create_table(p, d, t, schema, CREATE_NEVER, WRITE_APPEND)
except RuntimeError as e:
    if 'CREATE_NEVER' in str(e):
        log.error('destination table %s:%s.%s missing', p, d, t)
    raise

Prevention

Use CREATE_NEVER only when an upstream process guarantees table creation.
Verify table identifiers against `bq show` before launching batch jobs.
Check --project / environment configuration to avoid dev/prod mix-ups.

When it happens

Trigger: Writing to BigQuery with `create_disposition=BigQueryDisposition.CREATE_NEVER` while the target table `project:dataset.table` does not exist (typo, wrong dataset, table created in a different project).

Common situations: Typo'd table or dataset names in pipeline options; pointing a staging pipeline at a fresh project without migrations; case-sensitivity confusion between projects; environment misconfig (dev vs prod).

Understand the failure class

Background: 'Could not be found', 'does not exist', 'not found in database': the resource-not-found family when an ID, slug, key, or URI lookup comes back empty — this error's family across 20 libraries.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/35e7110bf95ede6b. Report an issue: GitHub.