apache/beam · error · RuntimeError

Table %s:%s.%s is not empty but write disposition is WRITE_E

Error message

Table %s:%s.%s is not empty but write disposition is WRITE_EMPTY.

What it means

Raised by `get_or_create_table` when the table exists, `write_disposition` is `WRITE_EMPTY`, and `_is_table_empty` finds rows in the table. WRITE_EMPTY semantics require the destination to be empty for the write to proceed, so Beam fails loudly rather than silently appending. This is expected BigQuery write-disposition behavior surfaced as a RuntimeError.

Source

Thrown at sdks/python/apache_beam/io/gcp/bigquery_tools.py:1195

      if exn.status_code == 404:
        if create_disposition == BigQueryDisposition.CREATE_NEVER:
          raise RuntimeError(
              'Table %s:%s.%s not found but create disposition is CREATE_NEVER.'
              % (project_id, dataset_id, table_id))
      else:
        raise

    # If table exists already then handle the semantics for WRITE_EMPTY and
    # WRITE_TRUNCATE write dispositions.
    if found_table and write_disposition in (
        BigQueryDisposition.WRITE_EMPTY, BigQueryDisposition.WRITE_TRUNCATE):
      # Delete the table and recreate it (later) if WRITE_TRUNCATE was
      # specified.
      if write_disposition == BigQueryDisposition.WRITE_TRUNCATE:
        self._delete_table(project_id, dataset_id, table_id)
      elif (write_disposition == BigQueryDisposition.WRITE_EMPTY and
            not self._is_table_empty(project_id, dataset_id, table_id)):
        raise RuntimeError(
            'Table %s:%s.%s is not empty but write disposition is WRITE_EMPTY.'
            % (project_id, dataset_id, table_id))

    # Create a new table potentially reusing the schema from a previously
    # found table in case the schema was not specified.
    if schema is None and found_table is None:
      raise RuntimeError(
          'Table %s:%s.%s requires a schema. None can be inferred because the '
          'table does not exist.' % (project_id, dataset_id, table_id))
    if found_table and write_disposition != BigQueryDisposition.WRITE_TRUNCATE:
      return found_table
    else:
      created_table = None
      try:
        created_table = self._create_table(
            project_id=project_id,
            dataset_id=dataset_id,
            table_id=table_id,

View on GitHub (pinned to 12126d8942)

Solutions

Switch `write_disposition` to `WRITE_APPEND` if you want to add data to a non-empty table.
Use `WRITE_TRUNCATE` if the pipeline output should fully replace the table contents.
If the table should be empty, clear it first (bq truncate, or delete via console) and rerun.
Audit pipeline idempotency: a rerun hitting this means the job already wrote once — check for duplicate execution before retrying.

Example fix

// before
get_or_create_table(p, d, t, schema, CREATE_IF_NEEDED, BigQueryDisposition.WRITE_EMPTY)
// after
get_or_create_table(p, d, t, schema, CREATE_IF_NEEDED, BigQueryDisposition.WRITE_APPEND)

Defensive patterns

Strategy: validation

Validate before calling

from google.cloud import bigquery
client = bigquery.Client()
def table_row_count(project, dataset, table):
    t = client.get_table(f'{project}.{dataset}.{table}')
    return t.num_rows
# choose disposition based on count
disposition = WRITE_EMPTY if table_row_count(p, d, t) == 0 else WRITE_APPEND

Try / catch

try:
    get_or_create_table(p, d, t, schema, CREATE_IF_NEEDED, WRITE_EMPTY)
except RuntimeError as e:
    if 'WRITE_EMPTY' in str(e):
        raise DecisionNeeded('table non-empty; choose WRITE_APPEND or WRITE_TRUNCATE') from e
    raise

Prevention

Decide WRITE_EMPTY vs WRITE_APPEND explicitly based on table state, not by default.
Make batch jobs idempotent (WRITE_TRUNCATE or per-run destination tables).
Check num_rows before one-shot loads into shared tables.

When it happens

Trigger: Calling `get_or_create_table` (or a WriteToBigQuery sink configured with WRITE_EMPTY) against a table that already contains data — e.g. rerunning a batch job that wrote to the same table previously.

Common situations: Rerunning a backfill without clearing the destination; writing periodic batches to a table intended for one-shot load; misunderstanding WRITE_EMPTY (fails if non-empty) vs WRITE_TRUNCATE (replaces) vs WRITE_APPEND.

Understand the failure class

Background: "Invalid state transition" errors: "status must be X, actually Y", "already rejected/charging/uninstalled", "cannot ... while running" — what they mean when a library rejects your call — this error's family across 31 libraries.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/435acb7ca25f0169. Report an issue: GitHub.