apache/beam · error · ValueError

Invalid BigQuery table name

Error message

Invalid BigQuery table name: %s 
See https://cloud.google.com/bigquery/docs/tables#table_naming

What it means

Raised by `_create_table` in `apache_beam/io/gcp/bigquery_tools.py` when the given table_id does not fullmatch `_TABLE_PATTERN` (checked with regex.ASCII). BigQuery table names must match `[a-zA-Z0-9_]+` (with optional dataset/project qualifiers), so illegal characters like dashes, dots, or unicode cause this ValueError before any API call is made. It is a client-side preflight guard so invalid names fail fast.

Solutions

1. Sanitize the table_id: replace invalid characters with '_' (e.g. re.sub(r'[^a-zA-Z0-9_]', '_', table_id)) before calling the API.
2. Verify you are passing only the table identifier (or a correctly formed 'project:dataset.table') and that dataset/project qualifiers use allowed characters.
3. If a hyphenated source name must be preserved, map it to a deterministic underscore-based name and keep the original as a label/metadata.
4. Test the name with re.fullmatch on the BigQuery pattern locally before launching the pipeline.

Example fix

// before
name = f'{tenant}-{date}'
beam.io.WriteToBigQuery(f'dataset.{name}', ...)
// after
import re
name = re.sub(r'[^a-zA-Z0-9_]', '_', f'{tenant}_{date}')
beam.io.WriteToBigQuery(f'dataset.{name}', ...)

import re
_TABLE_PATTERN = r'((?:[a-zA-Z0-9-]+[.:])?)([a-zA-Z0-9_]+)\.([a-zA-Z0-9_]+)'
def valid_table_id(table_id):
    return re.fullmatch(_TABLE_PATTERN, table_id, re.ASCII) is not None
# call before get_or_create_table / WriteToBigQuery

def is_valid_table_name(table_id):
    return isinstance(table_id, str) and bool(re.fullmatch(r'([\w-]+[.:])?\w+\.\w+', table_id, re.ASCII))

When it happens

Trigger: Calling `get_or_create_table` (or `WriteToBigQuery` with a computed table name) where table_id contains characters outside [a-zA-Z0-9_], e.g. 'my-table', 'table-2024-01', or a name built from user input with spaces/hyphens.

Common situations: Dynamically constructing table names from dates, filenames, or user/tenant IDs that contain hyphens or other punctuation; copying PostgreSQL/MySQL table names that allow hyphens; accidentally passing a full 'project:dataset.table' string where only the bare table id is expected with different validation.

Understand the failure class

Background: "invalid id" errors: invalid identifier format — why libraries reject IDs before lookup, and how to fix them — this error's family across 37 libraries.

Related errors

• buffer_sec must be >= 0, got
• change_function must be 'CHANGES' or 'APPENDS', got
• Invalid create disposition
• Invalid schema update option
• Invalid write disposition

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/832691721aad71e0. Report an issue: GitHub.

Appendix: source

Thrown at sdks/python/apache_beam/io/gcp/bigquery_tools.py:829

    Raises:
      HttpError: if lookup failed.
    """
    request = bigquery.BigqueryTablesGetRequest(
        projectId=project_id, datasetId=dataset_id, tableId=table_id)
    response = self.client.tables.Get(request)
    return response

  def _create_table(
      self,
      project_id,
      dataset_id,
      table_id,
      schema,
      additional_parameters=None):

    valid_tablename = regex.fullmatch(_TABLE_PATTERN, table_id, regex.ASCII)
    if not valid_tablename:
      raise ValueError(
          'Invalid BigQuery table name: %s \n'
          'See https://cloud.google.com/bigquery/docs/tables#table_naming' %
          table_id)

    additional_parameters = additional_parameters or {}
    table = bigquery.Table(
        tableReference=TableReference(
            projectId=project_id, datasetId=dataset_id, tableId=table_id),
        schema=schema,
        **additional_parameters)
    request = bigquery.BigqueryTablesInsertRequest(
        projectId=project_id, datasetId=dataset_id, table=table)
    response = self.client.tables.Insert(request)
    _LOGGER.debug("Created the table with id %s", table_id)
    # The response is a bigquery.Table instance.
    return response

  @retry.with_exponential_backoff(

View on GitHub (pinned to 12126d8942)