apache/beam · error · ValueError

Expected a table reference (PROJECT:DATASET.TABLE or…

Error message

Expected a table reference (PROJECT:DATASET.TABLE or DATASET.TABLE) instead of %s.

What it means

Raised by parse_table_reference when the table string does not fullmatch the pattern (optional PROJECT: then DATASET.TABLE, where DATASET may itself contain a project-qualified form). Beam cannot split the string into projectId/datasetId/tableId and refuses to build a TableReference.

Solutions

Format the string as 'PROJECT:DATASET.TABLE' or at minimum 'DATASET.TABLE'.
Strip whitespace/quotes from the value before passing it.
If project is implied, use 'DATASET.TABLE' and let Beam fill the default project.
Alternatively construct a TableReference programmatically (bigquery.TableReference) instead of parsing a string.

Example fix

// before
parse_table_reference('mytable')

// after
parse_table_reference('my_project:my_dataset.mytable')

Defensive patterns

Strategy: validation

Validate before calling

import re
_PATTERN = re.compile(r'((?P<project>[a-z0-9_.\-]+)[:\.])?(?P<dataset>[a-zA-Z0-9_]+)\.(?P<table>[a-zA-Z0-9_]+)$')
def validate_table_spec(s):
    if not _PATTERN.fullmatch(s.strip()):
        raise ValueError(f"table ref {s!r} must be PROJECT:DATASET.TABLE or DATASET.TABLE")

Type guard

def is_valid_table_reference(s):
    parts = s.strip().split('.')
    return len(parts) in (2, 3) and all(parts)

Try / catch

try:
    ref = parse_table_reference(table_spec)
except ValueError as e:
    raise PipelineConfigError(f"bad table spec {table_spec!r}; use PROJECT:DATASET.TABLE") from e

Prevention

Build table specs from components: f'{project}:{dataset}.{table}'
Strip whitespace/quotes from config values
Never paste console URLs as table references

When it happens

Trigger: Passing strings like 'mytable' (no dataset), 'project:table' (no dataset), 'a.b.c.d' (too many parts), or strings with invalid characters/spaces to bigquery_io.parse_table_reference or APIs that call it (e.g. temp table resolution in BigQuerySink).

Common situations: Omitting the dataset when only a table name is known; using ':' instead of '.' between dataset and table; URI-encoded or whitespace-polluted table IDs from config files; accidentally passing a full BQ console URL instead of the table spec.

Understand the failure class

Background: "Invalid ... format", "must be in format X", "does not look like a ..." — invalid argument format errors across CLI tools and libraries — this error's family across 17 libraries.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/563775568df3d5b2. Report an issue: GitHub.

Appendix: source

Thrown at sdks/python/apache_beam/io/gcp/bigquery_tools.py:293

        projectId=table.projectId,
        datasetId=table.datasetId,
        tableId=table.tableId)
  elif callable(table):
    return table
  elif isinstance(table, value_provider.ValueProvider):
    return table

  table_reference = TableReference()
  # If dataset argument is not specified, the expectation is that the
  # table argument will contain a full table reference instead of just a
  # table name.
  if dataset is None:
    pattern = (
        f'((?P<project>{_PROJECT_PATTERN})[:\\.])?'
        f'(?P<dataset>{_DATASET_PATTERN})\\.(?P<table>{_TABLE_PATTERN})')
    match = regex.fullmatch(pattern, table)
    if not match:
      raise ValueError(
          'Expected a table reference (PROJECT:DATASET.TABLE or '
          'DATASET.TABLE) instead of %s.' % table)
    table_reference.projectId = match.group('project')
    table_reference.datasetId = match.group('dataset')
    table_reference.tableId = match.group('table')
  else:
    table_reference.projectId = project
    table_reference.datasetId = dataset
    table_reference.tableId = table
  return table_reference


# -----------------------------------------------------------------------------
# BigQueryWrapper.


def _build_job_labels(input_labels):
  """Builds job label protobuf structure."""

View on GitHub (pinned to 12126d8942)