apache/beam · error · RuntimeError

Table %s:%s.%s requires a schema. None can be inferred becau

Error message

Table %s:%s.%s requires a schema. None can be inferred because the table does not exist.

What it means

Raised by `get_or_create_table` when no schema was supplied and the table does not exist, so Beam has no schema to create the table with and cannot infer one. Creating a BigQuery table requires a schema; with CREATE_IF_NEEDED and a missing table, a `schema=None` argument makes the operation impossible.

Source

Thrown at sdks/python/apache_beam/io/gcp/bigquery_tools.py:1202

    # If table exists already then handle the semantics for WRITE_EMPTY and
    # WRITE_TRUNCATE write dispositions.
    if found_table and write_disposition in (
        BigQueryDisposition.WRITE_EMPTY, BigQueryDisposition.WRITE_TRUNCATE):
      # Delete the table and recreate it (later) if WRITE_TRUNCATE was
      # specified.
      if write_disposition == BigQueryDisposition.WRITE_TRUNCATE:
        self._delete_table(project_id, dataset_id, table_id)
      elif (write_disposition == BigQueryDisposition.WRITE_EMPTY and
            not self._is_table_empty(project_id, dataset_id, table_id)):
        raise RuntimeError(
            'Table %s:%s.%s is not empty but write disposition is WRITE_EMPTY.'
            % (project_id, dataset_id, table_id))

    # Create a new table potentially reusing the schema from a previously
    # found table in case the schema was not specified.
    if schema is None and found_table is None:
      raise RuntimeError(
          'Table %s:%s.%s requires a schema. None can be inferred because the '
          'table does not exist.' % (project_id, dataset_id, table_id))
    if found_table and write_disposition != BigQueryDisposition.WRITE_TRUNCATE:
      return found_table
    else:
      created_table = None
      try:
        created_table = self._create_table(
            project_id=project_id,
            dataset_id=dataset_id,
            table_id=table_id,
            schema=schema or found_table.schema,
            additional_parameters=additional_create_parameters)
      except HttpError as exn:
        if exn.status_code == 409:
          _LOGGER.debug(
              'Skipping Creation. Table %s:%s.%s already exists.' %
              (project_id, dataset_id, table_id))

View on GitHub (pinned to 12126d8942)

Solutions

1. Pass an explicit schema (apache_beam.io.gcp.internal.clients.bigquery TableSchema or dict) when writing to a table that may not exist.
2. Load the schema from a JSON file via bigquery_tools.parse_table_schema_from_json and verify it isn't None before calling.
3. Create the table in advance (bq mk --schema=...) so the existing table's schema can be reused.
4. Add a guard: if schema is None, fail fast in your own code with a clear message about which destination lacks a schema.

Example fix

// before
schema = get_schema_option()  # may be None
get_or_create_table(p, d, t, schema, CREATE_IF_NEEDED, WRITE_APPEND)
// after
schema = get_schema_option() or parse_table_schema_from_json('schema.json')
assert schema is not None, 'schema required for non-existent table'
get_or_create_table(p, d, t, schema, CREATE_IF_NEEDED, WRITE_APPEND)

def require_schema(schema, dest):
    if schema is None:
        raise ValueError(f'A schema is required for destination {dest}; table may not exist yet.')
    return schema
# call before get_or_create_table with CREATE_IF_NEEDED

def has_schema(schema):
    return schema is not None and getattr(schema, 'fields', None)

try:
    get_or_create_table(p, d, t, schema, CREATE_IF_NEEDED, WRITE_APPEND)
except RuntimeError as e:
    if 'requires a schema' in str(e):
        schema = load_schema_from_config(dest)
        get_or_create_table(p, d, t, schema, CREATE_IF_NEEDED, WRITE_APPEND)
    else:
        raise

When it happens

Trigger: Calling `get_or_create_table(project, dataset, table_id, schema=None, ...)` where the table is absent, expecting the schema to be discovered from the existing table; passing a schema argument that resolves to None (e.g. an unset pipeline option or an empty variable).

Common situations: Copy code written for existing tables applied to new tables; schema loaded conditionally and silently None when file/config missing; dynamic destinations where schema lookup returns None for unseen destination.

Understand the failure class

Background: "missing required argument" and "the following required arguments were not provided": what required-argument errors mean and how to fix them — this error's family across 20 libraries.

Related errors

• We have observed a row of size %s bytes exceeding the BigQue
• Exception while trying to retrieve schema
• Failed to patch table schema.
• Schema must be set for table " + tableReference + " when wri
• Unexpected schema argument: %s.

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/969e2c618b58b8d8. Report an issue: GitHub.