apache/beam · error · ValueError

Encountered an empty schema

Error message

Encountered an empty schema

What it means

generate_user_type_from_bq_schema converts a BigQuery table schema into a Python user type for typed reads. It first normalizes the schema via get_dict_table_schema; if the result is an empty dict (no fields), there is nothing to build a row type from, so it raises this ValueError.

Source

Thrown at sdks/python/apache_beam/io/gcp/bigquery_schema_tools.py:78

    type_overrides=None) -> type:
  """Convert a schema of type TableSchema into a pcollection element.

  Args:
    the_table_schema: A BQ schema of type TableSchema
    selected_fields: if not None, the subset of fields to consider
    type_overrides: Optional mapping of BigQuery type names (uppercase)
      to Python types. These override the default mappings in
      BIG_QUERY_TO_PYTHON_TYPES. For example:
      ``{'DATE': datetime.date, 'JSON': dict}``

  Returns:
    type: type that can be used to work with pCollections.
  """
  effective_types = {**BIG_QUERY_TO_PYTHON_TYPES, **(type_overrides or {})}
  the_schema = beam.io.gcp.bigquery_tools.get_dict_table_schema(
      the_table_schema)
  if the_schema == {}:
    raise ValueError("Encountered an empty schema")
  field_names_and_types = []
  for field in the_schema['fields']:
    if selected_fields is not None and field['name'] not in selected_fields:
      continue
    if field['type'] in effective_types:
      typ = bq_field_to_type(field['type'], field['mode'], type_overrides)
    else:
      raise ValueError(
          f"Encountered "
          f"an unsupported type: {field['type']!r}")
    field_names_and_types.append((field['name'], typ))
  sample_schema = beam.typehints.schemas.named_fields_to_schema(
      field_names_and_types)
  usertype = beam.typehints.schemas.named_tuple_from_schema(sample_schema)
  return usertype


def bq_field_to_type(field, mode, type_overrides=None):

View on GitHub (pinned to 12126d8942)

Solutions

1. Verify the source table/query schema is non-empty before calling; run the query or table.get and check schema.fields.
2. If constructing TableSchema manually, populate at least one TableFieldSchema.
3. Pass an explicit the_table_schema from a known-good table instead of a dynamically fetched empty one.

Example fix

// before
if not the_table_schema:
  the_table_schema = TableSchema()
// after
if not the_table_schema or not the_table_schema.fields:
  raise ValueError('table schema must define at least one field')

d = beam.io.gcp.bigquery_tools.get_dict_table_schema(the_table_schema)
if not d or not d.get('fields'):
    raise ValueError('table schema has no fields; cannot build user type')

def has_fields(table_schema) -> bool:
    return bool(table_schema and getattr(table_schema, 'fields', None))

try:
    user_type = bigquery_schema_tools.generate_user_type_from_bq_schema(schema)
except ValueError as e:
    if 'empty schema' in str(e):
        fetch_schema_from_table()

When it happens

Trigger: Calling generate_user_type_from_bq_schema (or convert_to_usertype) with a table schema whose fields list is empty, or a TableSchema that serializes to {}.

Common situations: Reading from a query returning no columns (e.g. SELECT from an empty-defined view), passing an unpopulated TableSchema() constructed programmatically, or a metadata fetch returning no fields due to permission/table-not-found issues.

Understand the failure class

Background: Schema validation failed / invalid input schema: payload rejected because its shape doesn't match the expected schema — this error's family across 28 libraries.

Related errors

• Unknown logical type " + identifier
• Nested ROW missing row schema
• Unknown mode %s
• Unknown type %s
• Schema field not found: %s

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/25534dbd07ac4cb3. Report an issue: GitHub.