apache/beam · error · ValueError

When using 'query' in ReadFromBigQuery YAML transform, 'sche

Error message

When using 'query' in ReadFromBigQuery YAML transform, 'schema' is required to define the output row structure.

What it means

In the Beam YAML read_from_bigquery wrapper, when a 'query' is supplied (instead of a table), the output row structure cannot be inferred from a table, so a 'schema' argument is mandatory. If query is given and schema is None, the wrapper raises ValueError explaining that the schema is required to define the output rows.

Source

Thrown at sdks/python/apache_beam/yaml/yaml_io.py:136

    row_restriction (str): Optional SQL text filtering statement, similar to a
      WHERE clause in a query. Aggregates are not supported. Restricted to a
      maximum length for 1 MB.
    selected_fields (list[str]): Optional List of names of the fields in the
      table that should be read. If empty, all fields will be read. If the
      specified field is a nested field, all the sub-fields in the field will be
      selected. The output field order is unrelated to the order of fields
      given here.
    schema (dict): Required when query is set. A BigQuery schema describing
      the query result columns, e.g.
      ``{'fields': [{'name': 'col', 'type': 'STRING', 'mode': 'NULLABLE'}]}``.
      Not applicable when reading from a table (schema is auto-derived).
  """
  if query is None:
    assert table is not None
  else:
    assert table is None and row_restriction is None and fields is None
    if schema is None:
      raise ValueError(
          "When using 'query' in ReadFromBigQuery YAML transform, "
          "'schema' is required to define the output row structure.")
  return ReadFromBigQuery(
      query=query,
      table=table,
      row_restriction=row_restriction,
      selected_fields=fields,
      method='DIRECT_READ',
      output_type='BEAM_ROW',
      query_output_schema=schema)


def write_to_bigquery(
    table: str,
    *,
    create_disposition: Optional[str] = BigQueryDisposition.CREATE_IF_NEEDED,
    write_disposition: Optional[str] = BigQueryDisposition.WRITE_APPEND,
    error_handling=None):

View on GitHub (pinned to 12126d8942)

Solutions

Provide the 'schema' argument (JSON schema string/dict) describing the query's output columns.
Alternatively read from a table directly (table=...) so the schema can be derived.
Verify the schema matches the SELECT column names/types of the query.

Example fix

// before
- type: ReadFromBigQuery
  query: 'SELECT id, name FROM dataset.users'
// after
- type: ReadFromBigQuery
  query: 'SELECT id, name FROM dataset.users'
  schema: '{"fields": [{"name": "id", "type": "INT64"}, {"name": "name", "type": "STRING"}]}'

Defensive patterns

Strategy: validation

Validate before calling

if query is not None and not schema:
    raise ValueError("read_from_bigquery with 'query' requires 'schema'")

Try / catch

try:
    read_from_bigquery(query=q, schema=s)
except ValueError as e:
    if "'schema' is required" in str(e):
        add_schema_from_bq_information_schema(q)

Prevention

Always pair query-based BQ reads with an explicit schema in YAML configs.
Derive schema from BigQuery INFORMATION_SCHEMA during config generation.
Validate transform args in a unit test before submitting to a runner.

When it happens

Trigger: Calling read_from_bigquery with query=<sql> and schema=None (and table=None). The wrapper asserts no table/row_restriction/fields alongside a query, then requires schema for the query path.

Common situations: Using a custom SQL query in a Beam YAML pipeline and omitting the schema field; migrating a table-based read to a query-based one and removing the schema.

Understand the failure class

Background: "is required", "must be set", "missing required field": configuration validation errors across open-source libraries — this error's family across 36 libraries.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/76f4fedc898f2a34. Report an issue: GitHub.