{"record":{"id":"76f4fedc898f2a34","repo":"apache/beam","slug":"when-using-query-in-readfrombigquery-yaml-transform-schema","errorCode":null,"errorMessage":"When using 'query' in ReadFromBigQuery YAML transform, 'schema' is required to define the output row structure.","messagePattern":"When using 'query' in ReadFromBigQuery YAML transform, 'schema' is required to define the output row structure\\.","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"sdks/python/apache_beam/yaml/yaml_io.py","lineNumber":136,"sourceCode":"    row_restriction (str): Optional SQL text filtering statement, similar to a\n      WHERE clause in a query. Aggregates are not supported. Restricted to a\n      maximum length for 1 MB.\n    selected_fields (list[str]): Optional List of names of the fields in the\n      table that should be read. If empty, all fields will be read. If the\n      specified field is a nested field, all the sub-fields in the field will be\n      selected. The output field order is unrelated to the order of fields\n      given here.\n    schema (dict): Required when query is set. A BigQuery schema describing\n      the query result columns, e.g.\n      ``{'fields': [{'name': 'col', 'type': 'STRING', 'mode': 'NULLABLE'}]}``.\n      Not applicable when reading from a table (schema is auto-derived).\n  \"\"\"\n  if query is None:\n    assert table is not None\n  else:\n    assert table is None and row_restriction is None and fields is None\n    if schema is None:\n      raise ValueError(\n          \"When using 'query' in ReadFromBigQuery YAML transform, \"\n          \"'schema' is required to define the output row structure.\")\n  return ReadFromBigQuery(\n      query=query,\n      table=table,\n      row_restriction=row_restriction,\n      selected_fields=fields,\n      method='DIRECT_READ',\n      output_type='BEAM_ROW',\n      query_output_schema=schema)\n\n\ndef write_to_bigquery(\n    table: str,\n    *,\n    create_disposition: Optional[str] = BigQueryDisposition.CREATE_IF_NEEDED,\n    write_disposition: Optional[str] = BigQueryDisposition.WRITE_APPEND,\n    error_handling=None):","sourceCodeStart":118,"sourceCodeEnd":154,"githubUrl":"https://github.com/apache/beam/blob/12126d8942aaf848030c478b4c6a28c6af861c66/sdks/python/apache_beam/yaml/yaml_io.py#L118-L154","documentation":"In the Beam YAML read_from_bigquery wrapper, when a 'query' is supplied (instead of a table), the output row structure cannot be inferred from a table, so a 'schema' argument is mandatory. If query is given and schema is None, the wrapper raises ValueError explaining that the schema is required to define the output rows.","triggerScenarios":"Calling read_from_bigquery with query=<sql> and schema=None (and table=None). The wrapper asserts no table/row_restriction/fields alongside a query, then requires schema for the query path.","commonSituations":"Using a custom SQL query in a Beam YAML pipeline and omitting the schema field; migrating a table-based read to a query-based one and removing the schema.","solutions":["Provide the 'schema' argument (JSON schema string/dict) describing the query's output columns.","Alternatively read from a table directly (table=...) so the schema can be derived.","Verify the schema matches the SELECT column names/types of the query."],"exampleFix":"// before\n- type: ReadFromBigQuery\n  query: 'SELECT id, name FROM dataset.users'\n// after\n- type: ReadFromBigQuery\n  query: 'SELECT id, name FROM dataset.users'\n  schema: '{\"fields\": [{\"name\": \"id\", \"type\": \"INT64\"}, {\"name\": \"name\", \"type\": \"STRING\"}]}'","handlingStrategy":"validation","validationCode":"if query is not None and not schema:\n    raise ValueError(\"read_from_bigquery with 'query' requires 'schema'\")","typeGuard":null,"tryCatchPattern":"try:\n    read_from_bigquery(query=q, schema=s)\nexcept ValueError as e:\n    if \"'schema' is required\" in str(e):\n        add_schema_from_bq_information_schema(q)","preventionTips":["Always pair query-based BQ reads with an explicit schema in YAML configs.","Derive schema from BigQuery INFORMATION_SCHEMA during config generation.","Validate transform args in a unit test before submitting to a runner."],"tags":["python","apache-beam","yaml","bigquery","missing-argument"],"backgroundTag":"missing-required-config-field","analyzedSha":"12126d8942aaf848030c478b4c6a28c6af861c66","analyzedAt":"2026-09-13T01:50:10.254Z","contentChangedAt":"2026-09-13T01:50:10.254Z","schemaVersion":2},"datasetVersion":"2026-09-14T16:17:12.679Z"}