apache/beam · error · ValueError

A schema must be provided when writing to BigQuery using…

Error message

A schema must be provided when writing to BigQuery using Avro based file loads

What it means

With FILE_LOADS and Avro temp file format, expand() requires a schema to be present; schema=None means Beam cannot build the Avro load job. This is the companion check to auto-detection: with Avro, some explicit schema must be supplied.

Solutions

1. Pass an explicit schema argument (BigQuery schema string, dict, or callable).
2. Set temp_file_format='NEWLINE_DELIMITED_JSON' to allow schema auto-detection from untyped data.
3. Feed a typed PCollection (Beam schema rows) so a schema can be derived.

Example fix

// before
WriteToBigQuery(table='proj:ds.tbl', method='FILE_LOADS', temp_file_format='AVRO')
// after
WriteToBigQuery(table='proj:ds.tbl', method='FILE_LOADS', temp_file_format='AVRO', schema={'fields': [{'name': 'id', 'type': 'STRING'}]})

if temp_file_format == 'AVRO' and schema is None:
    raise ValueError('schema is required for Avro file loads')

When it happens

Trigger: WriteToBigQuery(method=FILE_LOADS, schema=None, temp_file_format=Avro) expanded on input whose element type cannot yield a schema.

Common situations: Forgetting to pass schema when the input is plain JSON dicts; refactoring removed the schema parameter; defaults changed between Beam versions.

Understand the failure class

Background: "missing required argument" and "the following required arguments were not provided": what required-argument errors mean and how to fix them — this error's family across 20 libraries.

Related errors

• A schema is required in order to prepare rows for writing…
• Field not nullable
• flush on closed file
• Received null value for non-nullable field " +…
• RECORD/STRUCT are not primitive types

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/7ec34ea71b6c4725. Report an issue: GitHub.

Appendix: source

Thrown at sdks/python/apache_beam/io/gcp/bigquery.py:2414

          max_insert_payload_size=self._max_insert_payload_size,
          max_retries=self._max_retries,
          num_streaming_keys=self._num_streaming_keys)

      return WriteResult(
          method=WriteToBigQuery.Method.STREAMING_INSERTS,
          failed_rows=outputs[BigQueryWriteFn.FAILED_ROWS],
          failed_rows_with_errors=outputs[
              BigQueryWriteFn.FAILED_ROWS_WITH_ERRORS])

    elif method_to_use == WriteToBigQuery.Method.FILE_LOADS:
      if self._temp_file_format == bigquery_tools.FileFormat.AVRO:
        if self.schema == SCHEMA_AUTODETECT:
          raise ValueError(
              'Schema auto-detection is not supported when using Avro based '
              'file loads into BigQuery. Please specify a schema or set '
              'temp_file_format="NEWLINE_DELIMITED_JSON"')
        if self.schema is None:
          raise ValueError(
              'A schema must be provided when writing to BigQuery using '
              'Avro based file loads')

      if self.schema and type(self.schema) is dict:

        def find_in_nested_dict(schema):
          for field in schema['fields']:
            if field['type'] == 'JSON':
              logging.warning(
                  'Found JSON type in TableSchema for "File_LOADS" write '
                  'method. Make sure the TableSchema field is a parsed '
                  'JSON to ensure the read as a JSON type. Otherwise it '
                  'will read as a raw (escaped) string.')
            elif field['type'] == 'STRUCT':
              find_in_nested_dict(field)

        find_in_nested_dict(self.schema)

View on GitHub (pinned to 12126d8942)