apache/beam · error · ValueError

Schema auto-detection is not supported when using Avro…

Error message

Schema auto-detection is not supported when using Avro based file loads into BigQuery. Please specify a schema or set temp_file_format="NEWLINE_DELIMITED_JSON"

What it means

When using FILE_LOADS with temp_file_format=AVRO, WriteToBigQuery.expand() rejects schema=SCHEMA_AUTODETECT because BigQuery Avro load jobs cannot auto-detect schemas (the Avro files themselves define the schema, and Beam cannot emit schemaless Avro).

Solutions

Provide an explicit schema (string, dict, or table schema).
Switch temp_file_format='NEWLINE_DELIMITED_JSON' so auto-detection is allowed.
Let the schema be inferred from typed elements by passing a dict/Row type input and omitting the string 'AUTO_DETECT'.

Example fix

// before
WriteToBigQuery(table='proj:ds.tbl', method='FILE_LOADS', schema='AUTO_DETECT', temp_file_format='AVRO')
// after
WriteToBigQuery(table='proj:ds.tbl', method='FILE_LOADS', schema='col1:STRING,col2:INTEGER', temp_file_format='AVRO')

Defensive patterns

Strategy: validation

Validate before calling

if temp_file_format == 'AVRO' and schema in (None, 'AUTO_DETECT', 'SCHEMA_AUTODETECT'):
    raise ValueError('Provide a schema or use NEWLINE_DELIMITED_JSON')

Prevention

Always pair Avro temp files with an explicit schema.
Prefer passing typed PCollections so schemas are derived automatically.

When it happens

Trigger: WriteToBigQuery(method=FILE_LOADS, schema=SCHEMA_AUTODETECT ('AUTO_DETECT'), temp_file_format=Avro) expanded on an untyped input.

Common situations: Switching a pipeline from JSON to Avro temp files (often for performance) while leaving schema='AUTO_DETECT' set from earlier config.

Understand the failure class

Background: Schema validation failed / invalid input schema: payload rejected because its shape doesn't match the expected schema — this error's family across 28 libraries.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/4013ef4636665316. Report an issue: GitHub.

Appendix: source

Thrown at sdks/python/apache_beam/io/gcp/bigquery.py:2409

          additional_bq_parameters=self.additional_bq_parameters,
          ignore_insert_ids=self._ignore_insert_ids,
          ignore_unknown_columns=self._ignore_unknown_columns,
          with_auto_sharding=self.with_auto_sharding,
          test_client=self.test_client,
          max_insert_payload_size=self._max_insert_payload_size,
          max_retries=self._max_retries,
          num_streaming_keys=self._num_streaming_keys)

      return WriteResult(
          method=WriteToBigQuery.Method.STREAMING_INSERTS,
          failed_rows=outputs[BigQueryWriteFn.FAILED_ROWS],
          failed_rows_with_errors=outputs[
              BigQueryWriteFn.FAILED_ROWS_WITH_ERRORS])

    elif method_to_use == WriteToBigQuery.Method.FILE_LOADS:
      if self._temp_file_format == bigquery_tools.FileFormat.AVRO:
        if self.schema == SCHEMA_AUTODETECT:
          raise ValueError(
              'Schema auto-detection is not supported when using Avro based '
              'file loads into BigQuery. Please specify a schema or set '
              'temp_file_format="NEWLINE_DELIMITED_JSON"')
        if self.schema is None:
          raise ValueError(
              'A schema must be provided when writing to BigQuery using '
              'Avro based file loads')

      if self.schema and type(self.schema) is dict:

        def find_in_nested_dict(schema):
          for field in schema['fields']:
            if field['type'] == 'JSON':
              logging.warning(
                  'Found JSON type in TableSchema for "File_LOADS" write '
                  'method. Make sure the TableSchema field is a parsed '
                  'JSON to ensure the read as a JSON type. Otherwise it '
                  'will read as a raw (escaped) string.')

View on GitHub (pinned to 12126d8942)