apache/beam · error · ValueError

Schema auto-detection is not supported for streaming…

Error message

Schema auto-detection is not supported for streaming inserts into BigQuery. Only for File Loads.

What it means

SCHEMA_AUTODETECT relies on BigQuery load jobs to infer the schema from the data files. Streaming inserts require the schema to be provided up front (each row insert carries typed values against a known table schema), so expand() raises ValueError when schema=SCHEMA_AUTODETECT is combined with STREAMING_INSERTS.

Solutions

Provide an explicit schema (dict or JSON string) instead of SCHEMA_AUTODETECT.
Switch to method=FILE_LOADS where auto-detection is supported.
Gate schema_autodetect so it is only set when FILE_LOADS is chosen.

Example fix

// before
beam.io.WriteToBigQuery(table, schema=SCHEMA_AUTODETECT, method='STREAMING_INSERTS')
// after
beam.io.WriteToBigQuery(table, schema={'fields': [{'name': 'id', 'type': 'INTEGER'}]}, method='STREAMING_INSERTS')

Defensive patterns

Strategy: validation

Validate before calling

if method == WriteToBigQuery.Method.STREAMING_INSERTS and schema == SCHEMA_AUTODETECT:
    raise ValueError('provide an explicit schema for streaming inserts')

Type guard

def schema_provided(schema):
    return schema is not None and schema != SCHEMA_AUTODETECT

Try / catch

try:
    _ = beam.io.WriteToBigQuery(table, schema=schema, method='STREAMING_INSERTS')
except ValueError:
    schema = load_schema_from_config()

Prevention

Never combine SCHEMA_AUTODETECT with STREAMING_INSERTS
Only set schema_autodetect=True for FILE_LOADS pipelines
Require an explicit schema in streaming pipeline configs

When it happens

Trigger: WriteToBigQuery(method=STREAMING_INSERTS, schema=beam.io.bigquery.SCHEMA_AUTODETECT) — including implicit STREAMING_INSERTS selection in batch pipelines when schema_autodetect=True.

Common situations: Setting schema_autodetect=True from a config flag shared across pipelines; switching method to STREAMING_INSERTS while leaving SCHEMA_AUTODETECT in place from a file-loads setup.

Understand the failure class

Background: UnsupportedOperationException and "is not supported" errors: when a library deliberately refuses a call — this error's family across 30 libraries.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/05457c7d709c4566. Report an issue: GitHub.

Appendix: source

Thrown at sdks/python/apache_beam/io/gcp/bigquery.py:2360

    # TODO(pabloem): Use a different method to determine if streaming or batch.
    is_streaming_pipeline = p.options.view_as(StandardOptions).streaming

    if not is_streaming_pipeline and self.with_auto_sharding:
      raise ValueError(
          'with_auto_sharding is not applicable to batch pipelines.')

    experiments = p.options.view_as(DebugOptions).experiments or []
    method_to_use = self._compute_method(experiments, is_streaming_pipeline)

    if (self.schema_update_options is not None and
        method_to_use != WriteToBigQuery.Method.FILE_LOADS):
      raise ValueError(
          'schema_update_options is only supported when writing to BigQuery '
          'with FILE_LOADS.')

    if method_to_use == WriteToBigQuery.Method.STREAMING_INSERTS:
      if self.schema == SCHEMA_AUTODETECT:
        raise ValueError(
            'Schema auto-detection is not supported for streaming '
            'inserts into BigQuery. Only for File Loads.')

      if self.triggering_frequency is not None and not self.with_auto_sharding:
        raise ValueError(
            'triggering_frequency with STREAMING_INSERTS can only be used with '
            'with_auto_sharding=True.')

      if self._max_insert_payload_size > MAX_INSERT_PAYLOAD_SIZE:
        raise ValueError(
            'max_insert_payload_size can only go up to '
            f'{MAX_INSERT_PAYLOAD_SIZE} bytes, as per BigQuery quota limits: '
            'https://cloud.google.com/bigquery/quotas#streaming_inserts.')

      if self._max_retries > MAX_INSERT_RETRIES:
        raise ValueError(
            'max_retries cannot be more than '
            f'{MAX_INSERT_RETRIES}, hence please reduce the value.')

View on GitHub (pinned to 12126d8942)