apache/beam · error · ValueError

schema_update_options is only supported when writing to…

Error message

schema_update_options is only supported when writing to BigQuery with FILE_LOADS.

What it means

schema_update_options (e.g. ALLOW_FIELD_ADDITION / ALLOW_FIELD_RELAXATION) are implemented by BigQuery load jobs only. When the chosen write method is not FILE_LOADS (e.g. STREAMING_INSERTS) the schema update options cannot be honored, so expand() raises ValueError rather than silently ignoring them.

Solutions

Pass method=WriteToBigQuery.Method.FILE_LOADS together with schema_update_options.
Remove schema_update_options if you must keep STREAMING_INSERTS.
Enable the storage write API (experiments/use_storage_write_api) only if your Beam version supports schema update there.

Example fix

// before
beam.io.WriteToBigQuery(table, schema_update_options=['ALLOW_FIELD_ADDITION'])
// after
beam.io.WriteToBigQuery(table, method=beam.io.WriteToBigQuery.Method.FILE_LOADS, schema_update_options=['ALLOW_FIELD_ADDITION'])

Defensive patterns

Strategy: validation

Validate before calling

if schema_update_options and method != WriteToBigQuery.Method.FILE_LOADS:
    raise ValueError('schema_update_options require FILE_LOADS')

Try / catch

try:
    _ = beam.io.WriteToBigQuery(table, method=method, schema_update_options=opts)
except ValueError as e:
    logger.error('%s; switch to FILE_LOADS or drop schema_update_options', e)

Prevention

Pair schema_update_options explicitly with method=FILE_LOADS
Remember STREAMING_INSERTS may be the default in batch
Validate the method/options combination in a shared helper

When it happens

Trigger: WriteToBigQuery with schema_update_options=['ALLOW_FIELD_ADDITION'] while method resolves to STREAMING_INSERTS (explicitly, by default in batch, or via _compute_method).

Common situations: Defaulting to STREAMING_INSERTS but copying schema_update_options from an existing file-loads pipeline; forgetting that STREAMING_INSERTS is the batch default in older Beam versions.

Understand the failure class

Background: Conflicting config options: "cannot be used together" — configuration validation errors across open-source libraries — this error's family across 162 libraries.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/c413c3daa1f82d1e. Report an issue: GitHub.

Appendix: source

Thrown at sdks/python/apache_beam/io/gcp/bigquery.py:2354

    if (isinstance(self.table_reference, TableReference) and
        self.table_reference.projectId is None):
      self.table_reference.projectId = pcoll.pipeline.options.view_as(
          GoogleCloudOptions).project

    # TODO(pabloem): Use a different method to determine if streaming or batch.
    is_streaming_pipeline = p.options.view_as(StandardOptions).streaming

    if not is_streaming_pipeline and self.with_auto_sharding:
      raise ValueError(
          'with_auto_sharding is not applicable to batch pipelines.')

    experiments = p.options.view_as(DebugOptions).experiments or []
    method_to_use = self._compute_method(experiments, is_streaming_pipeline)

    if (self.schema_update_options is not None and
        method_to_use != WriteToBigQuery.Method.FILE_LOADS):
      raise ValueError(
          'schema_update_options is only supported when writing to BigQuery '
          'with FILE_LOADS.')

    if method_to_use == WriteToBigQuery.Method.STREAMING_INSERTS:
      if self.schema == SCHEMA_AUTODETECT:
        raise ValueError(
            'Schema auto-detection is not supported for streaming '
            'inserts into BigQuery. Only for File Loads.')

      if self.triggering_frequency is not None and not self.with_auto_sharding:
        raise ValueError(
            'triggering_frequency with STREAMING_INSERTS can only be used with '
            'with_auto_sharding=True.')

      if self._max_insert_payload_size > MAX_INSERT_PAYLOAD_SIZE:
        raise ValueError(
            'max_insert_payload_size can only go up to '
            f'{MAX_INSERT_PAYLOAD_SIZE} bytes, as per BigQuery quota limits: '

View on GitHub (pinned to 12126d8942)