apache/beam · error · ValueError

%s can be set with either schema_update_options or additiona

Error message

%s can be set with either schema_update_options or additional_bq_parameters, but not both.

What it means

BigQuery sink configuration forbids setting the _SCHEMA_UPDATE_OPTIONS key inside additional_bq_parameters when schema_update_options is also provided, since the same option would be specified twice with possibly conflicting values.

Source

Thrown at sdks/python/apache_beam/io/gcp/bigquery.py:495

of retries.
"""
MAX_INSERT_RETRIES = 10000
"""
The maximum byte size for a BigQuery legacy streaming insert payload.

Note: The actual limit is 10MB, but we set it to 9MB to make room for request
overhead: https://cloud.google.com/bigquery/quotas#streaming_inserts
"""
MAX_INSERT_PAYLOAD_SIZE = 9 << 20

_SCHEMA_UPDATE_OPTIONS = 'schemaUpdateOptions'


def _merge_schema_update_options(
    additional_bq_parameters, schema_update_options):
  additional_bq_parameters = dict(additional_bq_parameters or {})
  if _SCHEMA_UPDATE_OPTIONS in additional_bq_parameters:
    raise ValueError(
        '%s can be set with either schema_update_options or '
        'additional_bq_parameters, but not both.' % _SCHEMA_UPDATE_OPTIONS)
  additional_bq_parameters[_SCHEMA_UPDATE_OPTIONS] = schema_update_options
  return additional_bq_parameters


class _AdditionalBQParametersWithSchemaUpdateOptions(object):
  def __init__(self, additional_bq_parameters, schema_update_options):
    self.additional_bq_parameters = additional_bq_parameters
    self.schema_update_options = schema_update_options

  def __call__(self, destination):
    if callable(self.additional_bq_parameters):
      additional_bq_parameters = self.additional_bq_parameters(destination)
    elif isinstance(self.additional_bq_parameters, vp.ValueProvider):
      additional_bq_parameters = self.additional_bq_parameters.get()
    else:
      additional_bq_parameters = self.additional_bq_parameters

View on GitHub (pinned to 12126d8942)

Solutions

1. Remove the SCHEMA_UPDATE_OPTIONS key from additional_bq_parameters and pass those options via schema_update_options instead
2. Or remove the schema_update_options argument and keep options only in additional_bq_parameters

Example fix

// before
WriteToBigQuery(table=t, additional_bq_parameters={'schemaUpdateOptions': ['ALLOW_FIELD_ADDITION']}, schema_update_options=['ALLOW_FIELD_ADDITION'])
// after
WriteToBigQuery(table=t, schema_update_options=['ALLOW_FIELD_ADDITION'])

if additional_bq_parameters and 'schemaUpdateOptions' in additional_bq_parameters and schema_update_options:
    raise ValueError('Set schema update options via schema_update_options only')

try:
    result = transform(args)
except ValueError as e:
    if 'but not both' in str(e):
        additional_bq_parameters.pop('schemaUpdateOptions', None)
        result = transform(args)
    else:
        raise

When it happens

Trigger: Calling BigQueryWriteResults/WriteToBigQuery transform path where additional_bq_parameters dict already contains SCHEMA_UPDATE_OPTIONS key and schema_update_options is also passed (via __call__ or _additional_bq_parameters_for_file_loads).

Common situations: Users migrating code that set schema update options through raw additional_bq_parameters (job config) and then also added the newer schema_update_options parameter.

Understand the failure class

Background: Conflicting config options: "cannot be used together" — configuration validation errors across open-source libraries — this error's family across 162 libraries.

Related errors

• Both a BigQuery table and a query were specified. Please spe
• The "use_native_datetime" parameter cannot be True for EXPOR
• Please specify a BigQuery table to read from.
• Encountered unsupported parameter(s) in read_gbq: {kwargs.ke
• Cannot create a temporary directory for root path prefix %s.

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/4e076732d3ae585c. Report an issue: GitHub.