apache/beam · error · ValueError

triggering_frequency with STREAMING_INSERTS can only be…

Error message

triggering_frequency with STREAMING_INSERTS can only be used with with_auto_sharding=True.

What it means

When using STREAMING_INSERTS with a triggering_frequency (which flushes buffered rows on a timer), the writer must be able to dynamically shard the inserts across workers, which requires with_auto_sharding=True. Setting triggering_frequency without auto-sharding would yield single-threaded, per-key flushes, so the transform rejects the combination.

Solutions

Pass with_auto_sharding=True alongside triggering_frequency.
Remove triggering_frequency if you do not need time-based flushes.
Switch to FILE_LOADS with triggering_frequency if you prefer file-based periodic loads.

Example fix

// before
beam.io.WriteToBigQuery(table, method='STREAMING_INSERTS', triggering_frequency=60)
// after
beam.io.WriteToBigQuery(table, method='STREAMING_INSERTS', triggering_frequency=60, with_auto_sharding=True)

Defensive patterns

Strategy: validation

Validate before calling

if triggering_frequency is not None and method == WriteToBigQuery.Method.STREAMING_INSERTS:
    assert with_auto_sharding, 'triggering_frequency requires with_auto_sharding=True'

Try / catch

try:
    _ = beam.io.WriteToBigQuery(table, method='STREAMING_INSERTS', triggering_frequency=freq, with_auto_sharding=shard)
except ValueError as e:
    logger.error('Bad streaming insert options: %s', e)

Prevention

Always set with_auto_sharding=True when using triggering_frequency with STREAMING_INSERTS
Review Beam version migration notes for triggering_frequency semantics
Validate the full streaming option set in one place before pipeline expansion

When it happens

Trigger: WriteToBigQuery(method=STREAMING_INSERTS, triggering_frequency=<duration>) with with_auto_sharding=False (the default) in a streaming pipeline.

Common situations: Adding triggering_frequency to control insert batching without realizing the auto-sharding requirement; upgrading Beam and enabling triggering_frequency in an existing streaming writer.

Understand the failure class

Background: Conflicting config options: "cannot be used together" — configuration validation errors across open-source libraries — this error's family across 162 libraries.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/d7433c27c0e43758. Report an issue: GitHub.

Appendix: source

Thrown at sdks/python/apache_beam/io/gcp/bigquery.py:2365

          'with_auto_sharding is not applicable to batch pipelines.')

    experiments = p.options.view_as(DebugOptions).experiments or []
    method_to_use = self._compute_method(experiments, is_streaming_pipeline)

    if (self.schema_update_options is not None and
        method_to_use != WriteToBigQuery.Method.FILE_LOADS):
      raise ValueError(
          'schema_update_options is only supported when writing to BigQuery '
          'with FILE_LOADS.')

    if method_to_use == WriteToBigQuery.Method.STREAMING_INSERTS:
      if self.schema == SCHEMA_AUTODETECT:
        raise ValueError(
            'Schema auto-detection is not supported for streaming '
            'inserts into BigQuery. Only for File Loads.')

      if self.triggering_frequency is not None and not self.with_auto_sharding:
        raise ValueError(
            'triggering_frequency with STREAMING_INSERTS can only be used with '
            'with_auto_sharding=True.')

      if self._max_insert_payload_size > MAX_INSERT_PAYLOAD_SIZE:
        raise ValueError(
            'max_insert_payload_size can only go up to '
            f'{MAX_INSERT_PAYLOAD_SIZE} bytes, as per BigQuery quota limits: '
            'https://cloud.google.com/bigquery/quotas#streaming_inserts.')

      if self._max_retries > MAX_INSERT_RETRIES:
        raise ValueError(
            'max_retries cannot be more than '
            f'{MAX_INSERT_RETRIES}, hence please reduce the value.')

      outputs = pcoll | _StreamToBigQuery(
          table_reference=self.table_reference,
          table_side_inputs=self.table_side_inputs,
          schema_side_inputs=self.schema_side_inputs,

View on GitHub (pinned to 12126d8942)