apache/beam · error · ValueError

with_auto_sharding is not applicable to batch pipelines.

Error message

with_auto_sharding is not applicable to batch pipelines.

What it means

WriteToBigQuery.expand validates that with_auto_sharding=True is only meaningful for streaming pipelines, where per-bundle insert grouping can be dynamically re-sharded. In batch pipelines there is no streaming source to reshuffle dynamically, so enabling the flag raises ValueError.

Solutions

Set with_auto_sharding=False (or omit it) when running in batch mode.
Gate the flag on the pipeline's streaming option: with_auto_sharding=options.view_as(StandardOptions).streaming.
If the data volume needs sharding in batch, use FILE_LOADS or a manual Reshuffle/GroupIntoBatches instead.

Example fix

// before
beam.io.WriteToBigQuery(table, with_auto_sharding=True)  # in batch job
// after
is_streaming = pipeline.options.view_as(StandardOptions).streaming
beam.io.WriteToBigQuery(table, with_auto_sharding=is_streaming)

Defensive patterns

Strategy: validation

Validate before calling

is_streaming = pipeline.options.view_as(StandardOptions).streaming
if not is_streaming and with_auto_sharding:
    with_auto_sharding = False

Try / catch

try:
    _ = beam.io.WriteToBigQuery(table, with_auto_sharding=flag)
except ValueError:
    flag = False

Prevention

Gate with_auto_sharding on StandardOptions.streaming
Keep streaming-only options out of shared batch/streaming configs
Smoke-test batch and streaming variants with the same transform builder

When it happens

Trigger: Running a batch pipeline (StandardOptions.streaming is not set) with WriteToBigQuery(..., with_auto_sharding=True) and (implicitly) STREAMING_INSERTS method.

Common situations: Sharing a transform construction between streaming and batch variants of a pipeline; setting with_auto_sharding=True unconditionally from config in a batch test run.

Understand the failure class

Background: Conflicting config options: "cannot be used together" — configuration validation errors across open-source libraries — this error's family across 162 libraries.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/c523ac03f56d7c5e. Report an issue: GitHub.

Appendix: source

Thrown at sdks/python/apache_beam/io/gcp/bigquery.py:2346

        isinstance(self.additional_bq_parameters, vp.ValueProvider)):
      return _AdditionalBQParametersWithSchemaUpdateOptions(
          self.additional_bq_parameters, self.schema_update_options)
    return _merge_schema_update_options(
        self.additional_bq_parameters, self.schema_update_options)

  def expand(self, pcoll):
    p = pcoll.pipeline

    if (isinstance(self.table_reference, TableReference) and
        self.table_reference.projectId is None):
      self.table_reference.projectId = pcoll.pipeline.options.view_as(
          GoogleCloudOptions).project

    # TODO(pabloem): Use a different method to determine if streaming or batch.
    is_streaming_pipeline = p.options.view_as(StandardOptions).streaming

    if not is_streaming_pipeline and self.with_auto_sharding:
      raise ValueError(
          'with_auto_sharding is not applicable to batch pipelines.')

    experiments = p.options.view_as(DebugOptions).experiments or []
    method_to_use = self._compute_method(experiments, is_streaming_pipeline)

    if (self.schema_update_options is not None and
        method_to_use != WriteToBigQuery.Method.FILE_LOADS):
      raise ValueError(
          'schema_update_options is only supported when writing to BigQuery '
          'with FILE_LOADS.')

    if method_to_use == WriteToBigQuery.Method.STREAMING_INSERTS:
      if self.schema == SCHEMA_AUTODETECT:
        raise ValueError(
            'Schema auto-detection is not supported for streaming '
            'inserts into BigQuery. Only for File Loads.')

      if self.triggering_frequency is not None and not self.with_auto_sharding:

View on GitHub (pinned to 12126d8942)