{"record":{"id":"c523ac03f56d7c5e","repo":"apache/beam","slug":"with-auto-sharding-is-not-applicable-to-batch-pipelines","errorCode":null,"errorMessage":"with_auto_sharding is not applicable to batch pipelines.","messagePattern":"with_auto_sharding is not applicable to batch pipelines\\.","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"sdks/python/apache_beam/io/gcp/bigquery.py","lineNumber":2346,"sourceCode":"        isinstance(self.additional_bq_parameters, vp.ValueProvider)):\n      return _AdditionalBQParametersWithSchemaUpdateOptions(\n          self.additional_bq_parameters, self.schema_update_options)\n    return _merge_schema_update_options(\n        self.additional_bq_parameters, self.schema_update_options)\n\n  def expand(self, pcoll):\n    p = pcoll.pipeline\n\n    if (isinstance(self.table_reference, TableReference) and\n        self.table_reference.projectId is None):\n      self.table_reference.projectId = pcoll.pipeline.options.view_as(\n          GoogleCloudOptions).project\n\n    # TODO(pabloem): Use a different method to determine if streaming or batch.\n    is_streaming_pipeline = p.options.view_as(StandardOptions).streaming\n\n    if not is_streaming_pipeline and self.with_auto_sharding:\n      raise ValueError(\n          'with_auto_sharding is not applicable to batch pipelines.')\n\n    experiments = p.options.view_as(DebugOptions).experiments or []\n    method_to_use = self._compute_method(experiments, is_streaming_pipeline)\n\n    if (self.schema_update_options is not None and\n        method_to_use != WriteToBigQuery.Method.FILE_LOADS):\n      raise ValueError(\n          'schema_update_options is only supported when writing to BigQuery '\n          'with FILE_LOADS.')\n\n    if method_to_use == WriteToBigQuery.Method.STREAMING_INSERTS:\n      if self.schema == SCHEMA_AUTODETECT:\n        raise ValueError(\n            'Schema auto-detection is not supported for streaming '\n            'inserts into BigQuery. Only for File Loads.')\n\n      if self.triggering_frequency is not None and not self.with_auto_sharding:","sourceCodeStart":2328,"sourceCodeEnd":2364,"githubUrl":"https://github.com/apache/beam/blob/12126d8942aaf848030c478b4c6a28c6af861c66/sdks/python/apache_beam/io/gcp/bigquery.py#L2328-L2364","documentation":"WriteToBigQuery.expand validates that with_auto_sharding=True is only meaningful for streaming pipelines, where per-bundle insert grouping can be dynamically re-sharded. In batch pipelines there is no streaming source to reshuffle dynamically, so enabling the flag raises ValueError.","triggerScenarios":"Running a batch pipeline (StandardOptions.streaming is not set) with WriteToBigQuery(..., with_auto_sharding=True) and (implicitly) STREAMING_INSERTS method.","commonSituations":"Sharing a transform construction between streaming and batch variants of a pipeline; setting with_auto_sharding=True unconditionally from config in a batch test run.","solutions":["Set with_auto_sharding=False (or omit it) when running in batch mode.","Gate the flag on the pipeline's streaming option: with_auto_sharding=options.view_as(StandardOptions).streaming.","If the data volume needs sharding in batch, use FILE_LOADS or a manual Reshuffle/GroupIntoBatches instead."],"exampleFix":"// before\nbeam.io.WriteToBigQuery(table, with_auto_sharding=True)  # in batch job\n// after\nis_streaming = pipeline.options.view_as(StandardOptions).streaming\nbeam.io.WriteToBigQuery(table, with_auto_sharding=is_streaming)","handlingStrategy":"validation","validationCode":"is_streaming = pipeline.options.view_as(StandardOptions).streaming\nif not is_streaming and with_auto_sharding:\n    with_auto_sharding = False","typeGuard":null,"tryCatchPattern":"try:\n    _ = beam.io.WriteToBigQuery(table, with_auto_sharding=flag)\nexcept ValueError:\n    flag = False","preventionTips":["Gate with_auto_sharding on StandardOptions.streaming","Keep streaming-only options out of shared batch/streaming configs","Smoke-test batch and streaming variants with the same transform builder"],"tags":["bigquery","python","apache-beam","batch-vs-streaming"],"backgroundTag":"conflicting-config-options","analyzedSha":"12126d8942aaf848030c478b4c6a28c6af861c66","analyzedAt":"2026-09-13T01:50:10.254Z","contentChangedAt":"2026-09-13T01:50:10.254Z","schemaVersion":2},"datasetVersion":"2026-09-20T03:17:13.778Z"}