{"record":{"id":"d7433c27c0e43758","repo":"apache/beam","slug":"triggering-frequency-with-streaming-inserts-can-only-be-used","errorCode":null,"errorMessage":"triggering_frequency with STREAMING_INSERTS can only be used with with_auto_sharding=True.","messagePattern":"triggering_frequency with STREAMING_INSERTS can only be used with with_auto_sharding=True\\.","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"sdks/python/apache_beam/io/gcp/bigquery.py","lineNumber":2365,"sourceCode":"          'with_auto_sharding is not applicable to batch pipelines.')\n\n    experiments = p.options.view_as(DebugOptions).experiments or []\n    method_to_use = self._compute_method(experiments, is_streaming_pipeline)\n\n    if (self.schema_update_options is not None and\n        method_to_use != WriteToBigQuery.Method.FILE_LOADS):\n      raise ValueError(\n          'schema_update_options is only supported when writing to BigQuery '\n          'with FILE_LOADS.')\n\n    if method_to_use == WriteToBigQuery.Method.STREAMING_INSERTS:\n      if self.schema == SCHEMA_AUTODETECT:\n        raise ValueError(\n            'Schema auto-detection is not supported for streaming '\n            'inserts into BigQuery. Only for File Loads.')\n\n      if self.triggering_frequency is not None and not self.with_auto_sharding:\n        raise ValueError(\n            'triggering_frequency with STREAMING_INSERTS can only be used with '\n            'with_auto_sharding=True.')\n\n      if self._max_insert_payload_size > MAX_INSERT_PAYLOAD_SIZE:\n        raise ValueError(\n            'max_insert_payload_size can only go up to '\n            f'{MAX_INSERT_PAYLOAD_SIZE} bytes, as per BigQuery quota limits: '\n            'https://cloud.google.com/bigquery/quotas#streaming_inserts.')\n\n      if self._max_retries > MAX_INSERT_RETRIES:\n        raise ValueError(\n            'max_retries cannot be more than '\n            f'{MAX_INSERT_RETRIES}, hence please reduce the value.')\n\n      outputs = pcoll | _StreamToBigQuery(\n          table_reference=self.table_reference,\n          table_side_inputs=self.table_side_inputs,\n          schema_side_inputs=self.schema_side_inputs,","sourceCodeStart":2347,"sourceCodeEnd":2383,"githubUrl":"https://github.com/apache/beam/blob/12126d8942aaf848030c478b4c6a28c6af861c66/sdks/python/apache_beam/io/gcp/bigquery.py#L2347-L2383","documentation":"When using STREAMING_INSERTS with a triggering_frequency (which flushes buffered rows on a timer), the writer must be able to dynamically shard the inserts across workers, which requires with_auto_sharding=True. Setting triggering_frequency without auto-sharding would yield single-threaded, per-key flushes, so the transform rejects the combination.","triggerScenarios":"WriteToBigQuery(method=STREAMING_INSERTS, triggering_frequency=<duration>) with with_auto_sharding=False (the default) in a streaming pipeline.","commonSituations":"Adding triggering_frequency to control insert batching without realizing the auto-sharding requirement; upgrading Beam and enabling triggering_frequency in an existing streaming writer.","solutions":["Pass with_auto_sharding=True alongside triggering_frequency.","Remove triggering_frequency if you do not need time-based flushes.","Switch to FILE_LOADS with triggering_frequency if you prefer file-based periodic loads."],"exampleFix":"// before\nbeam.io.WriteToBigQuery(table, method='STREAMING_INSERTS', triggering_frequency=60)\n// after\nbeam.io.WriteToBigQuery(table, method='STREAMING_INSERTS', triggering_frequency=60, with_auto_sharding=True)","handlingStrategy":"validation","validationCode":"if triggering_frequency is not None and method == WriteToBigQuery.Method.STREAMING_INSERTS:\n    assert with_auto_sharding, 'triggering_frequency requires with_auto_sharding=True'","typeGuard":null,"tryCatchPattern":"try:\n    _ = beam.io.WriteToBigQuery(table, method='STREAMING_INSERTS', triggering_frequency=freq, with_auto_sharding=shard)\nexcept ValueError as e:\n    logger.error('Bad streaming insert options: %s', e)","preventionTips":["Always set with_auto_sharding=True when using triggering_frequency with STREAMING_INSERTS","Review Beam version migration notes for triggering_frequency semantics","Validate the full streaming option set in one place before pipeline expansion"],"tags":["bigquery","python","apache-beam","streaming-inserts"],"backgroundTag":"conflicting-config-options","analyzedSha":"12126d8942aaf848030c478b4c6a28c6af861c66","analyzedAt":"2026-09-13T01:50:10.254Z","contentChangedAt":"2026-09-13T01:50:10.254Z","schemaVersion":2},"datasetVersion":"2026-09-20T03:17:13.778Z"}