apache/beam · error · ValueError

triggering_frequency can only be used with fileloads in…

Error message

triggering_frequency can only be used with fileloads in streaming

What it means

triggering_frequency controls periodic file flushes, which only makes sense for streaming file-loads. verify() raises this ValueError when a batch (non-streaming) pipeline sets triggering_frequency, since there is no periodic trigger in batch mode.

Solutions

1. Remove triggering_frequency for batch pipelines.
2. If periodic loading is desired, run the pipeline in streaming mode (set streaming=True in PipelineOptions).
3. Make the parameter conditional on the pipeline's streaming flag.

Example fix

// before
beam.io.WriteToBigQuery(table, triggering_frequency=300)  # batch pipeline
// after
beam.io.WriteToBigQuery(table)  # batch: drop triggering_frequency

if not is_streaming and triggering_frequency is not None:
    raise ValueError('triggering_frequency is only valid in streaming pipelines')

When it happens

Trigger: Passing triggering_frequency to WriteToBigQuery/BigQueryBatchFileLoads while the pipeline is a batch pipeline (no streaming options set).

Common situations: Copy-pasting a streaming pipeline's WriteToBigQuery options into a batch job, or enabling a trigger unconditionally in shared pipeline code.

Understand the failure class

Background: Conflicting config options: "cannot be used together" — configuration validation errors across open-source libraries — this error's family across 162 libraries.

Related errors

• Array of collection is not supported in BigQuery.
• BigQuery data contained value
• Both a BigQuery temp_dataset_id and a temp_table_ref were…
• cannot encode a null value
• cannot encode a null value

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/1c8aee3bd2b90aa5. Report an issue: GitHub.

Appendix: source

Thrown at sdks/python/apache_beam/io/gcp/bigquery_file_loads.py:1034

  def verify(self):
    if (isinstance(self._custom_gcs_temp_location.get(), vp.StaticValueProvider)
        and not self._custom_gcs_temp_location.get().startswith('gs://')):
      # Only fail if the custom location is provided, and it is not a GCS
      # location.
      raise ValueError(
          'Invalid GCS location: %r.\n'
          'Writing to BigQuery with FILE_LOADS method requires a '
          'GCS location to be provided to write files to be '
          'loaded into BigQuery. Please provide a GCS bucket, or '
          'pass method="STREAMING_INSERTS" to WriteToBigQuery.' %
          self._custom_gcs_temp_location.get())
    if self.is_streaming_pipeline and not self.triggering_frequency:
      raise ValueError(
          'triggering_frequency must be specified to use file'
          'loads in streaming')
    elif not self.is_streaming_pipeline and self.triggering_frequency:
      raise ValueError(
          'triggering_frequency can only be used with file'
          'loads in streaming')
    if not self.is_streaming_pipeline and self.with_auto_sharding:
      return ValueError(
          'with_auto_sharding can only be used with file loads in streaming.')

  def _window_fn(self):
    """Set the correct WindowInto PTransform"""

    # The user-supplied triggering_frequency is often chosen to control how
    # many BigQuery load jobs are triggered, to prevent going over BigQuery's
    # daily quota for load jobs. If this is set to a large value, currently we
    # have to buffer all the data until the trigger fires. Instead we ensure
    # that the files are written if a threshold number of records are ready.
    # We use only the user-supplied trigger on the actual BigQuery load.
    # This allows us to offload the data to the filesystem.
    #
    # In the case of dynamic sharding, however, we use a default trigger since

View on GitHub (pinned to 12126d8942)