apache/beam · error · ValueError

triggering_frequency must be specified to use fileloads in s

Error message

triggering_frequency must be specified to use fileloads in streaming

What it means

In streaming pipelines, FILE_LOADS needs triggering_frequency to know how often to flush staged files and run load jobs. verify() enforces that streaming pipelines set it, raising this ValueError when is_streaming_pipeline is True and triggering_frequency is None.

Source

Thrown at sdks/python/apache_beam/io/gcp/bigquery_file_loads.py:1030

    self.load_job_project_id = load_job_project_id
    self._validate = validate
    if self._validate:
      self.verify()

  def verify(self):
    if (isinstance(self._custom_gcs_temp_location.get(), vp.StaticValueProvider)
        and not self._custom_gcs_temp_location.get().startswith('gs://')):
      # Only fail if the custom location is provided, and it is not a GCS
      # location.
      raise ValueError(
          'Invalid GCS location: %r.\n'
          'Writing to BigQuery with FILE_LOADS method requires a '
          'GCS location to be provided to write files to be '
          'loaded into BigQuery. Please provide a GCS bucket, or '
          'pass method="STREAMING_INSERTS" to WriteToBigQuery.' %
          self._custom_gcs_temp_location.get())
    if self.is_streaming_pipeline and not self.triggering_frequency:
      raise ValueError(
          'triggering_frequency must be specified to use file'
          'loads in streaming')
    elif not self.is_streaming_pipeline and self.triggering_frequency:
      raise ValueError(
          'triggering_frequency can only be used with file'
          'loads in streaming')
    if not self.is_streaming_pipeline and self.with_auto_sharding:
      return ValueError(
          'with_auto_sharding can only be used with file loads in streaming.')

  def _window_fn(self):
    """Set the correct WindowInto PTransform"""

    # The user-supplied triggering_frequency is often chosen to control how
    # many BigQuery load jobs are triggered, to prevent going over BigQuery's
    # daily quota for load jobs. If this is set to a large value, currently we
    # have to buffer all the data until the trigger fires. Instead we ensure
    # that the files are written if a threshold number of records are ready.

View on GitHub (pinned to 12126d8942)

Solutions

1. Add triggering_frequency=300 (seconds) to WriteToBigQuery.
2. Alternatively use method='STREAMING_INSERTS' for streaming writes.
3. Ensure the pipeline options/windowing reflect streaming only when intended.

Example fix

// before
beam.io.WriteToBigQuery(table, method=beam.io.WriteToBigQuery.Method.FILE_LOADS)
// after
beam.io.WriteToBigQuery(table, method=beam.io.WriteToBigQuery.Method.FILE_LOADS, triggering_frequency=300)

if is_streaming and triggering_frequency is None:
    raise ValueError('streaming FILE_LOADS requires triggering_frequency')

When it happens

Trigger: Using WriteToBigQuery(method=FILE_LOADS) (or WriteRecordsToFile path) in a streaming pipeline without specifying triggering_frequency.

Common situations: Converting a batch BigQuery-load pipeline to streaming and forgetting the frequency parameter; running on a streaming runner (Flink/Spark streaming) with defaults.

Understand the failure class

Background: "is required", "must be set", "missing required field": configuration validation errors across open-source libraries — this error's family across 36 libraries.

Related errors

• project must be specified either in ReadBigQueryChangeHistor
• Unrecognized value for stable unique names:
• File path must be absolute:
• Secret option string cannot be null
• Secret string must contain a valid type parameter

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/47a77e0458693819. Report an issue: GitHub.