{"record":{"id":"47a77e0458693819","repo":"apache/beam","slug":"triggering-frequency-must-be-specified-to-use-fileloads-in","errorCode":null,"errorMessage":"triggering_frequency must be specified to use fileloads in streaming","messagePattern":"triggering_frequency must be specified to use fileloads in streaming","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"sdks/python/apache_beam/io/gcp/bigquery_file_loads.py","lineNumber":1030,"sourceCode":"    self.load_job_project_id = load_job_project_id\n    self._validate = validate\n    if self._validate:\n      self.verify()\n\n  def verify(self):\n    if (isinstance(self._custom_gcs_temp_location.get(), vp.StaticValueProvider)\n        and not self._custom_gcs_temp_location.get().startswith('gs://')):\n      # Only fail if the custom location is provided, and it is not a GCS\n      # location.\n      raise ValueError(\n          'Invalid GCS location: %r.\\n'\n          'Writing to BigQuery with FILE_LOADS method requires a '\n          'GCS location to be provided to write files to be '\n          'loaded into BigQuery. Please provide a GCS bucket, or '\n          'pass method=\"STREAMING_INSERTS\" to WriteToBigQuery.' %\n          self._custom_gcs_temp_location.get())\n    if self.is_streaming_pipeline and not self.triggering_frequency:\n      raise ValueError(\n          'triggering_frequency must be specified to use file'\n          'loads in streaming')\n    elif not self.is_streaming_pipeline and self.triggering_frequency:\n      raise ValueError(\n          'triggering_frequency can only be used with file'\n          'loads in streaming')\n    if not self.is_streaming_pipeline and self.with_auto_sharding:\n      return ValueError(\n          'with_auto_sharding can only be used with file loads in streaming.')\n\n  def _window_fn(self):\n    \"\"\"Set the correct WindowInto PTransform\"\"\"\n\n    # The user-supplied triggering_frequency is often chosen to control how\n    # many BigQuery load jobs are triggered, to prevent going over BigQuery's\n    # daily quota for load jobs. If this is set to a large value, currently we\n    # have to buffer all the data until the trigger fires. Instead we ensure\n    # that the files are written if a threshold number of records are ready.","sourceCodeStart":1012,"sourceCodeEnd":1048,"githubUrl":"https://github.com/apache/beam/blob/12126d8942aaf848030c478b4c6a28c6af861c66/sdks/python/apache_beam/io/gcp/bigquery_file_loads.py#L1012-L1048","documentation":"In streaming pipelines, FILE_LOADS needs triggering_frequency to know how often to flush staged files and run load jobs. verify() enforces that streaming pipelines set it, raising this ValueError when is_streaming_pipeline is True and triggering_frequency is None.","triggerScenarios":"Using WriteToBigQuery(method=FILE_LOADS) (or WriteRecordsToFile path) in a streaming pipeline without specifying triggering_frequency.","commonSituations":"Converting a batch BigQuery-load pipeline to streaming and forgetting the frequency parameter; running on a streaming runner (Flink/Spark streaming) with defaults.","solutions":["Add triggering_frequency=300 (seconds) to WriteToBigQuery.","Alternatively use method='STREAMING_INSERTS' for streaming writes.","Ensure the pipeline options/windowing reflect streaming only when intended."],"exampleFix":"// before\nbeam.io.WriteToBigQuery(table, method=beam.io.WriteToBigQuery.Method.FILE_LOADS)\n// after\nbeam.io.WriteToBigQuery(table, method=beam.io.WriteToBigQuery.Method.FILE_LOADS, triggering_frequency=300)","handlingStrategy":"validation","validationCode":"if is_streaming and triggering_frequency is None:\n    raise ValueError('streaming FILE_LOADS requires triggering_frequency')","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Set triggering_frequency whenever pipeline is streaming","Detect streaming mode from PipelineOptions before building transforms","Prefer STREAMING_INSERTS for simple streaming sinks"],"tags":["bigquery","streaming","beam","configuration"],"backgroundTag":"missing-required-config-field","analyzedSha":"12126d8942aaf848030c478b4c6a28c6af861c66","analyzedAt":"2026-09-13T01:50:10.254Z","contentChangedAt":"2026-09-13T01:50:10.254Z","schemaVersion":2},"datasetVersion":"2026-09-14T21:17:11.552Z"}