{"record":{"id":"1c8aee3bd2b90aa5","repo":"apache/beam","slug":"triggering-frequency-can-only-be-used-with-fileloads-in","errorCode":null,"errorMessage":"triggering_frequency can only be used with fileloads in streaming","messagePattern":"triggering_frequency can only be used with fileloads in streaming","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"sdks/python/apache_beam/io/gcp/bigquery_file_loads.py","lineNumber":1034,"sourceCode":"\n  def verify(self):\n    if (isinstance(self._custom_gcs_temp_location.get(), vp.StaticValueProvider)\n        and not self._custom_gcs_temp_location.get().startswith('gs://')):\n      # Only fail if the custom location is provided, and it is not a GCS\n      # location.\n      raise ValueError(\n          'Invalid GCS location: %r.\\n'\n          'Writing to BigQuery with FILE_LOADS method requires a '\n          'GCS location to be provided to write files to be '\n          'loaded into BigQuery. Please provide a GCS bucket, or '\n          'pass method=\"STREAMING_INSERTS\" to WriteToBigQuery.' %\n          self._custom_gcs_temp_location.get())\n    if self.is_streaming_pipeline and not self.triggering_frequency:\n      raise ValueError(\n          'triggering_frequency must be specified to use file'\n          'loads in streaming')\n    elif not self.is_streaming_pipeline and self.triggering_frequency:\n      raise ValueError(\n          'triggering_frequency can only be used with file'\n          'loads in streaming')\n    if not self.is_streaming_pipeline and self.with_auto_sharding:\n      return ValueError(\n          'with_auto_sharding can only be used with file loads in streaming.')\n\n  def _window_fn(self):\n    \"\"\"Set the correct WindowInto PTransform\"\"\"\n\n    # The user-supplied triggering_frequency is often chosen to control how\n    # many BigQuery load jobs are triggered, to prevent going over BigQuery's\n    # daily quota for load jobs. If this is set to a large value, currently we\n    # have to buffer all the data until the trigger fires. Instead we ensure\n    # that the files are written if a threshold number of records are ready.\n    # We use only the user-supplied trigger on the actual BigQuery load.\n    # This allows us to offload the data to the filesystem.\n    #\n    # In the case of dynamic sharding, however, we use a default trigger since","sourceCodeStart":1016,"sourceCodeEnd":1052,"githubUrl":"https://github.com/apache/beam/blob/12126d8942aaf848030c478b4c6a28c6af861c66/sdks/python/apache_beam/io/gcp/bigquery_file_loads.py#L1016-L1052","documentation":"triggering_frequency controls periodic file flushes, which only makes sense for streaming file-loads. verify() raises this ValueError when a batch (non-streaming) pipeline sets triggering_frequency, since there is no periodic trigger in batch mode.","triggerScenarios":"Passing triggering_frequency to WriteToBigQuery/BigQueryBatchFileLoads while the pipeline is a batch pipeline (no streaming options set).","commonSituations":"Copy-pasting a streaming pipeline's WriteToBigQuery options into a batch job, or enabling a trigger unconditionally in shared pipeline code.","solutions":["Remove triggering_frequency for batch pipelines.","If periodic loading is desired, run the pipeline in streaming mode (set streaming=True in PipelineOptions).","Make the parameter conditional on the pipeline's streaming flag."],"exampleFix":"// before\nbeam.io.WriteToBigQuery(table, triggering_frequency=300)  # batch pipeline\n// after\nbeam.io.WriteToBigQuery(table)  # batch: drop triggering_frequency","handlingStrategy":"validation","validationCode":"if not is_streaming and triggering_frequency is not None:\n    raise ValueError('triggering_frequency is only valid in streaming pipelines')","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Gate triggering_frequency on the pipeline's streaming flag","Avoid sharing one transform config between batch and streaming jobs","Review copy-pasted pipeline options when repurposing jobs"],"tags":["bigquery","batch","beam","conflicting-options"],"backgroundTag":"conflicting-config-options","analyzedSha":"12126d8942aaf848030c478b4c6a28c6af861c66","analyzedAt":"2026-09-13T01:50:10.254Z","contentChangedAt":"2026-09-13T01:50:10.254Z","schemaVersion":2},"datasetVersion":"2026-09-20T03:17:13.778Z"}