apache/beam · error · ValueError

poll_interval_sec must be >= 15, got {poll_interval_sec}

Error message

poll_interval_sec must be >= 15, got {poll_interval_sec}

What it means

The change-history poller enforces a minimum poll interval of 15 seconds, because BigQuery's change history has latency and polling more aggressively wastes API quota without yielding new data. A smaller poll_interval_sec raises ValueError at construction time.

Source

Thrown at sdks/python/apache_beam/io/gcp/bigquery_change_history.py:1218

      batch_arrow_read: bool = True,
      max_split_rounds: int = 1,
      reshuffle_decompress: bool = True) -> None:
    super().__init__()
    if bq_storage is None:
      raise ImportError(
          'google-cloud-bigquery-storage is required for '
          'ReadBigQueryChangeHistory. Install it with: '
          'pip install google-cloud-bigquery-storage')
    if pyarrow is None:
      raise ImportError(
          'pyarrow is required for ReadBigQueryChangeHistory. '
          'Install it with: pip install pyarrow')
    if change_function not in ('CHANGES', 'APPENDS'):
      raise ValueError(
          f"change_function must be 'CHANGES' or 'APPENDS', "
          f"got '{change_function}'")
    if poll_interval_sec < 15:
      raise ValueError(
          f'poll_interval_sec must be >= 15, got {poll_interval_sec}')
    if buffer_sec < 0:
      raise ValueError(f'buffer_sec must be >= 0, got {buffer_sec}')
    self._table = table
    self._poll_interval_sec = poll_interval_sec
    self._start_time = start_time
    self._stop_time = stop_time
    self._change_function = change_function
    self._buffer_sec = buffer_sec
    self._project = project
    self._temp_dataset = temp_dataset
    self._location = location
    self._change_type_column = change_type_column
    self._change_timestamp_column = change_timestamp_column
    self._columns = columns
    self._row_filter = row_filter
    self._batch_arrow_read = batch_arrow_read
    self._max_split_rounds = max_split_rounds

View on GitHub (pinned to 12126d8942)

Solutions

Set poll_interval_sec to at least 15 (e.g. poll_interval_sec=30).
If you need lower end-to-end latency, consider the arrow-batch/streaming read paths instead of faster polling.
Omit the parameter to use the transform's default interval if one exists.

Example fix

// before
ReadFromBigQueryChangeHistory(..., poll_interval_sec=5)
// after
ReadFromBigQueryChangeHistory(..., poll_interval_sec=30)

Defensive patterns

Strategy: validation

Validate before calling

if poll_interval_sec < 15:
    poll_interval_sec = 15  # or raise early with a clear message

Try / catch

try:
    transform = ReadFromBigQueryChangeHistory(..., poll_interval_sec=p)
except ValueError as e:
    log.error("invalid poll_interval_sec: %s", e)

Prevention

Clamp poll_interval_sec to >= 15 when sourced from pipeline options
Do not try to lower latency below the API-supported poll floor
Use 30-60s intervals in production to respect BigQuery quota

When it happens

Trigger: Constructing ReadFromBigQueryChangeHistory with poll_interval_sec < 15 (e.g. poll_interval_sec=5 or 0) in a streaming pipeline.

Common situations: Trying to reduce latency by tightening the poll interval; passing 0 or None-like values expecting defaults; unit-test configs with tiny intervals.

Understand the failure class

Background: "Invalid value" and "allowed values are" config errors: what your library rejected and how to fix it — this error's family across 41 libraries.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/2cff9863b7b1a6f7. Report an issue: GitHub.