What causes "buffer_sec must be >= 0, got {buffer_sec}" in apache/beam?

Set buffer_sec to a non-negative number (e.g. buffer_sec=0 or a small positive value like 1-5 seconds).

apache/beam · error · ValueError

buffer_sec must be >= 0, got {buffer_sec}

Error message

buffer_sec must be >= 0, got {buffer_sec}

What it means

buffer_sec is a safety buffer subtracted/added around poll boundaries to avoid missing late-committing rows near the watermark. A negative value is meaningless and rejected with ValueError at construction time.

Source

Thrown at sdks/python/apache_beam/io/gcp/bigquery_change_history.py:1221

    super().__init__()
    if bq_storage is None:
      raise ImportError(
          'google-cloud-bigquery-storage is required for '
          'ReadBigQueryChangeHistory. Install it with: '
          'pip install google-cloud-bigquery-storage')
    if pyarrow is None:
      raise ImportError(
          'pyarrow is required for ReadBigQueryChangeHistory. '
          'Install it with: pip install pyarrow')
    if change_function not in ('CHANGES', 'APPENDS'):
      raise ValueError(
          f"change_function must be 'CHANGES' or 'APPENDS', "
          f"got '{change_function}'")
    if poll_interval_sec < 15:
      raise ValueError(
          f'poll_interval_sec must be >= 15, got {poll_interval_sec}')
    if buffer_sec < 0:
      raise ValueError(f'buffer_sec must be >= 0, got {buffer_sec}')
    self._table = table
    self._poll_interval_sec = poll_interval_sec
    self._start_time = start_time
    self._stop_time = stop_time
    self._change_function = change_function
    self._buffer_sec = buffer_sec
    self._project = project
    self._temp_dataset = temp_dataset
    self._location = location
    self._change_type_column = change_type_column
    self._change_timestamp_column = change_timestamp_column
    self._columns = columns
    self._row_filter = row_filter
    self._batch_arrow_read = batch_arrow_read
    self._max_split_rounds = max_split_rounds
    self._reshuffle_decompress = reshuffle_decompress

  def expand(self, pbegin: beam.pvalue.PBegin) -> beam.PCollection:

View on GitHub (pinned to 12126d8942)

Solutions

Set buffer_sec to a non-negative number (e.g. buffer_sec=0 or a small positive value like 1-5 seconds).
Use 0 if you want no buffer; positive values add safety margin for late commits.
If the value comes from a pipeline option, clamp it: max(0, option_value).

Example fix

// before
ReadFromBigQueryChangeHistory(..., buffer_sec=-5)
// after
ReadFromBigQueryChangeHistory(..., buffer_sec=0)

Defensive patterns

Strategy: validation

Validate before calling

if buffer_sec < 0:
    raise ValueError("buffer_sec must be >= 0")

Try / catch

try:
    transform = ReadFromBigQueryChangeHistory(..., buffer_sec=b)
except ValueError as e:
    log.error("invalid buffer_sec: %s", e)

Prevention

Clamp buffer_sec with max(0, value) when it comes from options or config
Use 0 for no buffer; positive values only for late-commit safety
Check signs when computing buffer_sec from other durations

When it happens

Trigger: Constructing ReadFromBigQueryChangeHistory with a negative buffer_sec (e.g. buffer_sec=-10).

Common situations: Sign errors when computing the buffer from another duration; misreading buffer_sec as an offset that may be negative; templated parameters defaulting to negative values.

Understand the failure class

Background: "value must be between 0 and 1" / "out of range" / "must not be negative" errors: fixing range-validation failures across open-source libraries — this error's family across 42 libraries.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/594f272a394e3858. Report an issue: GitHub.