apache/beam · error · ValueError

max_retries cannot be more than 10000, hence please reduce t

Error message

max_retries cannot be more than 10000, hence please reduce the value.

What it means

WriteToBigQuery.expand() rejects max_retries values above MAX_INSERT_RETRIES (10000) for streaming inserts. The library caps retries because beyond this the insert is effectively never going to succeed and would consume unbounded resources.

Source

Thrown at sdks/python/apache_beam/io/gcp/bigquery.py:2376

    if method_to_use == WriteToBigQuery.Method.STREAMING_INSERTS:
      if self.schema == SCHEMA_AUTODETECT:
        raise ValueError(
            'Schema auto-detection is not supported for streaming '
            'inserts into BigQuery. Only for File Loads.')

      if self.triggering_frequency is not None and not self.with_auto_sharding:
        raise ValueError(
            'triggering_frequency with STREAMING_INSERTS can only be used with '
            'with_auto_sharding=True.')

      if self._max_insert_payload_size > MAX_INSERT_PAYLOAD_SIZE:
        raise ValueError(
            'max_insert_payload_size can only go up to '
            f'{MAX_INSERT_PAYLOAD_SIZE} bytes, as per BigQuery quota limits: '
            'https://cloud.google.com/bigquery/quotas#streaming_inserts.')

      if self._max_retries > MAX_INSERT_RETRIES:
        raise ValueError(
            'max_retries cannot be more than '
            f'{MAX_INSERT_RETRIES}, hence please reduce the value.')

      outputs = pcoll | _StreamToBigQuery(
          table_reference=self.table_reference,
          table_side_inputs=self.table_side_inputs,
          schema_side_inputs=self.schema_side_inputs,
          schema=self.schema,
          batch_size=self.batch_size,
          triggering_frequency=self.triggering_frequency,
          create_disposition=self.create_disposition,
          write_disposition=self.write_disposition,
          kms_key=self.kms_key,
          retry_strategy=self.insert_retry_strategy,
          additional_bq_parameters=self.additional_bq_parameters,
          ignore_insert_ids=self._ignore_insert_ids,
          ignore_unknown_columns=self._ignore_unknown_columns,
          with_auto_sharding=self.with_auto_sharding,

View on GitHub (pinned to 12126d8942)

Solutions

Set max_retries to a value <= 10000.
Omit max_retries to use the default retry limit.
Handle persistent failures via failed_rows output (FAILED_ROWS_WITH_ERRORS) instead of unbounded retries.

Example fix

// before
WriteToBigQuery(table='proj:ds.tbl', method='STREAMING_INSERTS', max_retries=100000)
// after
WriteToBigQuery(table='proj:ds.tbl', method='STREAMING_INSERTS', max_retries=10000)

Defensive patterns

Strategy: validation

Validate before calling

MAX_INSERT_RETRIES = 10000
if max_retries > MAX_INSERT_RETRIES:
    raise ValueError('max_retries must be <= 10000')

Prevention

Use modest retry counts and route permanent failures to the FAILED_ROWS_WITH_ERRORS output.
Remember max_retries is a count, not a duration.

When it happens

Trigger: Instantiating WriteToBigQuery with method=STREAMING_INSERTS and max_retries=N where N > 10000, then expanding the transform.

Common situations: Developers set a very large retry count (e.g. 1000000) to 'never give up' on transient streaming-insert failures; mistaken use of milliseconds/other units.

Understand the failure class

Background: "value must be between 0 and 1" / "out of range" / "must not be negative" errors: fixing range-validation failures across open-source libraries — this error's family across 42 libraries.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/244d4418a52fe62a. Report an issue: GitHub.