apache/beam · error · ValueError

max_insert_payload_size can only go up to 10485760 bytes, as

Error message

max_insert_payload_size can only go up to 10485760 bytes, as per BigQuery quota limits: https://cloud.google.com/bigquery/quotas#streaming_inserts.

What it means

WriteToBigQuery.expand() validates that max_insert_payload_size does not exceed MAX_INSERT_PAYLOAD_SIZE (10485760 bytes / 10MB), the BigQuery streaming-insert quota per batch. The library throws ValueError eagerly at pipeline construction time rather than failing at runtime when a batch exceeds BigQuery's limit.

Source

Thrown at sdks/python/apache_beam/io/gcp/bigquery.py:2370

    if (self.schema_update_options is not None and
        method_to_use != WriteToBigQuery.Method.FILE_LOADS):
      raise ValueError(
          'schema_update_options is only supported when writing to BigQuery '
          'with FILE_LOADS.')

    if method_to_use == WriteToBigQuery.Method.STREAMING_INSERTS:
      if self.schema == SCHEMA_AUTODETECT:
        raise ValueError(
            'Schema auto-detection is not supported for streaming '
            'inserts into BigQuery. Only for File Loads.')

      if self.triggering_frequency is not None and not self.with_auto_sharding:
        raise ValueError(
            'triggering_frequency with STREAMING_INSERTS can only be used with '
            'with_auto_sharding=True.')

      if self._max_insert_payload_size > MAX_INSERT_PAYLOAD_SIZE:
        raise ValueError(
            'max_insert_payload_size can only go up to '
            f'{MAX_INSERT_PAYLOAD_SIZE} bytes, as per BigQuery quota limits: '
            'https://cloud.google.com/bigquery/quotas#streaming_inserts.')

      if self._max_retries > MAX_INSERT_RETRIES:
        raise ValueError(
            'max_retries cannot be more than '
            f'{MAX_INSERT_RETRIES}, hence please reduce the value.')

      outputs = pcoll | _StreamToBigQuery(
          table_reference=self.table_reference,
          table_side_inputs=self.table_side_inputs,
          schema_side_inputs=self.schema_side_inputs,
          schema=self.schema,
          batch_size=self.batch_size,
          triggering_frequency=self.triggering_frequency,
          create_disposition=self.create_disposition,
          write_disposition=self.write_disposition,

View on GitHub (pinned to 12126d8942)

Solutions

1. Set max_insert_payload_size to at most 10485760 (the default).
2. Remove the custom max_insert_payload_size argument to use the quota-compliant default.
3. If you need higher throughput, use multiple workers/shards instead of larger payloads.

Example fix

// before
WriteToBigQuery(table='proj:ds.tbl', method='STREAMING_INSERTS', max_insert_payload_size=20*1024*1024)
// after
WriteToBigQuery(table='proj:ds.tbl', method='STREAMING_INSERTS', max_insert_payload_size=10*1024*1024)

MAX_INSERT_PAYLOAD_SIZE = 10485760
if max_insert_payload_size > MAX_INSERT_PAYLOAD_SIZE:
    raise ValueError('max_insert_payload_size must be <= 10485760')

When it happens

Trigger: Creating WriteToBigQuery(method=WriteToBigQuery.Method.STREAMING_INSERTS, max_insert_payload_size=N) with N > 10485760, then calling expand() on the transform.

Common situations: Developers try to increase the payload size to reduce insert batch counts, unaware of BigQuery's hard streaming-insert quota; copied configs from other systems with larger batch limits.

Understand the failure class

Background: "value must be between 0 and 1" / "out of range" / "must not be negative" errors: fixing range-validation failures across open-source libraries — this error's family across 42 libraries.

Related errors

• max_retries cannot be more than 10000, hence please reduce t
• Unsupported format for BigQuery table path: '{linkedResource
• host must not be empty.
• port must be between 1 and 65535, but was .
• table must not be empty.

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/46cbee477ec06d16. Report an issue: GitHub.