{"record":{"id":"46cbee477ec06d16","repo":"apache/beam","slug":"max-insert-payload-size-can-only-go-up-to-10485760-bytes-as","errorCode":null,"errorMessage":"max_insert_payload_size can only go up to 10485760 bytes, as per BigQuery quota limits: https://cloud.google.com/bigquery/quotas#streaming_inserts.","messagePattern":"max_insert_payload_size can only go up to 10485760 bytes, as per BigQuery quota limits: https://cloud\\.google\\.com/bigquery/quotas#streaming_inserts\\.","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"sdks/python/apache_beam/io/gcp/bigquery.py","lineNumber":2370,"sourceCode":"    if (self.schema_update_options is not None and\n        method_to_use != WriteToBigQuery.Method.FILE_LOADS):\n      raise ValueError(\n          'schema_update_options is only supported when writing to BigQuery '\n          'with FILE_LOADS.')\n\n    if method_to_use == WriteToBigQuery.Method.STREAMING_INSERTS:\n      if self.schema == SCHEMA_AUTODETECT:\n        raise ValueError(\n            'Schema auto-detection is not supported for streaming '\n            'inserts into BigQuery. Only for File Loads.')\n\n      if self.triggering_frequency is not None and not self.with_auto_sharding:\n        raise ValueError(\n            'triggering_frequency with STREAMING_INSERTS can only be used with '\n            'with_auto_sharding=True.')\n\n      if self._max_insert_payload_size > MAX_INSERT_PAYLOAD_SIZE:\n        raise ValueError(\n            'max_insert_payload_size can only go up to '\n            f'{MAX_INSERT_PAYLOAD_SIZE} bytes, as per BigQuery quota limits: '\n            'https://cloud.google.com/bigquery/quotas#streaming_inserts.')\n\n      if self._max_retries > MAX_INSERT_RETRIES:\n        raise ValueError(\n            'max_retries cannot be more than '\n            f'{MAX_INSERT_RETRIES}, hence please reduce the value.')\n\n      outputs = pcoll | _StreamToBigQuery(\n          table_reference=self.table_reference,\n          table_side_inputs=self.table_side_inputs,\n          schema_side_inputs=self.schema_side_inputs,\n          schema=self.schema,\n          batch_size=self.batch_size,\n          triggering_frequency=self.triggering_frequency,\n          create_disposition=self.create_disposition,\n          write_disposition=self.write_disposition,","sourceCodeStart":2352,"sourceCodeEnd":2388,"githubUrl":"https://github.com/apache/beam/blob/12126d8942aaf848030c478b4c6a28c6af861c66/sdks/python/apache_beam/io/gcp/bigquery.py#L2352-L2388","documentation":"WriteToBigQuery.expand() validates that max_insert_payload_size does not exceed MAX_INSERT_PAYLOAD_SIZE (10485760 bytes / 10MB), the BigQuery streaming-insert quota per batch. The library throws ValueError eagerly at pipeline construction time rather than failing at runtime when a batch exceeds BigQuery's limit.","triggerScenarios":"Creating WriteToBigQuery(method=WriteToBigQuery.Method.STREAMING_INSERTS, max_insert_payload_size=N) with N > 10485760, then calling expand() on the transform.","commonSituations":"Developers try to increase the payload size to reduce insert batch counts, unaware of BigQuery's hard streaming-insert quota; copied configs from other systems with larger batch limits.","solutions":["Set max_insert_payload_size to at most 10485760 (the default).","Remove the custom max_insert_payload_size argument to use the quota-compliant default.","If you need higher throughput, use multiple workers/shards instead of larger payloads."],"exampleFix":"// before\nWriteToBigQuery(table='proj:ds.tbl', method='STREAMING_INSERTS', max_insert_payload_size=20*1024*1024)\n// after\nWriteToBigQuery(table='proj:ds.tbl', method='STREAMING_INSERTS', max_insert_payload_size=10*1024*1024)","handlingStrategy":"validation","validationCode":"MAX_INSERT_PAYLOAD_SIZE = 10485760\nif max_insert_payload_size > MAX_INSERT_PAYLOAD_SIZE:\n    raise ValueError('max_insert_payload_size must be <= 10485760')","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Keep max_insert_payload_size at its default (10MB).","Scale throughput by adding workers, not by growing batch size."],"tags":["apache-beam","bigquery","config-validation","value-out-of-range"],"backgroundTag":"value-out-of-range","analyzedSha":"12126d8942aaf848030c478b4c6a28c6af861c66","analyzedAt":"2026-09-13T01:50:10.254Z","contentChangedAt":"2026-09-13T01:50:10.254Z","schemaVersion":2},"datasetVersion":"2026-09-14T16:17:12.679Z"}