{"record":{"id":"244d4418a52fe62a","repo":"apache/beam","slug":"max-retries-cannot-be-more-than-10000-hence-please-reduce","errorCode":null,"errorMessage":"max_retries cannot be more than 10000, hence please reduce the value.","messagePattern":"max_retries cannot be more than 10000, hence please reduce the value\\.","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"sdks/python/apache_beam/io/gcp/bigquery.py","lineNumber":2376,"sourceCode":"    if method_to_use == WriteToBigQuery.Method.STREAMING_INSERTS:\n      if self.schema == SCHEMA_AUTODETECT:\n        raise ValueError(\n            'Schema auto-detection is not supported for streaming '\n            'inserts into BigQuery. Only for File Loads.')\n\n      if self.triggering_frequency is not None and not self.with_auto_sharding:\n        raise ValueError(\n            'triggering_frequency with STREAMING_INSERTS can only be used with '\n            'with_auto_sharding=True.')\n\n      if self._max_insert_payload_size > MAX_INSERT_PAYLOAD_SIZE:\n        raise ValueError(\n            'max_insert_payload_size can only go up to '\n            f'{MAX_INSERT_PAYLOAD_SIZE} bytes, as per BigQuery quota limits: '\n            'https://cloud.google.com/bigquery/quotas#streaming_inserts.')\n\n      if self._max_retries > MAX_INSERT_RETRIES:\n        raise ValueError(\n            'max_retries cannot be more than '\n            f'{MAX_INSERT_RETRIES}, hence please reduce the value.')\n\n      outputs = pcoll | _StreamToBigQuery(\n          table_reference=self.table_reference,\n          table_side_inputs=self.table_side_inputs,\n          schema_side_inputs=self.schema_side_inputs,\n          schema=self.schema,\n          batch_size=self.batch_size,\n          triggering_frequency=self.triggering_frequency,\n          create_disposition=self.create_disposition,\n          write_disposition=self.write_disposition,\n          kms_key=self.kms_key,\n          retry_strategy=self.insert_retry_strategy,\n          additional_bq_parameters=self.additional_bq_parameters,\n          ignore_insert_ids=self._ignore_insert_ids,\n          ignore_unknown_columns=self._ignore_unknown_columns,\n          with_auto_sharding=self.with_auto_sharding,","sourceCodeStart":2358,"sourceCodeEnd":2394,"githubUrl":"https://github.com/apache/beam/blob/12126d8942aaf848030c478b4c6a28c6af861c66/sdks/python/apache_beam/io/gcp/bigquery.py#L2358-L2394","documentation":"WriteToBigQuery.expand() rejects max_retries values above MAX_INSERT_RETRIES (10000) for streaming inserts. The library caps retries because beyond this the insert is effectively never going to succeed and would consume unbounded resources.","triggerScenarios":"Instantiating WriteToBigQuery with method=STREAMING_INSERTS and max_retries=N where N > 10000, then expanding the transform.","commonSituations":"Developers set a very large retry count (e.g. 1000000) to 'never give up' on transient streaming-insert failures; mistaken use of milliseconds/other units.","solutions":["Set max_retries to a value <= 10000.","Omit max_retries to use the default retry limit.","Handle persistent failures via failed_rows output (FAILED_ROWS_WITH_ERRORS) instead of unbounded retries."],"exampleFix":"// before\nWriteToBigQuery(table='proj:ds.tbl', method='STREAMING_INSERTS', max_retries=100000)\n// after\nWriteToBigQuery(table='proj:ds.tbl', method='STREAMING_INSERTS', max_retries=10000)","handlingStrategy":"validation","validationCode":"MAX_INSERT_RETRIES = 10000\nif max_retries > MAX_INSERT_RETRIES:\n    raise ValueError('max_retries must be <= 10000')","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Use modest retry counts and route permanent failures to the FAILED_ROWS_WITH_ERRORS output.","Remember max_retries is a count, not a duration."],"tags":["apache-beam","bigquery","config-validation","retry-limit"],"backgroundTag":"value-out-of-range","analyzedSha":"12126d8942aaf848030c478b4c6a28c6af861c66","analyzedAt":"2026-09-13T01:50:10.254Z","contentChangedAt":"2026-09-13T01:50:10.254Z","schemaVersion":2},"datasetVersion":"2026-09-14T16:17:12.679Z"}