{"record":{"id":"05457c7d709c4566","repo":"apache/beam","slug":"schema-auto-detection-is-not-supported-for-streaming-inserts","errorCode":null,"errorMessage":"Schema auto-detection is not supported for streaming inserts into BigQuery. Only for File Loads.","messagePattern":"Schema auto-detection is not supported for streaming inserts into BigQuery\\. Only for File Loads\\.","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"sdks/python/apache_beam/io/gcp/bigquery.py","lineNumber":2360,"sourceCode":"    # TODO(pabloem): Use a different method to determine if streaming or batch.\n    is_streaming_pipeline = p.options.view_as(StandardOptions).streaming\n\n    if not is_streaming_pipeline and self.with_auto_sharding:\n      raise ValueError(\n          'with_auto_sharding is not applicable to batch pipelines.')\n\n    experiments = p.options.view_as(DebugOptions).experiments or []\n    method_to_use = self._compute_method(experiments, is_streaming_pipeline)\n\n    if (self.schema_update_options is not None and\n        method_to_use != WriteToBigQuery.Method.FILE_LOADS):\n      raise ValueError(\n          'schema_update_options is only supported when writing to BigQuery '\n          'with FILE_LOADS.')\n\n    if method_to_use == WriteToBigQuery.Method.STREAMING_INSERTS:\n      if self.schema == SCHEMA_AUTODETECT:\n        raise ValueError(\n            'Schema auto-detection is not supported for streaming '\n            'inserts into BigQuery. Only for File Loads.')\n\n      if self.triggering_frequency is not None and not self.with_auto_sharding:\n        raise ValueError(\n            'triggering_frequency with STREAMING_INSERTS can only be used with '\n            'with_auto_sharding=True.')\n\n      if self._max_insert_payload_size > MAX_INSERT_PAYLOAD_SIZE:\n        raise ValueError(\n            'max_insert_payload_size can only go up to '\n            f'{MAX_INSERT_PAYLOAD_SIZE} bytes, as per BigQuery quota limits: '\n            'https://cloud.google.com/bigquery/quotas#streaming_inserts.')\n\n      if self._max_retries > MAX_INSERT_RETRIES:\n        raise ValueError(\n            'max_retries cannot be more than '\n            f'{MAX_INSERT_RETRIES}, hence please reduce the value.')","sourceCodeStart":2342,"sourceCodeEnd":2378,"githubUrl":"https://github.com/apache/beam/blob/12126d8942aaf848030c478b4c6a28c6af861c66/sdks/python/apache_beam/io/gcp/bigquery.py#L2342-L2378","documentation":"SCHEMA_AUTODETECT relies on BigQuery load jobs to infer the schema from the data files. Streaming inserts require the schema to be provided up front (each row insert carries typed values against a known table schema), so expand() raises ValueError when schema=SCHEMA_AUTODETECT is combined with STREAMING_INSERTS.","triggerScenarios":"WriteToBigQuery(method=STREAMING_INSERTS, schema=beam.io.bigquery.SCHEMA_AUTODETECT) — including implicit STREAMING_INSERTS selection in batch pipelines when schema_autodetect=True.","commonSituations":"Setting schema_autodetect=True from a config flag shared across pipelines; switching method to STREAMING_INSERTS while leaving SCHEMA_AUTODETECT in place from a file-loads setup.","solutions":["Provide an explicit schema (dict or JSON string) instead of SCHEMA_AUTODETECT.","Switch to method=FILE_LOADS where auto-detection is supported.","Gate schema_autodetect so it is only set when FILE_LOADS is chosen."],"exampleFix":"// before\nbeam.io.WriteToBigQuery(table, schema=SCHEMA_AUTODETECT, method='STREAMING_INSERTS')\n// after\nbeam.io.WriteToBigQuery(table, schema={'fields': [{'name': 'id', 'type': 'INTEGER'}]}, method='STREAMING_INSERTS')","handlingStrategy":"validation","validationCode":"if method == WriteToBigQuery.Method.STREAMING_INSERTS and schema == SCHEMA_AUTODETECT:\n    raise ValueError('provide an explicit schema for streaming inserts')","typeGuard":"def schema_provided(schema):\n    return schema is not None and schema != SCHEMA_AUTODETECT","tryCatchPattern":"try:\n    _ = beam.io.WriteToBigQuery(table, schema=schema, method='STREAMING_INSERTS')\nexcept ValueError:\n    schema = load_schema_from_config()","preventionTips":["Never combine SCHEMA_AUTODETECT with STREAMING_INSERTS","Only set schema_autodetect=True for FILE_LOADS pipelines","Require an explicit schema in streaming pipeline configs"],"tags":["bigquery","python","apache-beam","schema-autodetect"],"backgroundTag":"unsupported-operation","analyzedSha":"12126d8942aaf848030c478b4c6a28c6af861c66","analyzedAt":"2026-09-13T01:50:10.254Z","contentChangedAt":"2026-09-13T01:50:10.254Z","schemaVersion":2},"datasetVersion":"2026-09-20T03:17:13.778Z"}