{"record":{"id":"4013ef4636665316","repo":"apache/beam","slug":"schema-auto-detection-is-not-supported-when-using-avro-based","errorCode":null,"errorMessage":"Schema auto-detection is not supported when using Avro based file loads into BigQuery. Please specify a schema or set temp_file_format=\"NEWLINE_DELIMITED_JSON\"","messagePattern":"Schema auto-detection is not supported when using Avro based file loads into BigQuery\\. Please specify a schema or set temp_file_format=\"NEWLINE_DELIMITED_JSON\"","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"sdks/python/apache_beam/io/gcp/bigquery.py","lineNumber":2409,"sourceCode":"          additional_bq_parameters=self.additional_bq_parameters,\n          ignore_insert_ids=self._ignore_insert_ids,\n          ignore_unknown_columns=self._ignore_unknown_columns,\n          with_auto_sharding=self.with_auto_sharding,\n          test_client=self.test_client,\n          max_insert_payload_size=self._max_insert_payload_size,\n          max_retries=self._max_retries,\n          num_streaming_keys=self._num_streaming_keys)\n\n      return WriteResult(\n          method=WriteToBigQuery.Method.STREAMING_INSERTS,\n          failed_rows=outputs[BigQueryWriteFn.FAILED_ROWS],\n          failed_rows_with_errors=outputs[\n              BigQueryWriteFn.FAILED_ROWS_WITH_ERRORS])\n\n    elif method_to_use == WriteToBigQuery.Method.FILE_LOADS:\n      if self._temp_file_format == bigquery_tools.FileFormat.AVRO:\n        if self.schema == SCHEMA_AUTODETECT:\n          raise ValueError(\n              'Schema auto-detection is not supported when using Avro based '\n              'file loads into BigQuery. Please specify a schema or set '\n              'temp_file_format=\"NEWLINE_DELIMITED_JSON\"')\n        if self.schema is None:\n          raise ValueError(\n              'A schema must be provided when writing to BigQuery using '\n              'Avro based file loads')\n\n      if self.schema and type(self.schema) is dict:\n\n        def find_in_nested_dict(schema):\n          for field in schema['fields']:\n            if field['type'] == 'JSON':\n              logging.warning(\n                  'Found JSON type in TableSchema for \"File_LOADS\" write '\n                  'method. Make sure the TableSchema field is a parsed '\n                  'JSON to ensure the read as a JSON type. Otherwise it '\n                  'will read as a raw (escaped) string.')","sourceCodeStart":2391,"sourceCodeEnd":2427,"githubUrl":"https://github.com/apache/beam/blob/12126d8942aaf848030c478b4c6a28c6af861c66/sdks/python/apache_beam/io/gcp/bigquery.py#L2391-L2427","documentation":"When using FILE_LOADS with temp_file_format=AVRO, WriteToBigQuery.expand() rejects schema=SCHEMA_AUTODETECT because BigQuery Avro load jobs cannot auto-detect schemas (the Avro files themselves define the schema, and Beam cannot emit schemaless Avro).","triggerScenarios":"WriteToBigQuery(method=FILE_LOADS, schema=SCHEMA_AUTODETECT ('AUTO_DETECT'), temp_file_format=Avro) expanded on an untyped input.","commonSituations":"Switching a pipeline from JSON to Avro temp files (often for performance) while leaving schema='AUTO_DETECT' set from earlier config.","solutions":["Provide an explicit schema (string, dict, or table schema).","Switch temp_file_format='NEWLINE_DELIMITED_JSON' so auto-detection is allowed.","Let the schema be inferred from typed elements by passing a dict/Row type input and omitting the string 'AUTO_DETECT'."],"exampleFix":"// before\nWriteToBigQuery(table='proj:ds.tbl', method='FILE_LOADS', schema='AUTO_DETECT', temp_file_format='AVRO')\n// after\nWriteToBigQuery(table='proj:ds.tbl', method='FILE_LOADS', schema='col1:STRING,col2:INTEGER', temp_file_format='AVRO')","handlingStrategy":"validation","validationCode":"if temp_file_format == 'AVRO' and schema in (None, 'AUTO_DETECT', 'SCHEMA_AUTODETECT'):\n    raise ValueError('Provide a schema or use NEWLINE_DELIMITED_JSON')","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Always pair Avro temp files with an explicit schema.","Prefer passing typed PCollections so schemas are derived automatically."],"tags":["apache-beam","bigquery","avro","schema-validation"],"backgroundTag":"schema-validation-failed","analyzedSha":"12126d8942aaf848030c478b4c6a28c6af861c66","analyzedAt":"2026-09-13T01:50:10.254Z","contentChangedAt":"2026-09-13T01:50:10.254Z","schemaVersion":2},"datasetVersion":"2026-09-20T03:17:13.778Z"}