{"record":{"id":"416961f3e66572e3","repo":"apache/beam","slug":"unable-save-rows-in-bigquery","errorCode":null,"errorMessage":"Unable save rows in BigQuery: {}","messagePattern":"Unable save rows in BigQuery: (.+?)","errorType":"exception","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"sdks/python/apache_beam/testing/load_tests/load_test_metrics_utils.py","lineNumber":465,"sourceCode":"    else:\n      _LOGGER.info(\"No test results were collected.\")\n\n\nclass BigQueryMetricsPublisher(MetricsPublisher):\n  \"\"\"A :class:`BigQueryMetricsPublisher` publishes collected metrics\n  to BigQuery output.\"\"\"\n  def __init__(self, project_name, table, dataset, bq_schema=None):\n    if not bq_schema:\n      bq_schema = SCHEMA\n    self.bq = BigQueryClient(project_name, table, dataset, bq_schema)\n\n  def publish(self, results):\n    outputs = self.bq.save(results)\n    if len(outputs) > 0:\n      for output in outputs:\n        if output['errors']:\n          _LOGGER.error(output)\n          raise ValueError(\n              'Unable save rows in BigQuery: {}'.format(output['errors']))\n\n\nclass BigQueryClient(object):\n  \"\"\"A :class:`BigQueryClient` publishes collected metrics to\n  BigQuery output.\"\"\"\n  def __init__(self, project_name, table, dataset, bq_schema=None):\n    self.schema = bq_schema\n    self._namespace = table\n    self._client = bigquery.Client(project=project_name)\n    self._schema_names = self._get_schema_names()\n    schema = self._prepare_schema()\n    self._get_or_create_table(schema, dataset)\n\n  def _get_schema_names(self):\n    return [schema['name'] for schema in self.schema]\n\n  def _prepare_schema(self):","sourceCodeStart":447,"sourceCodeEnd":483,"githubUrl":"https://github.com/apache/beam/blob/12126d8942aaf848030c478b4c6a28c6af861c66/sdks/python/apache_beam/testing/load_tests/load_test_metrics_utils.py#L447-L483","documentation":"Raised by the load-test MetricsPublisher when BigQuery's insert_all returns rows with per-row errors. The publisher calls self.bq.save(results), and if any returned row contains a non-empty 'errors' key it logs the full output and raises ValueError. It signals that some metric rows failed to persist to the BigQuery sink.","triggerScenarios":"Calling publish(results) when BigQuery insert_all rejects one or more rows — typically schema mismatches (missing/extra fields, wrong types vs the configured schema), rows exceeding size limits, or invalid table state.","commonSituations":"Load-test config schema diverges from the actual BigQuery table schema after a metrics format change; metric values serialized with wrong types; streaming insert quota or malformed-row rejections during long load-test runs.","solutions":["Inspect the logged output row to see the exact per-row error message from BigQuery","Compare the row's fields/types against the table schema returned by _prepare_schema/get_table and fix the producer to emit matching fields","Recreate or update the BigQuery table so its schema matches the current metrics format","Re-run the load test and confirm outputs contains no rows with 'errors'"],"exampleFix":"# before: rows emitted with wrong field\n{'metric': 'latency_ms', 'value': '123'}  # schema expects INTEGER\n// after\n{'metric': 'latency_ms', 'value': 123}","handlingStrategy":"try-catch","validationCode":"def rows_valid(results, schema_fields):\n    required = {f['name'] for f in schema_fields}\n    return all(required.issubset(r.keys()) for r in results)","typeGuard":"def is_valid_output(output):\n    return isinstance(output, dict) and not output.get('errors')","tryCatchPattern":"try:\n    publisher.publish(results)\nexcept ValueError as e:\n    _LOGGER.error('BigQuery row insert failed: %s', e)\n    # inspect failed rows, fix schema/types, retry with backoff","preventionTips":["Keep the metrics schema and BigQuery table schema in sync","Validate row field types before publishing","Monitor per-row insert errors in load-test logs"],"tags":["bigquery","python","load-testing","data-persistence"],"backgroundTag":"database-write-failed","analyzedSha":"12126d8942aaf848030c478b4c6a28c6af861c66","analyzedAt":"2026-09-13T01:50:10.254Z","contentChangedAt":"2026-09-13T01:50:10.254Z","schemaVersion":2},"datasetVersion":"2026-09-20T03:17:13.778Z"}