apache/beam · error · ValueError

Unable save rows in BigQuery

Error message

Unable save rows in BigQuery: {}

What it means

Raised by the load-test MetricsPublisher when BigQuery's insert_all returns rows with per-row errors. The publisher calls self.bq.save(results), and if any returned row contains a non-empty 'errors' key it logs the full output and raises ValueError. It signals that some metric rows failed to persist to the BigQuery sink.

Solutions

1. Inspect the logged output row to see the exact per-row error message from BigQuery
2. Compare the row's fields/types against the table schema returned by _prepare_schema/get_table and fix the producer to emit matching fields
3. Recreate or update the BigQuery table so its schema matches the current metrics format
4. Re-run the load test and confirm outputs contains no rows with 'errors'

Example fix

# before: rows emitted with wrong field
{'metric': 'latency_ms', 'value': '123'}  # schema expects INTEGER
// after
{'metric': 'latency_ms', 'value': 123}

def rows_valid(results, schema_fields):
    required = {f['name'] for f in schema_fields}
    return all(required.issubset(r.keys()) for r in results)

def is_valid_output(output):
    return isinstance(output, dict) and not output.get('errors')

try:
    publisher.publish(results)
except ValueError as e:
    _LOGGER.error('BigQuery row insert failed: %s', e)
    # inspect failed rows, fix schema/types, retry with backoff

When it happens

Trigger: Calling publish(results) when BigQuery insert_all rejects one or more rows — typically schema mismatches (missing/extra fields, wrong types vs the configured schema), rows exceeding size limits, or invalid table state.

Common situations: Load-test config schema diverges from the actual BigQuery table schema after a metrics format change; metric values serialized with wrong types; streaming insert quota or malformed-row rejections during long load-test runs.

Related errors

• Bigquery dependencies are not installed.
• test_name not provided in config.
• A BigQuery table or a query must be specified
• AvroRowWriter is not readable
• Bigquery dependencies are not installed.

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/416961f3e66572e3. Report an issue: GitHub.

Appendix: source

Thrown at sdks/python/apache_beam/testing/load_tests/load_test_metrics_utils.py:465

    else:
      _LOGGER.info("No test results were collected.")


class BigQueryMetricsPublisher(MetricsPublisher):
  """A :class:`BigQueryMetricsPublisher` publishes collected metrics
  to BigQuery output."""
  def __init__(self, project_name, table, dataset, bq_schema=None):
    if not bq_schema:
      bq_schema = SCHEMA
    self.bq = BigQueryClient(project_name, table, dataset, bq_schema)

  def publish(self, results):
    outputs = self.bq.save(results)
    if len(outputs) > 0:
      for output in outputs:
        if output['errors']:
          _LOGGER.error(output)
          raise ValueError(
              'Unable save rows in BigQuery: {}'.format(output['errors']))


class BigQueryClient(object):
  """A :class:`BigQueryClient` publishes collected metrics to
  BigQuery output."""
  def __init__(self, project_name, table, dataset, bq_schema=None):
    self.schema = bq_schema
    self._namespace = table
    self._client = bigquery.Client(project=project_name)
    self._schema_names = self._get_schema_names()
    schema = self._prepare_schema()
    self._get_or_create_table(schema, dataset)

  def _get_schema_names(self):
    return [schema['name'] for schema in self.schema]

  def _prepare_schema(self):

View on GitHub (pinned to 12126d8942)