apache/beam · error

bigquery write error

Error message

bigquery write error [len=%d, size=%d]

What it means

When the accumulating batch exceeds BigQuery's writeRowLimit (rows per insert) or writeSizeLimit (bytes per insert), the DoFn flushes the batch via put(), which streams rows to the BigQuery insert API. If that flush fails, ProcessElement returns "bigquery write error [len=%d, size=%d]" including the batch row count and byte size, wrapping the BigQuery API error.

Solutions

1. Read the wrapped BigQuery API error; if it names bad rows, fix or filter those rows upstream.
2. Verify the destination table exists and its schema matches the element type (or pin it with WithSchema).
3. Retry the pipeline / rely on Beam's retry for transient quota errors, and request a streaming-inserts quota increase if limits are hit repeatedly.
4. Check project billing and BigQuery API enablement if errors indicate service-level rejection.

if err := put(ctx, table, typ, batch); err != nil {
	var bqErr *googleapi.Error
	if errors.As(err, &bqErr) && bqErr.Code >= 500 {
		// transient: retry batch with backoff
	}
}

When it happens

Trigger: put(ctx, table, f.Type.T, data) returns an error on a mid-stream batch flush: rows violate table schema, table missing, quota/billing issues, or transient BigQuery insert failures on batches over the row/size limits.

Common situations: High-throughput pipelines hitting streaming-insert quotas; schema mismatch between the struct and the destination table; dataset/table deleted or renamed while the pipeline runs; rows with invalid field values (e.g. out-of-range timestamps).

Related errors

• bigquery write error
• Array of collection is not supported in BigQuery.
• bigqueryio: query parameter
• bigqueryio.Read: type
• datamgr.go [ ]: error flushing buffer of length

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/3d7b5c812a2cde7d. Report an issue: GitHub.

Appendix: source

Thrown at sdks/go/pkg/beam/io/bigqueryio/bigquery.go:383

		if err := table.Create(ctx, &bigquery.TableMetadata{Schema: schema}); err != nil {
			return err
		}
	}

	var data []reflect.Value
	// This stores the running byte size estimate of a BQ request.
	size := writeOverheadBytes

	var val beam.X
	for iter(&val) {
		current, err := getInsertSize(val.(any), schema)
		if err != nil {
			return errors.Wrapf(err, "bigquery write error")
		}
		if len(data)+1 > writeRowLimit || size+current > writeSizeLimit {
			// Write rows in batches to comply with BQ limits.
			if err := put(ctx, table, f.Type.T, data); err != nil {
				return errors.Wrapf(err, "bigquery write error [len=%d, size=%d]", len(data), size)
			}
			data = nil
			size = writeOverheadBytes
		}
		data = append(data, reflect.ValueOf(val.(any)))
		size += current
	}
	if len(data) == 0 {
		return nil
	}
	if err := put(ctx, table, f.Type.T, data); err != nil {
		return errors.Wrapf(err, "bigquery write error [len=%d, size=%d]", len(data), size)
	}
	return nil
}

func put(ctx context.Context, table *bigquery.Table, t reflect.Type, data []reflect.Value) error {
	// list : []T to allow Put to infer the schema

View on GitHub (pinned to 12126d8942)