apache/beam · error

bigquery write error

Error message

bigquery write error

What it means

In bigqueryio's insert-batching DoFn, each row's insert size is computed with getInsertSize before it is added to the current batch. If computing a row's size fails (the value cannot be inspected/serialized for the given schema), ProcessElement aborts the whole element with "bigquery write error" wrapping the underlying cause. This signals the row itself is incompatible with the inferred/declared schema.

Solutions

1. Read the wrapped error to identify which field/value fails schema conformance, and fix the producing PCollection's element type.
2. Supply an explicit schema via WithSchema that matches the actual element struct.
3. Add validation (or a filter/ParDo) upstream so only schema-conformant rows reach the BigQuery sink.
4. Regenerate or update the row struct after any schema change so type and schema stay in sync.

Example fix

// before
bigqueryio.Write(s, proj, ds, tbl, bigqueryio.WriteParams()) // schema no longer matches struct

// after
bigqueryio.Write(s, proj, ds, tbl, bigqueryio.WithSchema(bigquery.Schema{ {Name: "Id", Type: bigquery.IntegerFieldType}, {Name: "Name", Type: bigquery.StringFieldType} }))

if err := bigqueryio.Write(scope, proj, ds, tbl, ...); err != nil {
	log.Printf("bigquery write failed (row size check): %v", err)
	// route to dead-letter or fail the pipeline
}

When it happens

Trigger: Iterating elements in the batching DoFn when getInsertSize(val.(any), schema) returns an error — e.g. a value whose dynamic type does not match the schema established for the write.

Common situations: PCollection element type drifted from the configured schema after a pipeline change; nested struct fields that cannot be measured against the BigQuery schema; feeding a different type into a sink built for another type.

Related errors

• bigquery write error
• invalid schema type
• Array of collection is not supported in BigQuery.
• beam.RegisterSchemaProvider: schema type provider for
• beam.RegisterSchemaProvider: unsupported type kind for…

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/240a673da3d759b6. Report an issue: GitHub.

Appendix: source

Thrown at sdks/go/pkg/beam/io/bigqueryio/bigquery.go:378

			return err
		}
		if f.Options.CreateDisposition == bigquery.CreateNever {
			return fmt.Errorf("table does not exist and create disposition is 'CreateNever': %v", err)
		}
		if err := table.Create(ctx, &bigquery.TableMetadata{Schema: schema}); err != nil {
			return err
		}
	}

	var data []reflect.Value
	// This stores the running byte size estimate of a BQ request.
	size := writeOverheadBytes

	var val beam.X
	for iter(&val) {
		current, err := getInsertSize(val.(any), schema)
		if err != nil {
			return errors.Wrapf(err, "bigquery write error")
		}
		if len(data)+1 > writeRowLimit || size+current > writeSizeLimit {
			// Write rows in batches to comply with BQ limits.
			if err := put(ctx, table, f.Type.T, data); err != nil {
				return errors.Wrapf(err, "bigquery write error [len=%d, size=%d]", len(data), size)
			}
			data = nil
			size = writeOverheadBytes
		}
		data = append(data, reflect.ValueOf(val.(any)))
		size += current
	}
	if len(data) == 0 {
		return nil
	}
	if err := put(ctx, table, f.Type.T, data); err != nil {
		return errors.Wrapf(err, "bigquery write error [len=%d, size=%d]", len(data), size)
	}

View on GitHub (pinned to 12126d8942)