{"record":{"id":"240a673da3d759b6","repo":"apache/beam","slug":"bigquery-write-error","errorCode":null,"errorMessage":"bigquery write error","messagePattern":"bigquery write error","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"sdks/go/pkg/beam/io/bigqueryio/bigquery.go","lineNumber":378,"sourceCode":"\t\t\treturn err\n\t\t}\n\t\tif f.Options.CreateDisposition == bigquery.CreateNever {\n\t\t\treturn fmt.Errorf(\"table does not exist and create disposition is 'CreateNever': %v\", err)\n\t\t}\n\t\tif err := table.Create(ctx, &bigquery.TableMetadata{Schema: schema}); err != nil {\n\t\t\treturn err\n\t\t}\n\t}\n\n\tvar data []reflect.Value\n\t// This stores the running byte size estimate of a BQ request.\n\tsize := writeOverheadBytes\n\n\tvar val beam.X\n\tfor iter(&val) {\n\t\tcurrent, err := getInsertSize(val.(any), schema)\n\t\tif err != nil {\n\t\t\treturn errors.Wrapf(err, \"bigquery write error\")\n\t\t}\n\t\tif len(data)+1 > writeRowLimit || size+current > writeSizeLimit {\n\t\t\t// Write rows in batches to comply with BQ limits.\n\t\t\tif err := put(ctx, table, f.Type.T, data); err != nil {\n\t\t\t\treturn errors.Wrapf(err, \"bigquery write error [len=%d, size=%d]\", len(data), size)\n\t\t\t}\n\t\t\tdata = nil\n\t\t\tsize = writeOverheadBytes\n\t\t}\n\t\tdata = append(data, reflect.ValueOf(val.(any)))\n\t\tsize += current\n\t}\n\tif len(data) == 0 {\n\t\treturn nil\n\t}\n\tif err := put(ctx, table, f.Type.T, data); err != nil {\n\t\treturn errors.Wrapf(err, \"bigquery write error [len=%d, size=%d]\", len(data), size)\n\t}","sourceCodeStart":360,"sourceCodeEnd":396,"githubUrl":"https://github.com/apache/beam/blob/12126d8942aaf848030c478b4c6a28c6af861c66/sdks/go/pkg/beam/io/bigqueryio/bigquery.go#L360-L396","documentation":"In bigqueryio's insert-batching DoFn, each row's insert size is computed with getInsertSize before it is added to the current batch. If computing a row's size fails (the value cannot be inspected/serialized for the given schema), ProcessElement aborts the whole element with \"bigquery write error\" wrapping the underlying cause. This signals the row itself is incompatible with the inferred/declared schema.","triggerScenarios":"Iterating elements in the batching DoFn when getInsertSize(val.(any), schema) returns an error — e.g. a value whose dynamic type does not match the schema established for the write.","commonSituations":"PCollection element type drifted from the configured schema after a pipeline change; nested struct fields that cannot be measured against the BigQuery schema; feeding a different type into a sink built for another type.","solutions":["Read the wrapped error to identify which field/value fails schema conformance, and fix the producing PCollection's element type.","Supply an explicit schema via WithSchema that matches the actual element struct.","Add validation (or a filter/ParDo) upstream so only schema-conformant rows reach the BigQuery sink.","Regenerate or update the row struct after any schema change so type and schema stay in sync."],"exampleFix":"// before\nbigqueryio.Write(s, proj, ds, tbl, bigqueryio.WriteParams()) // schema no longer matches struct\n\n// after\nbigqueryio.Write(s, proj, ds, tbl, bigqueryio.WithSchema(bigquery.Schema{ {Name: \"Id\", Type: bigquery.IntegerFieldType}, {Name: \"Name\", Type: bigquery.StringFieldType} }))","handlingStrategy":"try-catch","validationCode":null,"typeGuard":null,"tryCatchPattern":"if err := bigqueryio.Write(scope, proj, ds, tbl, ...); err != nil {\n\tlog.Printf(\"bigquery write failed (row size check): %v\", err)\n\t// route to dead-letter or fail the pipeline\n}","preventionTips":["Keep the element struct and the write schema in lockstep; change both together.","Validate rows against the schema in an upstream ParDo before the sink.","Add a dead-letter output for schema-incompatible rows."],"tags":["bigquery","write","schema","go"],"backgroundTag":"database-write-failed","analyzedSha":"12126d8942aaf848030c478b4c6a28c6af861c66","analyzedAt":"2026-09-13T01:50:10.254Z","contentChangedAt":"2026-09-13T01:50:10.254Z","schemaVersion":2},"datasetVersion":"2026-09-20T03:17:13.778Z"}