{"record":{"id":"3d7b5c812a2cde7d","repo":"apache/beam","slug":"bigquery-write-error-len-d-size-d","errorCode":null,"errorMessage":"bigquery write error [len=%d, size=%d]","messagePattern":"bigquery write error \\[len=(.+?), size=(.+?)\\]","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"sdks/go/pkg/beam/io/bigqueryio/bigquery.go","lineNumber":383,"sourceCode":"\t\tif err := table.Create(ctx, &bigquery.TableMetadata{Schema: schema}); err != nil {\n\t\t\treturn err\n\t\t}\n\t}\n\n\tvar data []reflect.Value\n\t// This stores the running byte size estimate of a BQ request.\n\tsize := writeOverheadBytes\n\n\tvar val beam.X\n\tfor iter(&val) {\n\t\tcurrent, err := getInsertSize(val.(any), schema)\n\t\tif err != nil {\n\t\t\treturn errors.Wrapf(err, \"bigquery write error\")\n\t\t}\n\t\tif len(data)+1 > writeRowLimit || size+current > writeSizeLimit {\n\t\t\t// Write rows in batches to comply with BQ limits.\n\t\t\tif err := put(ctx, table, f.Type.T, data); err != nil {\n\t\t\t\treturn errors.Wrapf(err, \"bigquery write error [len=%d, size=%d]\", len(data), size)\n\t\t\t}\n\t\t\tdata = nil\n\t\t\tsize = writeOverheadBytes\n\t\t}\n\t\tdata = append(data, reflect.ValueOf(val.(any)))\n\t\tsize += current\n\t}\n\tif len(data) == 0 {\n\t\treturn nil\n\t}\n\tif err := put(ctx, table, f.Type.T, data); err != nil {\n\t\treturn errors.Wrapf(err, \"bigquery write error [len=%d, size=%d]\", len(data), size)\n\t}\n\treturn nil\n}\n\nfunc put(ctx context.Context, table *bigquery.Table, t reflect.Type, data []reflect.Value) error {\n\t// list : []T to allow Put to infer the schema","sourceCodeStart":365,"sourceCodeEnd":401,"githubUrl":"https://github.com/apache/beam/blob/12126d8942aaf848030c478b4c6a28c6af861c66/sdks/go/pkg/beam/io/bigqueryio/bigquery.go#L365-L401","documentation":"When the accumulating batch exceeds BigQuery's writeRowLimit (rows per insert) or writeSizeLimit (bytes per insert), the DoFn flushes the batch via put(), which streams rows to the BigQuery insert API. If that flush fails, ProcessElement returns \"bigquery write error [len=%d, size=%d]\" including the batch row count and byte size, wrapping the BigQuery API error.","triggerScenarios":"put(ctx, table, f.Type.T, data) returns an error on a mid-stream batch flush: rows violate table schema, table missing, quota/billing issues, or transient BigQuery insert failures on batches over the row/size limits.","commonSituations":"High-throughput pipelines hitting streaming-insert quotas; schema mismatch between the struct and the destination table; dataset/table deleted or renamed while the pipeline runs; rows with invalid field values (e.g. out-of-range timestamps).","solutions":["Read the wrapped BigQuery API error; if it names bad rows, fix or filter those rows upstream.","Verify the destination table exists and its schema matches the element type (or pin it with WithSchema).","Retry the pipeline / rely on Beam's retry for transient quota errors, and request a streaming-inserts quota increase if limits are hit repeatedly.","Check project billing and BigQuery API enablement if errors indicate service-level rejection."],"exampleFix":null,"handlingStrategy":"retry","validationCode":null,"typeGuard":null,"tryCatchPattern":"if err := put(ctx, table, typ, batch); err != nil {\n\tvar bqErr *googleapi.Error\n\tif errors.As(err, &bqErr) && bqErr.Code >= 500 {\n\t\t// transient: retry batch with backoff\n\t}\n}","preventionTips":["Verify the destination table schema matches before launching the job.","Monitor BigQuery streaming-insert quotas; request increases for high-volume sinks.","Check table/dataset existence and IAM dataEditor rights at pipeline startup."],"tags":["bigquery","write","quota","go"],"backgroundTag":"database-write-failed","analyzedSha":"12126d8942aaf848030c478b4c6a28c6af861c66","analyzedAt":"2026-09-13T01:50:10.254Z","contentChangedAt":"2026-09-13T01:50:10.254Z","schemaVersion":2},"datasetVersion":"2026-09-20T03:17:13.778Z"}