{"record":{"id":"52a0c62a5cd57a73","repo":"apache/beam","slug":"we-have-observed-a-row-of-size-s-bytes-exceeding-the","errorCode":null,"errorMessage":"We have observed a row of size %s bytes exceeding the BigQueryIO limit of %s.","messagePattern":"We have observed a row of size (.+?) bytes exceeding the BigQueryIO limit of (.+?)\\.","errorType":"exception","errorClass":"RuntimeException","httpStatus":null,"severity":"error","filePath":"sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryServicesImpl.java","lineNumber":1217,"sourceCode":"                rowDetails = validateRowSchema(row, tableSchema);\n              }\n\n              // Basic log to return\n              String bqLimitLog =\n                  String.format(\n                      \"We have observed a row of size %s bytes exceeding the \"\n                          + \"BigQueryIO limit of %s.\",\n                      nextRowSize, MAX_BQ_ROW_PAYLOAD_DESC);\n\n              // Add on row schema diff details if present\n              if (!rowDetails.isEmpty()) {\n                bqLimitLog +=\n                    String.format(\n                        \" This is probably due to a schema \"\n                            + \"mismatch. Problematic row had extra schema fields: %s.\",\n                        rowDetails);\n              }\n              throw new RuntimeException(bqLimitLog);\n            } else {\n              numFailedRows += 1;\n              errorContainer.add(failedInserts, error, ref, rowsToPublish.get(rowIndex));\n              failedIndices.add(rowIndex);\n              rowIndex++;\n              continue;\n            }\n          }\n\n          // If adding the next row will push the request above BQ row limits, or\n          // if the current batch of elements is larger than the targeted request size,\n          // we immediately go and issue the data insertion.\n          if (dataSize + nextRowSize >= MAX_BQ_ROW_PAYLOAD_BYTES\n              || dataSize >= maxRowBatchSize\n              || rows.size() + 1 > maxRowsPerBatch) {\n            // If the row does not fit into the insert buffer, then we take the current buffer,\n            // issue the insert call, and we retry adding the same row to the troublesome buffer.\n            // Add a future to insert the current batch into BQ.","sourceCodeStart":1199,"sourceCodeEnd":1235,"githubUrl":"https://github.com/apache/beam/blob/12126d8942aaf848030c478b4c6a28c6af861c66/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryServicesImpl.java#L1199-L1235","documentation":"Thrown when a single row's encoded size exceeds BigQueryIO's maximum per-row payload limit. The message includes the row's byte size and the BigQueryIO limit; for schema-mismatch cases it appends the extra schema fields that inflated the row. Such a row can never be inserted and is an unconditional failure unless written to a dead-letter queue.","triggerScenarios":"A TableRow's JSON encoding exceeds the per-row limit (a few MB), typically from a huge string/bytes field, or from fields not present in the target schema (schema mismatch inflating the row).","commonSituations":"Writing very large documents/embeddings to BigQuery via streaming insert; dynamic destination schemas where row keys don't match the table schema so extra fields are serialized; unbounded user-generated content columns.","solutions":["Shrink the offending row: split large fields across rows or store them in Cloud Storage and reference the URI","Use BigQueryIO with a dead-letter queue (failedInserts / ErrorContainer) so oversized rows are routed instead of failing the pipeline","Align row fields with the table schema so no extra fields are serialized","Enable USE_AVRO_LOGICAL_TIME or batch load (BigQuery Load job) instead of streaming for very large records"],"exampleFix":"// before\nrow.set(\"blob\", giantString); // 20 MB\n// after\nrow.set(\"blobUri\", writeToFileAndUpload(giantString));\nrow.set(\"blobSize\", (long) giantString.length());","handlingStrategy":"validation","validationCode":"long maxRowBytes = 5_000_000L; // conservative limit below BigQueryIO cap\nlong size = TableRowJsonCoder.of().getEncodedElementByteSize(row);\nif (size > maxRowBytes) {\n  // route to dead-letter queue or split the row\n}","typeGuard":null,"tryCatchPattern":"// use BigQueryIO DLQ instead of try-catch:\n// .withFailedInsertRetryPolicy(...) and consume failedInserts tag\nTupleTag<FailedInsert> failed = new TupleTag<>();\nWriteResult result = rows.apply(BigQueryIO.writeTableRows()...withFailedInsertRetryPolicy(policy));\nresult.getFailedInserts().apply(...); // inspect oversized rows","preventionTips":["Measure row sizes in tests before deploying","Store large blobs in GCS and reference them","Keep row fields aligned with the table schema to avoid extra-field inflation","Configure failedInserts DLQ for every streaming write"],"tags":["bigquery","gcp","row-size-limit","schema"],"backgroundTag":"payload-too-large","analyzedSha":"12126d8942aaf848030c478b4c6a28c6af861c66","analyzedAt":"2026-09-13T01:50:10.254Z","contentChangedAt":"2026-09-13T01:50:10.254Z","schemaVersion":2},"datasetVersion":"2026-09-14T21:17:11.552Z"}