apache/beam · error · RuntimeException

We have observed a row of size %s bytes exceeding the BigQue

Error message

We have observed a row of size %s bytes exceeding the BigQueryIO limit of %s.

What it means

Thrown when a single row's encoded size exceeds BigQueryIO's maximum per-row payload limit. The message includes the row's byte size and the BigQueryIO limit; for schema-mismatch cases it appends the extra schema fields that inflated the row. Such a row can never be inserted and is an unconditional failure unless written to a dead-letter queue.

Source

Thrown at sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryServicesImpl.java:1217

                rowDetails = validateRowSchema(row, tableSchema);
              }

              // Basic log to return
              String bqLimitLog =
                  String.format(
                      "We have observed a row of size %s bytes exceeding the "
                          + "BigQueryIO limit of %s.",
                      nextRowSize, MAX_BQ_ROW_PAYLOAD_DESC);

              // Add on row schema diff details if present
              if (!rowDetails.isEmpty()) {
                bqLimitLog +=
                    String.format(
                        " This is probably due to a schema "
                            + "mismatch. Problematic row had extra schema fields: %s.",
                        rowDetails);
              }
              throw new RuntimeException(bqLimitLog);
            } else {
              numFailedRows += 1;
              errorContainer.add(failedInserts, error, ref, rowsToPublish.get(rowIndex));
              failedIndices.add(rowIndex);
              rowIndex++;
              continue;
            }
          }

          // If adding the next row will push the request above BQ row limits, or
          // if the current batch of elements is larger than the targeted request size,
          // we immediately go and issue the data insertion.
          if (dataSize + nextRowSize >= MAX_BQ_ROW_PAYLOAD_BYTES
              || dataSize >= maxRowBatchSize
              || rows.size() + 1 > maxRowsPerBatch) {
            // If the row does not fit into the insert buffer, then we take the current buffer,
            // issue the insert call, and we retry adding the same row to the troublesome buffer.
            // Add a future to insert the current batch into BQ.

View on GitHub (pinned to 12126d8942)

Solutions

1. Shrink the offending row: split large fields across rows or store them in Cloud Storage and reference the URI
2. Use BigQueryIO with a dead-letter queue (failedInserts / ErrorContainer) so oversized rows are routed instead of failing the pipeline
3. Align row fields with the table schema so no extra fields are serialized
4. Enable USE_AVRO_LOGICAL_TIME or batch load (BigQuery Load job) instead of streaming for very large records

Example fix

// before
row.set("blob", giantString); // 20 MB
// after
row.set("blobUri", writeToFileAndUpload(giantString));
row.set("blobSize", (long) giantString.length());

long maxRowBytes = 5_000_000L; // conservative limit below BigQueryIO cap
long size = TableRowJsonCoder.of().getEncodedElementByteSize(row);
if (size > maxRowBytes) {
  // route to dead-letter queue or split the row
}

// use BigQueryIO DLQ instead of try-catch:
// .withFailedInsertRetryPolicy(...) and consume failedInserts tag
TupleTag<FailedInsert> failed = new TupleTag<>();
WriteResult result = rows.apply(BigQueryIO.writeTableRows()...withFailedInsertRetryPolicy(policy));
result.getFailedInserts().apply(...); // inspect oversized rows

When it happens

Trigger: A TableRow's JSON encoding exceeds the per-row limit (a few MB), typically from a huge string/bytes field, or from fields not present in the target schema (schema mismatch inflating the row).

Common situations: Writing very large documents/embeddings to BigQuery via streaming insert; dynamic destination schemas where row keys don't match the table schema so extra fields are serialized; unbounded user-generated content columns.

Understand the failure class

Background: payload too large / request exceeds maximum size: why libraries cap bytes and how to fix oversize payloads — this error's family across 50 libraries.

Related errors

• Exception while trying to retrieve schema
• Failed to patch table schema.
• Schema must be set for table " + tableReference + " when wri
• Table %s:%s.%s requires a schema. None can be inferred becau
• Reserved field name <field.name()> in user schema.

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/52a0c62a5cd57a73. Report an issue: GitHub.