apache/beam · error · RuntimeException

Failed to convert the row to JSON

Error message

Failed to convert the row to JSON

What it means

DatasetServiceImpl measures each TableRow's encoded JSON size using TableRowJsonCoder before batching it for insertAll. If encoding the row throws, the code wraps the exception in a RuntimeException with this message. It means the row cannot be serialized to the JSON representation BigQuery streaming insert requires.

Solutions

1. Inspect getCause() to see which field/value fails to encode
2. Ensure all row values are JSON-compatible (String, Long, Double, Boolean, List, Map, or nested TableRow)
3. Convert custom types explicitly (e.g. use String or Double representations) before building the TableRow
4. Use TableRowJsonCoder.of().getEncodedElementByteSize(row) in a unit test to reproduce the failing row

Example fix

// before
row.set("payload", myCustomObject);
// after
row.set("payload", objectMapper.writeValueAsString(myCustomObject));

// validate rows before writing
for (TableRow row : rows) {
  try {
    TableRowJsonCoder.of().getEncodedElementByteSize(row);
  } catch (Exception e) {
    throw new IllegalArgumentException("Row not JSON-encodable: " + e.getMessage());
  }
}

boolean isEncodable(TableRow row) {
  return row.entrySet().stream()
      .allMatch(e -> e.getValue() instanceof String
          || e.getValue() instanceof Number
          || e.getValue() instanceof Boolean
          || e.getValue() instanceof TableRow);
}

try {
  rows.forEach(this::write);
} catch (RuntimeException e) {
  if ("Failed to convert the row to JSON".equals(e.getMessage())) {
    // route row to DLQ, log e.getCause()
  }
}

When it happens

Trigger: A TableRow contains values that TableRowJsonCoder cannot encode (e.g. unsupported nested value types, non-serializable objects placed in the row map, or a corrupted row produced upstream).

Common situations: User DoFns putting arbitrary Java objects (BigDecimal variants, byte arrays, custom POJOs) into TableRows instead of supported JSON-compatible values; schema/row construction bugs in dynamic destinations.

Understand the failure class

Background: json.Marshal / "failed to marshal" errors in Go: why "unsupported type" happens and how to fix it — this error's family across 22 libraries.

Related errors

• Failed to convert the row to JSON
• Found JSON type in TableSchema for 'FILE_LOADS' write…
• Object of type ' ' is not JSON serializable
• . . Row: %r
• Append to stream failed with invalid offset of

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/7285d4904af65da8. Report an issue: GitHub.

Appendix: source

Thrown at sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryServicesImpl.java:1164

        int strideIndex = 0;
        // Upload in batches.
        List<TableDataInsertAllRequest.Rows> rows = new ArrayList<>();
        long dataSize = 0L;

        List<Future<List<TableDataInsertAllResponse.InsertErrors>>> futures = new ArrayList<>();
        List<Integer> strideIndices = new ArrayList<>();
        // Store the longest throttled time across all parallel threads
        final AtomicLong maxThrottlingMsec = new AtomicLong();

        int rowIndex = 0;
        while (rowIndex < rowsToPublish.size()) {
          TableRow row = rowsToPublish.get(rowIndex).getValue();
          long nextRowSize = 0L;
          try {
            nextRowSize = TableRowJsonCoder.of().getEncodedElementByteSize(row);
          } catch (Exception ex) {
            throw new RuntimeException("Failed to convert the row to JSON", ex);
          }

          // The following scenario must be *extremely* rare.
          // If this row's encoding by itself is larger than the maximum row payload, then it's
          // impossible to insert into BigQuery, and so we send it out through the dead-letter
          // queue.
          if (nextRowSize >= MAX_BQ_ROW_PAYLOAD_BYTES) {
            InsertErrors error =
                new InsertErrors()
                    .setErrors(ImmutableList.of(new ErrorProto().setReason("row-too-large")));
            // We verify whether the retryPolicy parameter expects us to retry. If it does, then
            // it will return true. Otherwise it will return false.
            if (retryPolicy.shouldRetry(new InsertRetryPolicy.Context(error))) {
              // Obtain table schema
              TableSchema tableSchema = null;
              try {
                String tableSpec = BigQueryHelpers.toTableSpec(ref);
                if (tableSchemaCache.containsKey(tableSpec)) {

View on GitHub (pinned to 12126d8942)