apache/beam · error · RuntimeException

Failed to convert the row to JSON

Error message

Failed to convert the row to JSON

What it means

TableContainer tracks fake table size by encoding each TableRow with TableRowJsonCoder; if computing the encoded size fails for any reason the row cannot be represented as JSON and this RuntimeException wraps the cause. This mirrors size accounting in the fake BigQuery service.

Source

Thrown at sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/testing/TableContainer.java:131

    if (primaryKey != null && !primaryKey.isEmpty()) {
      if (keyedRows.putIfAbsent(primaryKey, row) != null) {
        throw new RuntimeException(
            "Primary key validation error! Multiple inserts with the same primary key.");
      }
    } else {
      rows.add(row);
      if (id != null) {
        ids.add(id);
      }
    }

    long tableSize = table.getNumBytes() == null ? 0L : table.getNumBytes();
    try {
      long rowSize = TableRowJsonCoder.of().getEncodedElementByteSize(row);
      table.setNumBytes(tableSize + rowSize);
      return rowSize;
    } catch (Exception ex) {
      throw new RuntimeException("Failed to convert the row to JSON", ex);
    }
  }

  void upsertRow(TableRow row, long sequenceNumber) {
    List<Object> primaryKey = getPrimaryKey(row);
    if (primaryKey == null) {
      throw new RuntimeException("Upserts only allowed when using primary keys");
    }
    long lastSequenceNumberForKey = lastSequenceNumber.getOrDefault(primaryKey, Long.MIN_VALUE);
    if (sequenceNumber <= lastSequenceNumberForKey) {
      // Out-of-order upsert - ignore it as we've already seen a more-recent update.
      return;
    }

    TableRow oldValue = keyedRows.put(primaryKey, row);
    try {
      long tableSize = table.getNumBytes() == null ? 0L : table.getNumBytes();
      if (oldValue != null) {

View on GitHub (pinned to 12126d8942)

Solutions

1. Inspect the cause exception to find the non-encodable field
2. Convert row values to JSON-safe types (String, Long, Double, Boolean, nested Map/List) before insertAll
3. Compare against real BigQuery behavior — validate the row with TableRowJsonCoder.of().encodeToString(row) in a unit test

Example fix

// before
row.put("payload", myByteArray);
// after
row.put("payload", Base64.getEncoder().encodeToString(myByteArray));

TableRowJsonCoder.of().getEncodedElementByteSize(row); // throws early if not encodable

try {
  container.insertAll(rows);
} catch (RuntimeException e) {
  if (e.getMessage().contains("Failed to convert the row to JSON")) {
    // inspect row values, fix types, retry
  } else throw e;
}

When it happens

Trigger: addRow invoked (via insertAll) with a TableRow that TableRowJsonCoder cannot encode — e.g. non-JSON-serializable values or a malformed/nested value type.

Common situations: Tests writing rows containing unsupported Java types (e.g. byte arrays, custom POJOs, BigDecimal variants) into the fake BigQuery table.

Understand the failure class

Background: json.Marshal / "failed to marshal" errors in Go: why "unsupported type" happens and how to fix it — this error's family across 22 libraries.

Related errors

• Failed to serialize %s value: %s
• Unsupported logical type " + logicalType.getIdentifier()
• Unexpected beam type " + beamFieldType
• Error while converting table schema %s to JSON!
• Unable to infer coder for output of parseFn. Specify it expl

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/48aae930811410b1. Report an issue: GitHub.