apache/beam · error · IllegalStateException

Could not encode message as bytes

Error message

Could not encode message as bytes

What it means

RowMessages.rowToBytesFn converts a Row to a protobuf message via fromRowFn and then serializes it. If the toBytes step throws (serialization failure), it is rethrown as IllegalStateException 'Could not encode message as bytes'.

Source

Thrown at sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/RowMessages.java:122

    private RowToBytesFn(
        Schema schema,
        SerializableFunction<Row, T> fromRowFn,
        ProcessFunction<? super T, byte[]> toBytesFn) {
      this.schema = schema;
      this.fromRowFn = fromRowFn;
      this.toBytesFn = toBytesFn;
    }

    @Override
    public byte[] apply(Row row) {
      if (!schema.equivalent(row.getSchema())) {
        row = switchFieldsOrder(row);
      }
      final T message = fromRowFn.apply(row);
      try {
        return toBytesFn.apply(message);
      } catch (Exception e) {
        throw new IllegalStateException("Could not encode message as bytes", e);
      }
    }

    private Row switchFieldsOrder(Row row) {
      Row.Builder convertedRow = Row.withSchema(schema);
      schema.getFields().forEach(field -> convertedRow.addValue(row.getValue(field.getName())));
      return convertedRow.build();
    }
  }
}

View on GitHub (pinned to 12126d8942)

Solutions

1. Confirm the Row schema matches the current proto descriptor (field names, types, nullability).
2. Regenerate/rebuild rows from the updated proto-generated class so fromRowFn/toBytesFn agree.
3. Catch and log the IllegalStateException with the underlying cause to identify the failing field during serialization.

Example fix

// before
bytes = row.apply(RowMessages.rowToBytesFn(Msg.getDescriptor(), Msg.class));

// after
if (!row.getSchema().equivTo(SchemaOptions... )) { /* validate row schema matches proto */ }
bytes = row.apply("RowsToProto", RowMessages.rowToBytesFn(Msg.getDescriptor(), Msg.class));

// Java: confirm row schema matches proto before serializing
if (!row.getSchema().getFieldNames().equals(expectedProtoFieldNames)) {
  throw new IllegalArgumentException("Row schema mismatch with proto descriptor");
}

// Java
try {
  byte[] out = rowToBytesFn.apply(row);
} catch (IllegalStateException e) {
  // log e.getCause(); send row to a dead-letter output
}

When it happens

Trigger: Applying rowToBytesFn where the Row does not match the proto schema (wrong types/field names) so message construction succeeds partially but toBytesFn fails, or the supplied toBytes function itself fails.

Common situations: Row schema drift after proto regeneration (field renamed/retyped), null values in non-nullable proto fields, writing rows built against an outdated Schema to a proto-based sink.

Understand the failure class

Background: json.Marshal / "failed to marshal" errors in Go: why "unsupported type" happens and how to fix it — this error's family across 22 libraries.

Related errors

• Could not decode bytes as message
• Failed to decode Schema due to an error decoding Field proto
• Encountered UNSPECIFIED AtomicType
• Null value found for field that doesn't support nulls.
• FieldType unexpected +fieldType.getTypeName()

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/70484f6504e67378. Report an issue: GitHub.