apache/beam · error · IllegalStateException

Could not encode message as bytes

Error message

Could not encode message as bytes

What it means

RowMessages.rowToBytesFn converts a Row to a protobuf message via fromRowFn and then serializes it. If the toBytes step throws (serialization failure), it is rethrown as IllegalStateException 'Could not encode message as bytes'.

Solutions

1. Confirm the Row schema matches the current proto descriptor (field names, types, nullability).
2. Regenerate/rebuild rows from the updated proto-generated class so fromRowFn/toBytesFn agree.
3. Catch and log the IllegalStateException with the underlying cause to identify the failing field during serialization.

Example fix

// before
bytes = row.apply(RowMessages.rowToBytesFn(Msg.getDescriptor(), Msg.class));

// after
if (!row.getSchema().equivTo(SchemaOptions... )) { /* validate row schema matches proto */ }
bytes = row.apply("RowsToProto", RowMessages.rowToBytesFn(Msg.getDescriptor(), Msg.class));

// Java: confirm row schema matches proto before serializing
if (!row.getSchema().getFieldNames().equals(expectedProtoFieldNames)) {
  throw new IllegalArgumentException("Row schema mismatch with proto descriptor");
}

// Java
try {
  byte[] out = rowToBytesFn.apply(row);
} catch (IllegalStateException e) {
  // log e.getCause(); send row to a dead-letter output
}

When it happens

Trigger: Applying rowToBytesFn where the Row does not match the proto schema (wrong types/field names) so message construction succeeds partially but toBytesFn fails, or the supplied toBytes function itself fails.

Common situations: Row schema drift after proto regeneration (field renamed/retyped), null values in non-nullable proto fields, writing rows built against an outdated Schema to a proto-based sink.

Understand the failure class

Background: json.Marshal / "failed to marshal" errors in Go: why "unsupported type" happens and how to fix it — this error's family across 22 libraries.

Related errors

• Cannot create creator for
• Could not decode bytes as message
• DynamicMessage is not allowed for the standard…
• Encountered UNSPECIFIED AtomicType
• Error while parsing input element

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/70484f6504e67378. Report an issue: GitHub.

Appendix: source

Thrown at sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/RowMessages.java:122

    private RowToBytesFn(
        Schema schema,
        SerializableFunction<Row, T> fromRowFn,
        ProcessFunction<? super T, byte[]> toBytesFn) {
      this.schema = schema;
      this.fromRowFn = fromRowFn;
      this.toBytesFn = toBytesFn;
    }

    @Override
    public byte[] apply(Row row) {
      if (!schema.equivalent(row.getSchema())) {
        row = switchFieldsOrder(row);
      }
      final T message = fromRowFn.apply(row);
      try {
        return toBytesFn.apply(message);
      } catch (Exception e) {
        throw new IllegalStateException("Could not encode message as bytes", e);
      }
    }

    private Row switchFieldsOrder(Row row) {
      Row.Builder convertedRow = Row.withSchema(schema);
      schema.getFields().forEach(field -> convertedRow.addValue(row.getValue(field.getName())));
      return convertedRow.build();
    }
  }
}

View on GitHub (pinned to 12126d8942)