apache/beam · error · RuntimeException

Unexpected error while serializing PubsubMessage to a byte a

Error message

Unexpected error while serializing PubsubMessage to a byte array.

What it means

PubsubMessages.toSerializedPubsubMessageProto serializes a payload directly into a pre-sized PubsubMessage protobuf byte array. Writing to an in-memory array should never throw IOException; if it does, an internal invariant is broken, so it rethrows as RuntimeException with this message and the IOException as cause.

Source

Thrown at sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsub/PubsubMessages.java:77

    if ((attributes == null || attributes.isEmpty())
        && (messageId == null || messageId.isEmpty())
        && (orderingKey == null || orderingKey.isEmpty())) {
      // Optimize the case where we are just sending a payload.
      byte[] payload = input.getPayload();
      if (payload == null || payload.length == 0) {
        return new byte[0];
      }
      int size =
          CodedOutputStream.computeByteArraySize(
              com.google.pubsub.v1.PubsubMessage.DATA_FIELD_NUMBER, payload);
      byte[] serialized = new byte[size];
      try {
        CodedOutputStream output = CodedOutputStream.newInstance(serialized);
        output.writeByteArray(com.google.pubsub.v1.PubsubMessage.DATA_FIELD_NUMBER, payload);
        output.checkNoSpaceLeft();
      } catch (IOException e) {
        // Should not happen since we are writing to a byte array of the exact size.
        throw new RuntimeException(
            "Unexpected error while serializing PubsubMessage to a byte array.", e);
      }
      return serialized;
    }
    // Fallback to general case by building up a protobuf and serializing it.
    return toProto(input).toByteArray();
  }

  public static PubsubMessage fromProto(com.google.pubsub.v1.PubsubMessage input) {
    return new PubsubMessage(
        input.getData().toByteArray(),
        input.getAttributesMap(),
        input.getMessageId(),
        input.getOrderingKey());
  }

  // Convert the beam PubsubMessage to a serialized com.google.pubsub.v1.PubsubMessage proto
  // representation.

View on GitHub (pinned to 12126d8942)

Solutions

1. Report the issue to the Beam project with the payload size if reproducible; this indicates an internal size-computation bug.
2. Use the public API (PubsubMessages.toProto(...).toByteArray() or PubsubIO APIs) instead of the internal fast-path helper.
3. Upgrade Beam to a version where the fast-path sizing bug is fixed.

try {
  byte[] bytes = PubsubMessages.toSerializedPubsubMessageProto(msg);
} catch (RuntimeException e) {
  // internal invariant broken: fall back to toProto(msg).toByteArray() and report
}

When it happens

Trigger: Internal-only: CodedOutputStream.writeByteArray on the exact-size buffer reports no space left or another IOException. Triggered in practice by corrupted size computation or misuse of the internal API, not by user input.

Common situations: Rarely seen by end users; may appear during Beam upgrades or when calling internal PubsubMessage-to-proto helpers with unusual payload sizes.

Understand the failure class

Background: "JSON serialization failed", "not JSON serializable", "Failed to serialize": why JSON marshaling errors happen and how to fix them — this error's family across 46 libraries.

Related errors

• Could not encode message as bytes
• Failed to convert PipelineOptions to JSON
• Internal error determining boundedness of Read
• Failed to parse DataStore key from bytes.
• Could not parse Pub/Sub message

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/7a270c6531b73425. Report an issue: GitHub.