apache/beam · error · RuntimeException

Unexpected error while serializing PubsubMessage to a byte…

Error message

Unexpected error while serializing PubsubMessage to a byte array.

What it means

PubsubMessages.toSerializedPubsubMessageProto serializes a payload directly into a pre-sized PubsubMessage protobuf byte array. Writing to an in-memory array should never throw IOException; if it does, an internal invariant is broken, so it rethrows as RuntimeException with this message and the IOException as cause.

Solutions

1. Report the issue to the Beam project with the payload size if reproducible; this indicates an internal size-computation bug.
2. Use the public API (PubsubMessages.toProto(...).toByteArray() or PubsubIO APIs) instead of the internal fast-path helper.
3. Upgrade Beam to a version where the fast-path sizing bug is fixed.

try {
  byte[] bytes = PubsubMessages.toSerializedPubsubMessageProto(msg);
} catch (RuntimeException e) {
  // internal invariant broken: fall back to toProto(msg).toByteArray() and report
}

When it happens

Trigger: Internal-only: CodedOutputStream.writeByteArray on the exact-size buffer reports no space left or another IOException. Triggered in practice by corrupted size computation or misuse of the internal API, not by user input.

Common situations: Rarely seen by end users; may appear during Beam upgrades or when calling internal PubsubMessage-to-proto helpers with unusual payload sizes.

Understand the failure class

Background: "JSON serialization failed", "not JSON serializable", "Failed to serialize": why JSON marshaling errors happen and how to fix them — this error's family across 46 libraries.

Related errors

• Could not decode Pubsub message
• Could not encode message as bytes
• Could not encode Pubsub message
• Could not parse Pub/Sub message
• Failed to convert PipelineOptions to JSON

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/7a270c6531b73425. Report an issue: GitHub.

Appendix: source

Thrown at sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsub/PubsubMessages.java:77

    if ((attributes == null || attributes.isEmpty())
        && (messageId == null || messageId.isEmpty())
        && (orderingKey == null || orderingKey.isEmpty())) {
      // Optimize the case where we are just sending a payload.
      byte[] payload = input.getPayload();
      if (payload == null || payload.length == 0) {
        return new byte[0];
      }
      int size =
          CodedOutputStream.computeByteArraySize(
              com.google.pubsub.v1.PubsubMessage.DATA_FIELD_NUMBER, payload);
      byte[] serialized = new byte[size];
      try {
        CodedOutputStream output = CodedOutputStream.newInstance(serialized);
        output.writeByteArray(com.google.pubsub.v1.PubsubMessage.DATA_FIELD_NUMBER, payload);
        output.checkNoSpaceLeft();
      } catch (IOException e) {
        // Should not happen since we are writing to a byte array of the exact size.
        throw new RuntimeException(
            "Unexpected error while serializing PubsubMessage to a byte array.", e);
      }
      return serialized;
    }
    // Fallback to general case by building up a protobuf and serializing it.
    return toProto(input).toByteArray();
  }

  public static PubsubMessage fromProto(com.google.pubsub.v1.PubsubMessage input) {
    return new PubsubMessage(
        input.getData().toByteArray(),
        input.getAttributesMap(),
        input.getMessageId(),
        input.getOrderingKey());
  }

  // Convert the beam PubsubMessage to a serialized com.google.pubsub.v1.PubsubMessage proto
  // representation.

View on GitHub (pinned to 12126d8942)