apache/beam · error · ValueError

Serialized pubsub message exceeds the publish request limit

Error message

Serialized pubsub message exceeds the publish request limit of 10MB

What it means

After building the PubsubMessage proto, _to_proto_str serializes it and enforces the overall publish request limit of 10MB on the serialized message. Even if individual fields are within their own limits, the combined serialized protobuf must not exceed 10,000,000 bytes.

Source

Thrown at sdks/python/apache_beam/io/gcp/pubsub.py:188

    if not for_publish:
      if self.message_id:
        message_id = self.message_id
        if self.publish_time:
          publish_time = self.publish_time

    if len(self.ordering_key) > 1024:
      raise ValueError(
          'A pubsub message ordering key must not exceed 1024 bytes.')

    msg = pubsub.types.PubsubMessage(
        data=self.data,
        attributes=self.attributes,
        message_id=message_id,
        publish_time=publish_time,
        ordering_key=self.ordering_key)
    serialized = pubsub.types.PubsubMessage.serialize(msg)
    if len(serialized) > (10_000_000):
      raise ValueError(
          'Serialized pubsub message exceeds the publish request limit of 10MB')
    return serialized

  @staticmethod
  def _from_message(msg: Any) -> 'PubsubMessage':
    """Construct from ``google.cloud.pubsub_v1.subscriber.message.Message``.

    https://googleapis.github.io/google-cloud-python/latest/pubsub/subscriber/api/message.html
    """
    # Convert ScalarMapContainer to dict.
    attributes = dict(msg.attributes)
    pubsubmessage = PubsubMessage(msg.data, attributes)
    if msg.message_id:
      pubsubmessage.message_id = msg.message_id
    if msg.publish_time:
      pubsubmessage.publish_time = msg.publish_time
    if msg.ordering_key:
      pubsubmessage.ordering_key = msg.ordering_key

View on GitHub (pinned to 12126d8942)

Solutions

Reduce total message size: shrink data and/or trim attributes so the serialized message is under 10MB.
Split into multiple messages below the limit.
Offload the bulk payload to GCS/BigQuery and publish a reference with small attributes.
Add a pre-serialization size check in the emitting DoFn to split or spill oversized messages.

Example fix

// before
yield PubsubMessage(data=payload, attributes=all_metadata)  # serialized > 10MB
// after
ref = write_to_gcs(payload)
yield PubsubMessage(data=ref.encode(), attributes={'size': str(len(payload))})

Defensive patterns

Strategy: validation

Validate before calling

import json
attrs_json = json.dumps(attributes or {}, default=str)
if len(data) + len(attrs_json) > 9_500_000:
    raise ValueError('message may exceed serialized 10MB publish limit')

Type guard

def fits_publish_limit(data, attributes):
    return len(data) + len(json.dumps(attributes or {}, default=str)) <= 10_000_000

Try / catch

try:
    proto = message_to_proto_str(msg)
except ValueError as e:
    if 'exceeds the publish request limit' in str(e):
        split_or_spill_message(msg)
    else:
        raise

Prevention

Estimate total serialized size (data + attributes) before constructing messages
Split large batches into multiple messages during aggregation
Offload bulk payloads to GCS and publish references

When it happens

Trigger: message_to_proto_str / _to_proto_str on a message whose serialized protobuf (data + attributes + metadata) exceeds 10,000,000 bytes.

Common situations: Messages with near-limit data plus many attributes pushing total size over 10MB; many moderately-sized attributes accumulating with a large payload.

Understand the failure class

Background: payload too large / request exceeds maximum size: why libraries cap bytes and how to fix oversize payloads — this error's family across 50 libraries.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/b84648567d18ed9d. Report an issue: GitHub.