apache/beam · error · ValueError

A pubsub message data field must not exceed 10MB

Error message

A pubsub message data field must not exceed 10MB

What it means

PubsubMessage._to_proto_str enforces the Pub/Sub publisher limit that a message's data field may not exceed 10,000,000 bytes. Beam checks this client-side before serializing so the publish fails early with a clear error instead of a server-side rejection.

Source

Thrown at sdks/python/apache_beam/io/gcp/pubsub.py:154

  def _to_proto_str(self, for_publish=False):
    """Get serialized form of ``PubsubMessage``.

    The serialized message is validated against pubsub message limits specified
    at https://cloud.google.com/pubsub/quotas#resource_limits

    Args:
      proto_msg: str containing a serialized protobuf.
      for_publish: bool, if True strip out message fields which cannot be
        published (currently message_id and publish_time) per
        https://cloud.google.com/pubsub/docs/reference/rpc/google.pubsub.v1#pubsubmessage

    Returns:
      A str containing a serialized protobuf of type
      https://cloud.google.com/pubsub/docs/reference/rpc/google.pubsub.v1#google.pubsub.v1.PubsubMessage
      containing the payload of this object.
    """
    if len(self.data) > (10_000_000):
      raise ValueError('A pubsub message data field must not exceed 10MB')

    if self.attributes:
      if len(self.attributes) > 100:
        raise ValueError(
            'A pubsub message must not have more than 100 attributes.')
      for key, value in self.attributes.items():
        if len(key) > 256:
          raise ValueError(
              'A pubsub message attribute key must not exceed 256 bytes.')
        if len(value) > 1024:
          raise ValueError(
              'A pubsub message attribute value must not exceed 1024 bytes')

    message_id = None
    publish_time = None
    if not for_publish:
      if self.message_id:
        message_id = self.message_id

View on GitHub (pinned to 12126d8942)

Solutions

1. Split the payload into multiple messages smaller than 10MB and emit each.
2. Compress the data (e.g. gzip) before publishing if it compresses under the limit.
3. Store large payloads in GCS/BigQuery and publish a reference/URI in the message.
4. Adjust the upstream DoFn/aggregation to cap message sizes.

Example fix

// before
yield PubsubMessage(data=blob, attributes={})  # blob > 10MB
// after
if len(blob) > 10_000_000:
  uri = write_to_gcs(blob)
  yield PubsubMessage(data=uri.encode(), attributes={'ref': 'gcs'})
else:
  yield PubsubMessage(data=blob, attributes={})

if len(data) > 10_000_000:
    raise ValueError(f'data too large for PubsubMessage: {len(data)} bytes')

def fits_pubsub_data(data):
    return len(data) <= 10_000_000

try:
    proto = PubsubMessage(data=data, attributes={})._to_proto_str()
except ValueError as e:
    if 'must not exceed 10MB' in str(e):
        spill_to_gcs_and_emit_reference(data)
    else:
        raise

When it happens

Trigger: Calling message_to_proto_str / _to_proto_str on a PubsubMessage whose data blob is larger than 10MB (len(self.data) > 10_000_000).

Common situations: Reading very large blobs or batched payloads from a source and emitting them as a single Pub/Sub message; aggregation windows that concatenate records into one oversized message.

Understand the failure class

Background: payload too large / request exceeds maximum size: why libraries cap bytes and how to fix oversize payloads — this error's family across 50 libraries.

Related errors

• A pubsub message must not have more than 100 attributes.
• A pubsub message attribute key must not exceed 256 bytes.
• A pubsub message attribute value must not exceed 1024 bytes
• A pubsub message ordering key must not exceed 1024 bytes.
• Serialized pubsub message exceeds the publish request limit

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/dac4541922dfe6b6. Report an issue: GitHub.