apache/beam · error · ValueError

A pubsub message must not have more than 100 attributes.

Error message

A pubsub message must not have more than 100 attributes.

What it means

PubsubMessage._to_proto_str enforces the Cloud Pub/Sub resource limit of at most 100 attributes per message; the message being serialized carries more, and the API would reject it at publish time, so the error is raised client-side during protobuf conversion.

Source

Thrown at sdks/python/apache_beam/io/gcp/pubsub.py:158

    at https://cloud.google.com/pubsub/quotas#resource_limits

    Args:
      proto_msg: str containing a serialized protobuf.
      for_publish: bool, if True strip out message fields which cannot be
        published (currently message_id and publish_time) per
        https://cloud.google.com/pubsub/docs/reference/rpc/google.pubsub.v1#pubsubmessage

    Returns:
      A str containing a serialized protobuf of type
      https://cloud.google.com/pubsub/docs/reference/rpc/google.pubsub.v1#google.pubsub.v1.PubsubMessage
      containing the payload of this object.
    """
    if len(self.data) > (10_000_000):
      raise ValueError('A pubsub message data field must not exceed 10MB')

    if self.attributes:
      if len(self.attributes) > 100:
        raise ValueError(
            'A pubsub message must not have more than 100 attributes.')
      for key, value in self.attributes.items():
        if len(key) > 256:
          raise ValueError(
              'A pubsub message attribute key must not exceed 256 bytes.')
        if len(value) > 1024:
          raise ValueError(
              'A pubsub message attribute value must not exceed 1024 bytes')

    message_id = None
    publish_time = None
    if not for_publish:
      if self.message_id:
        message_id = self.message_id
        if self.publish_time:
          publish_time = self.publish_time

    if len(self.ordering_key) > 1024:

View on GitHub (pinned to 12126d8942)

Solutions

1. Reduce attributes to <= 100 by dropping optional/derived keys.
2. Batch overflow metadata into the message data payload (e.g. as JSON) instead of attributes.
3. Compact related attributes into a single key with structured values.

Example fix

// before
attrs = {f'field_{i}': str(v) for i, v in enumerate(record)}  # may exceed 100
// after
essential = {k: record[k] for k in REQUIRED_FIELDS}
yield PubsubMessage(data=json.dumps(record).encode(), attributes=essential)

if attributes is not None and len(attributes) > 100:
    raise ValueError(f'too many attributes: {len(attributes)}')

def fits_pubsub_attribute_count(attrs):
    return not attrs or len(attrs) <= 100

try:
    proto = PubsubMessage(data=data, attributes=attrs)._to_proto_str()
except ValueError as e:
    if 'more than 100 attributes' in str(e):
        move_overflow_to_payload(data, attrs)
    else:
        raise

When it happens

Trigger: Calling message_to_proto_str / _to_proto_str on a PubsubMessage whose attributes dict contains more than 100 keys.

Common situations: Programmatically generating attributes from a wide record (one attribute per field); merging attribute maps from multiple sources; misusing attributes to carry bulk metadata.

Understand the failure class

Background: payload too large / request exceeds maximum size: why libraries cap bytes and how to fix oversize payloads — this error's family across 50 libraries.

Related errors

• A pubsub message attribute key must not exceed 256 bytes.
• A pubsub message attribute value must not exceed 1024 bytes
• A pubsub message data field must not exceed 10MB
• A pubsub message ordering key must not exceed 1024 bytes.
• Attribute fields {missing_attribute_names} not found in sche

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/17302af213833bde. Report an issue: GitHub.