apache/beam · error · ValueError

A pubsub message attribute value must not exceed 1024 bytes

Error message

A pubsub message attribute value must not exceed 1024 bytes

What it means

PubsubMessage._to_proto_str enforces the Cloud Pub/Sub limit that attribute values be at most 1024 bytes; one of the values in self.attributes exceeds it, so the message cannot be converted to a publishable protobuf.

Source

Thrown at sdks/python/apache_beam/io/gcp/pubsub.py:165

    Returns:
      A str containing a serialized protobuf of type
      https://cloud.google.com/pubsub/docs/reference/rpc/google.pubsub.v1#google.pubsub.v1.PubsubMessage
      containing the payload of this object.
    """
    if len(self.data) > (10_000_000):
      raise ValueError('A pubsub message data field must not exceed 10MB')

    if self.attributes:
      if len(self.attributes) > 100:
        raise ValueError(
            'A pubsub message must not have more than 100 attributes.')
      for key, value in self.attributes.items():
        if len(key) > 256:
          raise ValueError(
              'A pubsub message attribute key must not exceed 256 bytes.')
        if len(value) > 1024:
          raise ValueError(
              'A pubsub message attribute value must not exceed 1024 bytes')

    message_id = None
    publish_time = None
    if not for_publish:
      if self.message_id:
        message_id = self.message_id
        if self.publish_time:
          publish_time = self.publish_time

    if len(self.ordering_key) > 1024:
      raise ValueError(
          'A pubsub message ordering key must not exceed 1024 bytes.')

    msg = pubsub.types.PubsubMessage(
        data=self.data,
        attributes=self.attributes,
        message_id=message_id,

View on GitHub (pinned to 12126d8942)

Solutions

1. Move large values into the message data payload and keep only small metadata in attributes.
2. Truncate or hash the value before setting it as an attribute.
3. Compress/encode the value if it compresses under 1024 bytes, though storing in data is cleaner.

Example fix

// before
attrs['trace'] = traceback_text  # may exceed 1024 bytes
// after
attrs['trace_ref'] = write_to_gcs(traceback_text)

bad = [(k, v) for k, v in (attributes or {}).items() if len(v) > 1024]
if bad:
    raise ValueError(f'attribute values exceed 1024 bytes: {[k for k, _ in bad]}')

def has_valid_attribute_values(attrs):
    return not attrs or all(len(v) <= 1024 for v in attrs.values())

try:
    proto = PubsubMessage(data=data, attributes=attrs)._to_proto_str()
except ValueError as e:
    if 'attribute value must not exceed 1024' in str(e):
        log.error('Move oversized attribute value into payload or external store')
    raise

When it happens

Trigger: Calling message_to_proto_str / _to_proto_str where any value in self.attributes has len(value) > 1024.

Common situations: Putting large payloads (JSON blobs, stack traces, serialized objects) into attributes; forwarding values from other systems without size checks.

Understand the failure class

Background: payload too large / request exceeds maximum size: why libraries cap bytes and how to fix oversize payloads — this error's family across 50 libraries.

Related errors

• A pubsub message must not have more than 100 attributes.
• A pubsub message attribute key must not exceed 256 bytes.
• A pubsub message data field must not exceed 10MB
• A pubsub message ordering key must not exceed 1024 bytes.
• Attribute fields {missing_attribute_names} not found in sche

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/10ee0d149bfad2e4. Report an issue: GitHub.