apache/beam · error · ValueError

A pubsub message attribute key must not exceed 256 bytes.

Error message

A pubsub message attribute key must not exceed 256 bytes.

What it means

PubsubMessage._to_proto_str enforces the Cloud Pub/Sub limit that attribute keys be at most 256 bytes; one of the keys in self.attributes exceeds it, so serialization is aborted before the API rejects the message.

Source

Thrown at sdks/python/apache_beam/io/gcp/pubsub.py:162

      for_publish: bool, if True strip out message fields which cannot be
        published (currently message_id and publish_time) per
        https://cloud.google.com/pubsub/docs/reference/rpc/google.pubsub.v1#pubsubmessage

    Returns:
      A str containing a serialized protobuf of type
      https://cloud.google.com/pubsub/docs/reference/rpc/google.pubsub.v1#google.pubsub.v1.PubsubMessage
      containing the payload of this object.
    """
    if len(self.data) > (10_000_000):
      raise ValueError('A pubsub message data field must not exceed 10MB')

    if self.attributes:
      if len(self.attributes) > 100:
        raise ValueError(
            'A pubsub message must not have more than 100 attributes.')
      for key, value in self.attributes.items():
        if len(key) > 256:
          raise ValueError(
              'A pubsub message attribute key must not exceed 256 bytes.')
        if len(value) > 1024:
          raise ValueError(
              'A pubsub message attribute value must not exceed 1024 bytes')

    message_id = None
    publish_time = None
    if not for_publish:
      if self.message_id:
        message_id = self.message_id
        if self.publish_time:
          publish_time = self.publish_time

    if len(self.ordering_key) > 1024:
      raise ValueError(
          'A pubsub message ordering key must not exceed 1024 bytes.')

    msg = pubsub.types.PubsubMessage(

View on GitHub (pinned to 12126d8942)

Solutions

Shorten the attribute key (e.g. use a short token and put the full value in the message body).
Hash or truncate long keys before constructing PubsubMessage.
Move long key-value metadata into the data payload (JSON) instead of attributes.

Example fix

// before
attrs[full_url] = 'true'  # key > 256 bytes
// after
attrs['source_url_sha'] = hashlib.sha256(full_url.encode()).hexdigest()[:32]

Defensive patterns

Strategy: validation

Validate before calling

bad = [k for k in (attributes or {}) if len(k) > 256]
if bad:
    raise ValueError(f'attribute keys exceed 256 bytes: {bad[:3]}')

Type guard

def has_valid_attribute_keys(attrs):
    return not attrs or all(len(k) <= 256 for k in attrs)

Try / catch

try:
    proto = PubsubMessage(data=data, attributes=attrs)._to_proto_str()
except ValueError as e:
    if 'attribute key must not exceed 256' in str(e):
        log.error('Shorten or hash oversized attribute keys')
    raise

Prevention

Use short token keys, not URLs or long identifiers
Sanitize user-supplied header names before mapping to attributes
Add a key-length check in your message builder helper

When it happens

Trigger: Calling message_to_proto_str / _to_proto_str where any key in self.attributes has len(key) > 256.

Common situations: Using long identifiers, full URLs, or concatenated names as attribute keys; attributes derived from untrusted/user-supplied header names.

Understand the failure class

Background: payload too large / request exceeds maximum size: why libraries cap bytes and how to fix oversize payloads — this error's family across 50 libraries.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/7a3c4877f9a1ca65. Report an issue: GitHub.