apache/beam · error · ValueError

A pubsub message ordering key must not exceed 1024 bytes.

Error message

A pubsub message ordering key must not exceed 1024 bytes.

What it means

PubsubMessage._to_proto_str enforces the Cloud Pub/Sub limit that the ordering key be at most 1024 bytes; the ordering_key set on this message is longer, so serialization fails client-side rather than at publish.

Source

Thrown at sdks/python/apache_beam/io/gcp/pubsub.py:177

            'A pubsub message must not have more than 100 attributes.')
      for key, value in self.attributes.items():
        if len(key) > 256:
          raise ValueError(
              'A pubsub message attribute key must not exceed 256 bytes.')
        if len(value) > 1024:
          raise ValueError(
              'A pubsub message attribute value must not exceed 1024 bytes')

    message_id = None
    publish_time = None
    if not for_publish:
      if self.message_id:
        message_id = self.message_id
        if self.publish_time:
          publish_time = self.publish_time

    if len(self.ordering_key) > 1024:
      raise ValueError(
          'A pubsub message ordering key must not exceed 1024 bytes.')

    msg = pubsub.types.PubsubMessage(
        data=self.data,
        attributes=self.attributes,
        message_id=message_id,
        publish_time=publish_time,
        ordering_key=self.ordering_key)
    serialized = pubsub.types.PubsubMessage.serialize(msg)
    if len(serialized) > (10_000_000):
      raise ValueError(
          'Serialized pubsub message exceeds the publish request limit of 10MB')
    return serialized

  @staticmethod
  def _from_message(msg: Any) -> 'PubsubMessage':
    """Construct from ``google.cloud.pubsub_v1.subscriber.message.Message``.

View on GitHub (pinned to 12126d8942)

Solutions

  1. Shorten the ordering key to <= 1024 bytes, e.g. by hashing the logical key.
  2. Design ordering keys around a bounded entity id (user/device id) rather than concatenated metadata.
  3. Drop the ordering key entirely if strict ordering is not required (default empty string).

Example fix

// before
PubsubMessage(data=b'x', attributes={}, ordering_key='|'.join(record.values()))
// after
key = hashlib.sha1('|'.join(record.values()).encode()).hexdigest()
yield PubsubMessage(data=b'x', attributes={}, ordering_key=key)
Defensive patterns

Strategy: validation

Validate before calling

if ordering_key and len(ordering_key) > 1024:
    raise ValueError(f'ordering_key too long: {len(ordering_key)} bytes')

Type guard

def has_valid_ordering_key(msg):
    return len(getattr(msg, 'ordering_key', '') or '') <= 1024

Try / catch

try:
    proto = PubsubMessage(data=data, attributes=attrs, ordering_key=key)._to_proto_str()
except ValueError as e:
    if 'ordering key must not exceed 1024' in str(e):
        key = hashlib.sha1(key.encode()).hexdigest()
    else:
        raise

Prevention

When it happens

Trigger: Serializing a PubsubMessage whose ordering_key (a string set at construction) exceeds 1024 bytes in length.

Common situations: Using long composite keys (e.g. concatenated tenant+entity+timestamp strings) as ordering keys; deriving ordering keys from unbounded record fields.

Understand the failure class

Background: payload too large / request exceeds maximum size: why libraries cap bytes and how to fix oversize payloads — this error's family across 50 libraries.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/f78f7c18db559ce4. Report an issue: GitHub.