{"record":{"id":"b84648567d18ed9d","repo":"apache/beam","slug":"serialized-pubsub-message-exceeds-the-publish-request-limit","errorCode":null,"errorMessage":"Serialized pubsub message exceeds the publish request limit of 10MB","messagePattern":"Serialized pubsub message exceeds the publish request limit of 10MB","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"sdks/python/apache_beam/io/gcp/pubsub.py","lineNumber":188,"sourceCode":"    if not for_publish:\n      if self.message_id:\n        message_id = self.message_id\n        if self.publish_time:\n          publish_time = self.publish_time\n\n    if len(self.ordering_key) > 1024:\n      raise ValueError(\n          'A pubsub message ordering key must not exceed 1024 bytes.')\n\n    msg = pubsub.types.PubsubMessage(\n        data=self.data,\n        attributes=self.attributes,\n        message_id=message_id,\n        publish_time=publish_time,\n        ordering_key=self.ordering_key)\n    serialized = pubsub.types.PubsubMessage.serialize(msg)\n    if len(serialized) > (10_000_000):\n      raise ValueError(\n          'Serialized pubsub message exceeds the publish request limit of 10MB')\n    return serialized\n\n  @staticmethod\n  def _from_message(msg: Any) -> 'PubsubMessage':\n    \"\"\"Construct from ``google.cloud.pubsub_v1.subscriber.message.Message``.\n\n    https://googleapis.github.io/google-cloud-python/latest/pubsub/subscriber/api/message.html\n    \"\"\"\n    # Convert ScalarMapContainer to dict.\n    attributes = dict(msg.attributes)\n    pubsubmessage = PubsubMessage(msg.data, attributes)\n    if msg.message_id:\n      pubsubmessage.message_id = msg.message_id\n    if msg.publish_time:\n      pubsubmessage.publish_time = msg.publish_time\n    if msg.ordering_key:\n      pubsubmessage.ordering_key = msg.ordering_key","sourceCodeStart":170,"sourceCodeEnd":206,"githubUrl":"https://github.com/apache/beam/blob/12126d8942aaf848030c478b4c6a28c6af861c66/sdks/python/apache_beam/io/gcp/pubsub.py#L170-L206","documentation":"After building the PubsubMessage proto, _to_proto_str serializes it and enforces the overall publish request limit of 10MB on the serialized message. Even if individual fields are within their own limits, the combined serialized protobuf must not exceed 10,000,000 bytes.","triggerScenarios":"message_to_proto_str / _to_proto_str on a message whose serialized protobuf (data + attributes + metadata) exceeds 10,000,000 bytes.","commonSituations":"Messages with near-limit data plus many attributes pushing total size over 10MB; many moderately-sized attributes accumulating with a large payload.","solutions":["Reduce total message size: shrink data and/or trim attributes so the serialized message is under 10MB.","Split into multiple messages below the limit.","Offload the bulk payload to GCS/BigQuery and publish a reference with small attributes.","Add a pre-serialization size check in the emitting DoFn to split or spill oversized messages."],"exampleFix":"// before\nyield PubsubMessage(data=payload, attributes=all_metadata)  # serialized > 10MB\n// after\nref = write_to_gcs(payload)\nyield PubsubMessage(data=ref.encode(), attributes={'size': str(len(payload))})","handlingStrategy":"validation","validationCode":"import json\nattrs_json = json.dumps(attributes or {}, default=str)\nif len(data) + len(attrs_json) > 9_500_000:\n    raise ValueError('message may exceed serialized 10MB publish limit')","typeGuard":"def fits_publish_limit(data, attributes):\n    return len(data) + len(json.dumps(attributes or {}, default=str)) <= 10_000_000","tryCatchPattern":"try:\n    proto = message_to_proto_str(msg)\nexcept ValueError as e:\n    if 'exceeds the publish request limit' in str(e):\n        split_or_spill_message(msg)\n    else:\n        raise","preventionTips":["Estimate total serialized size (data + attributes) before constructing messages","Split large batches into multiple messages during aggregation","Offload bulk payloads to GCS and publish references"],"tags":["pubsub","payload-too-large","serialization","python","apache-beam"],"backgroundTag":"payload-too-large","analyzedSha":"12126d8942aaf848030c478b4c6a28c6af861c66","analyzedAt":"2026-09-13T01:50:10.254Z","contentChangedAt":"2026-09-13T01:50:10.254Z","schemaVersion":2},"datasetVersion":"2026-09-14T16:17:12.679Z"}