{"record":{"id":"dac4541922dfe6b6","repo":"apache/beam","slug":"a-pubsub-message-data-field-must-not-exceed-10mb","errorCode":null,"errorMessage":"A pubsub message data field must not exceed 10MB","messagePattern":"A pubsub message data field must not exceed 10MB","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"sdks/python/apache_beam/io/gcp/pubsub.py","lineNumber":154,"sourceCode":"  def _to_proto_str(self, for_publish=False):\n    \"\"\"Get serialized form of ``PubsubMessage``.\n\n    The serialized message is validated against pubsub message limits specified\n    at https://cloud.google.com/pubsub/quotas#resource_limits\n\n    Args:\n      proto_msg: str containing a serialized protobuf.\n      for_publish: bool, if True strip out message fields which cannot be\n        published (currently message_id and publish_time) per\n        https://cloud.google.com/pubsub/docs/reference/rpc/google.pubsub.v1#pubsubmessage\n\n    Returns:\n      A str containing a serialized protobuf of type\n      https://cloud.google.com/pubsub/docs/reference/rpc/google.pubsub.v1#google.pubsub.v1.PubsubMessage\n      containing the payload of this object.\n    \"\"\"\n    if len(self.data) > (10_000_000):\n      raise ValueError('A pubsub message data field must not exceed 10MB')\n\n    if self.attributes:\n      if len(self.attributes) > 100:\n        raise ValueError(\n            'A pubsub message must not have more than 100 attributes.')\n      for key, value in self.attributes.items():\n        if len(key) > 256:\n          raise ValueError(\n              'A pubsub message attribute key must not exceed 256 bytes.')\n        if len(value) > 1024:\n          raise ValueError(\n              'A pubsub message attribute value must not exceed 1024 bytes')\n\n    message_id = None\n    publish_time = None\n    if not for_publish:\n      if self.message_id:\n        message_id = self.message_id","sourceCodeStart":136,"sourceCodeEnd":172,"githubUrl":"https://github.com/apache/beam/blob/12126d8942aaf848030c478b4c6a28c6af861c66/sdks/python/apache_beam/io/gcp/pubsub.py#L136-L172","documentation":"PubsubMessage._to_proto_str enforces the Pub/Sub publisher limit that a message's data field may not exceed 10,000,000 bytes. Beam checks this client-side before serializing so the publish fails early with a clear error instead of a server-side rejection.","triggerScenarios":"Calling message_to_proto_str / _to_proto_str on a PubsubMessage whose data blob is larger than 10MB (len(self.data) > 10_000_000).","commonSituations":"Reading very large blobs or batched payloads from a source and emitting them as a single Pub/Sub message; aggregation windows that concatenate records into one oversized message.","solutions":["Split the payload into multiple messages smaller than 10MB and emit each.","Compress the data (e.g. gzip) before publishing if it compresses under the limit.","Store large payloads in GCS/BigQuery and publish a reference/URI in the message.","Adjust the upstream DoFn/aggregation to cap message sizes."],"exampleFix":"// before\nyield PubsubMessage(data=blob, attributes={})  # blob > 10MB\n// after\nif len(blob) > 10_000_000:\n  uri = write_to_gcs(blob)\n  yield PubsubMessage(data=uri.encode(), attributes={'ref': 'gcs'})\nelse:\n  yield PubsubMessage(data=blob, attributes={})","handlingStrategy":"validation","validationCode":"if len(data) > 10_000_000:\n    raise ValueError(f'data too large for PubsubMessage: {len(data)} bytes')","typeGuard":"def fits_pubsub_data(data):\n    return len(data) <= 10_000_000","tryCatchPattern":"try:\n    proto = PubsubMessage(data=data, attributes={})._to_proto_str()\nexcept ValueError as e:\n    if 'must not exceed 10MB' in str(e):\n        spill_to_gcs_and_emit_reference(data)\n    else:\n        raise","preventionTips":["Cap aggregation window/batch sizes so payloads stay under 10MB","Offload large blobs to GCS and publish URIs","Compress payloads when they compress well below the limit"],"tags":["pubsub","payload-too-large","size-limit","python","apache-beam"],"backgroundTag":"payload-too-large","analyzedSha":"12126d8942aaf848030c478b4c6a28c6af861c66","analyzedAt":"2026-09-13T01:50:10.254Z","contentChangedAt":"2026-09-13T01:50:10.254Z","schemaVersion":2},"datasetVersion":"2026-09-14T16:17:12.679Z"}