{"record":{"id":"d4a2cee8a50e01e6","repo":"apache/beam","slug":"buffer-size-size-exceeds-grpc-limit-flush-max-size-this-is","errorCode":null,"errorMessage":"Buffer size {size} exceeds GRPC limit {_FLUSH_MAX_SIZE}. This is likely due to a single element that is too large. To resolve, prefer multiple small elements over single large elements in PCollections. If needed, store large blobs in external storage systems, and use PCollections to pass their metadata, or use a custom coder that reduces the element's size.","messagePattern":"Buffer size (.+?) exceeds GRPC limit (.+?)\\. This is likely due to a single element that is too large\\. To resolve, prefer multiple small elements over single large elements in PCollections\\. If needed, store large blobs in external storage systems, and use PCollections to pass their metadata, or use a custom coder that reduces the element's size\\.","errorType":"exception","errorClass":"ValueError","httpStatus":null,"severity":"critical","filePath":"sdks/python/apache_beam/runners/worker/data_plane.py","lineNumber":162,"sourceCode":"    super().__init__(close_callback)\n    self._flush_callback = flush_callback\n    self._size_flush_threshold = size_flush_threshold\n    self._large_buffer_warn_threshold_bytes = large_buffer_warn_threshold_bytes\n\n  # This must be called explicitly to avoid flushing partial elements.\n  def maybe_flush(self):\n    # type: () -> None\n    if self.size() > self._size_flush_threshold:\n      self.flush()\n\n  def flush(self):\n    # type: () -> None\n    if self._flush_callback:\n      size = self.size()\n      if (self._large_buffer_warn_threshold_bytes and\n          size > self._large_buffer_warn_threshold_bytes):\n        if size > _FLUSH_MAX_SIZE:\n          raise ValueError(\n              f'Buffer size {size} exceeds GRPC limit {_FLUSH_MAX_SIZE}. '\n              'This is likely due to a single element that is too large. '\n              'To resolve, prefer multiple small elements over single large '\n              'elements in PCollections. If needed, store large blobs in '\n              'external storage systems, and use PCollections to pass their '\n              'metadata, or use a custom coder that reduces the element\\'s '\n              'size.')\n\n        if self._large_flush_last_observed_timestamp + 600 < time.time():\n          self._large_flush_last_observed_timestamp = time.time()\n          _LOGGER.warning(\n              'Data output stream buffer size %s exceeds %s bytes. '\n              'This is likely due to a large element in a PCollection. '\n              'Large elements increase pipeline RAM requirements and '\n              'can cause runtime errors. '\n              'Prefer multiple small elements over single large elements '\n              'in PCollections. If needed, store large blobs in external '\n              'storage systems, and use PCollections to pass their metadata, '","sourceCodeStart":144,"sourceCodeEnd":180,"githubUrl":"https://github.com/apache/beam/blob/12126d8942aaf848030c478b4c6a28c6af861c66/sdks/python/apache_beam/runners/worker/data_plane.py#L144-L180","documentation":"The worker's outbound data buffer exceeds _FLUSH_MAX_SIZE (the gRPC message limit, ~2GB by protocol but capped lower here) at flush time, meaning buffered encoded elements cannot be sent as one message. This usually indicates a single enormous element or a huge backlog between flushes, and the worker raises ValueError instead of sending an impossible gRPC message.","triggerScenarios":"maybe_flush -> flush() computes size() > _FLUSH_MAX_SIZE because one encoded element (or an un-flushed accumulation) exceeds the gRPC message limit in data_plane.py.","commonSituations":"PCollections carrying very large rows/blobs (large JSON, embedded binaries, huge images); forgetting to flush/checkpoint large outputs; missing externalization of large payloads.","solutions":["Split large elements into multiple smaller elements before writing to PCollections","Store large blobs in external storage (GCS/S3) and pass only references/metadata through PCollections","Implement a custom coder that compresses or reduces encoded element size","Ensure the sink/DoFn doesn't buffer many elements into one output (e.g. giant lists); flush per element","Increase parallel sharding so fewer elements accumulate per output buffer"],"exampleFix":"// before\nyield {'id': row_id, 'payload': huge_blob}  # single 3GB element\n// after\nblob_ref = upload_to_gcs(huge_blob)\nyield {'id': row_id, 'payload_ref': blob_ref}  # small metadata element","handlingStrategy":"validation","validationCode":"MAX_ENCODED = 64 * 1024 * 1024  # stay well below gRPC/flush limits\ndef element_too_large(elem):\n    encoded = sys.getsizeof(repr(elem))  # or use the actual coder to measure\n    return encoded > MAX_ENCODED\n# reject or split before writing to the PCollection","typeGuard":null,"tryCatchPattern":"try:\n    out = pcoll | beam.Map(emit)\n    pipeline.run().wait_until_finish()\nexcept ValueError as e:\n    if 'exceeds GRPC limit' in str(e):\n        logging.error('Element(s) too large for gRPC data plane; externalize blobs')\n        raise\n    raise","preventionTips":["Keep encoded elements small (KB-scale); reference external blobs instead of embedding them","Split giant JSON/rows into multiple elements or chunks","Use a compact/compressing custom coder for large payloads","Monitor the 'large buffer' warnings in worker logs as an early signal before the hard limit"],"tags":["apache-beam","python","grpc","payload-size"],"backgroundTag":"payload-too-large","analyzedSha":"12126d8942aaf848030c478b4c6a28c6af861c66","analyzedAt":"2026-09-13T01:50:10.254Z","contentChangedAt":"2026-09-13T01:50:10.254Z","schemaVersion":2},"datasetVersion":"2026-09-14T16:17:12.679Z"}