{"record":{"id":"08a82d4003951db6","repo":"apache/beam","slug":"bigquery-storage-source-must-be-split-before-being-read","errorCode":null,"errorMessage":"BigQuery storage source must be split before being read","messagePattern":"BigQuery storage source must be split before being read","errorType":"exception","errorClass":"NotImplementedError","httpStatus":null,"severity":"error","filePath":"sdks/python/apache_beam/io/gcp/bigquery.py","lineNumber":1310,"sourceCode":"\n    for source in self.split_result:\n      yield SourceBundle(\n          weight=1.0, source=source, start_position=None, stop_position=None)\n\n  def get_range_tracker(self, start_position, stop_position):\n    class NonePositionRangeTracker(RangeTracker):\n      \"\"\"A RangeTracker that always returns positions as None. Prevents the\n      BigQuery Storage source from being read() before being split().\"\"\"\n      def start_position(self):\n        return None\n\n      def stop_position(self):\n        return None\n\n    return NonePositionRangeTracker()\n\n  def read(self, range_tracker):\n    raise NotImplementedError(\n        'BigQuery storage source must be split before being read')\n\n\nclass _CustomBigQueryStorageStreamSource(BoundedSource):\n  \"\"\"A source representing a single stream in a read session.\"\"\"\n\n  # Runner will act on this counter on scaling event, if supported\n  THROTTLE_COUNTER = Metrics.counter(__name__, 'cumulativeThrottlingSeconds')\n\n  def __init__(\n      self,\n      read_stream_name: str,\n      use_native_datetime: Optional[bool] = True,\n      timeout: Optional[float] = None):\n    self.read_stream_name = read_stream_name\n    self.use_native_datetime = use_native_datetime\n    self.timeout = timeout\n","sourceCodeStart":1292,"sourceCodeEnd":1328,"githubUrl":"https://github.com/apache/beam/blob/12126d8942aaf848030c478b4c6a28c6af861c66/sdks/python/apache_beam/io/gcp/bigquery.py#L1292-L1328","documentation":"The BigQuery storage API source (ReadFromBigQuery with method=STORAGE_API_READ) likewise raises NotImplementedError from read(): the top-level source is only a placeholder that must be split into _CustomBigQueryStorageStreamSource objects (one per read-session stream) before reading. Direct read() on the un-split storage source is invalid.","triggerScenarios":"Calling read(range_tracker) directly on the source produced for BigQuery storage API reads instead of first calling split(), e.g. in custom runner code or tests exercising the storage API path.","commonSituations":"Testing the storage API read path manually, or a custom/direct runner that bypasses BoundedSource.split before read.","solutions":["Call source.split() and read only the returned stream sub-sources.","Use the beam.io.ReadFromBigQuery(method=THE_STORAGE_API) transform so the framework performs split/read.","Fix custom runner code to follow the BoundedSource split-then-read protocol."],"exampleFix":"// before\nsrc = beam.io.bigquery._CustomBigQueryStorageSource(...)\nrows = src.read(src.default_range_tracker())  # NotImplementedError\n// after\nfor sub in src.split(float('inf'), None):\n    rows = sub.read(sub.default_range_tracker())","handlingStrategy":"try-catch","validationCode":"splits = source.split(float('inf'), None)\nassert splits, 'storage source must produce stream sub-sources before read()'","typeGuard":"def is_stream_sub_source(src):\n    return isinstance(src, _CustomBigQueryStorageStreamSource)","tryCatchPattern":"try:\n    rows = source.read(rt)\nexcept NotImplementedError:\n    rows = [s.read(s.default_range_tracker()) for s in source.split(float('inf'), None)]","preventionTips":["Use ReadFromBigQuery(method=THE_STORAGE_API) instead of raw sources","Always split() a BoundedSource before read()","Test custom runners against the split-then-read contract"],"tags":["bigquery","python","apache-beam","storage-api"],"backgroundTag":"method-not-implemented","analyzedSha":"12126d8942aaf848030c478b4c6a28c6af861c66","analyzedAt":"2026-09-13T01:50:10.254Z","contentChangedAt":"2026-09-13T01:50:10.254Z","schemaVersion":2},"datasetVersion":"2026-09-20T03:17:13.778Z"}