{"record":{"id":"849e961539775cbf","repo":"apache/beam","slug":"bigquery-source-must-be-split-before-being-read-849e96","errorCode":null,"errorMessage":"BigQuery source must be split before being read","messagePattern":"BigQuery source must be split before being read","errorType":"exception","errorClass":"NotImplementedError","httpStatus":null,"severity":"error","filePath":"sdks/python/apache_beam/io/gcp/bigquery.py","lineNumber":898,"sourceCode":"\n    for path in self.export_result.paths:\n      source = self._create_source(path, self.export_result.coder)\n      yield SourceBundle(\n          weight=1.0, source=source, start_position=None, stop_position=None)\n\n  def get_range_tracker(self, start_position, stop_position):\n    class CustomBigQuerySourceRangeTracker(RangeTracker):\n      \"\"\"A RangeTracker that always returns positions as None.\"\"\"\n      def start_position(self):\n        return None\n\n      def stop_position(self):\n        return None\n\n    return CustomBigQuerySourceRangeTracker()\n\n  def read(self, range_tracker):\n    raise NotImplementedError('BigQuery source must be split before being read')\n\n  @check_accessible(['query'])\n  def _setup_temporary_dataset(self, bq):\n    if self.temp_dataset:\n      # Temp dataset was provided by the user so we can just return.\n      return\n    location = bq.get_query_location(\n        self._get_project(), self.query.get(), self.use_legacy_sql)\n    bq.create_temporary_dataset(\n        self._get_project(), location, kms_key=self.kms_key)\n\n  @check_accessible(['query'])\n  def _execute_query(self, bq):\n    query_job_name = bigquery_tools.generate_bq_job_name(\n        self._job_name,\n        self._source_uuid,\n        bigquery_tools.BigQueryJobTypes.QUERY,\n        '%s_%s' % (int(time.time()), random.randint(0, 1000)))","sourceCodeStart":880,"sourceCodeEnd":916,"githubUrl":"https://github.com/apache/beam/blob/12126d8942aaf848030c478b4c6a28c6af861c66/sdks/python/apache_beam/io/gcp/bigquery.py#L880-L916","documentation":"BoundedSource.read() on BigQuerySourceBase raises NotImplementedError because a BigQuery source is a logical source that must first be split into concrete sub-sources (e.g. per-stream storage API sources) before any data can be read. The Beam framework is expected to call split() and then read() on each returned sub-source; calling read() directly on the un-split source is a programming error.","triggerScenarios":"Calling read(range_tracker) directly on the BigQuery source returned by beam.io.BigQuerySource (or gcsio-backed query source) instead of splitting it first, typically in custom source code, tests, or a custom runner that bypasses the standard BoundedSource protocol.","commonSituations":"Custom I/O experimentation, unit tests that instantiate BigQuerySource and try to read() it, or a runner/DoFn harness that does not implement the split-then-read BoundedSource contract.","solutions":["Call source.split(desired_num_splits, pipeline_options) first and call read() only on the returned sub-sources.","Use the high-level beam.io.ReadFromBigQuery() transform instead of consuming the source directly, which handles splitting and reading.","If implementing a custom runner, implement the BoundedSource protocol: split() then read() each split with its RangeTracker."],"exampleFix":"// before\nsource = beam.io.BigQuerySource('project:dataset.table')\nrows = source.read(source.default_range_tracker())  # NotImplementedError\n// after\nsource = beam.io.BigQuerySource('project:dataset.table')\nfor split in source.split(1, None):\n    rows = split.read(split.default_range_tracker())\n# or simply:\n_ = p | beam.io.ReadFromBigQuery(query='SELECT ...')","handlingStrategy":"try-catch","validationCode":"if isinstance(source, BoundedSource) and source is the top-level BigQuery source:\n    splits = source.split(desired_num_splits, pipeline_options)\n    # only call read() on each split","typeGuard":"def is_split_source(src):\n    return isinstance(src, iobase.BoundedSource) and not isinstance(src, bigquery.BigQuerySourceBase)","tryCatchPattern":"try:\n    rows = source.read(rt)\nexcept NotImplementedError:\n    rows = [s.read(s.default_range_tracker()) for s in source.split(1, None)]","preventionTips":["Prefer the ReadFromBigQuery transform over consuming sources directly","Follow the BoundedSource split-then-read protocol in custom runners","Never call read() on a source you did not get from split()"],"tags":["bigquery","python","apache-beam","bounded-source"],"backgroundTag":"method-not-implemented","analyzedSha":"12126d8942aaf848030c478b4c6a28c6af861c66","analyzedAt":"2026-09-13T01:50:10.254Z","contentChangedAt":"2026-09-13T01:50:10.254Z","schemaVersion":2},"datasetVersion":"2026-09-14T16:17:12.679Z"}