{"record":{"id":"cfc98bb818d86f4e","repo":"apache/beam","slug":"bigquery-source-must-be-split-before-being-read","errorCode":null,"errorMessage":"BigQuery source must be split before being read","messagePattern":"BigQuery source must be split before being read","errorType":"exception","errorClass":"UnsupportedOperationException","httpStatus":null,"severity":"error","filePath":"sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQuerySourceBase.java","lineNumber":189,"sourceCode":"        // Match all files in the destination directory to stat them in bulk.\n        List<MatchResult> matches = match(ImmutableList.of(extractDestinationDir + \"*\"));\n        if (matches.size() > 0) {\n          res.metadata = matches.get(0).metadata();\n        }\n      }\n      cleanupTempResource(options.as(BigQueryOptions.class));\n      cachedSplitResult = createSources(res.extractedFiles, res.schema, res.metadata);\n    }\n    return cachedSplitResult;\n  }\n\n  protected abstract TableReference getTableToExtract(BigQueryOptions bqOptions) throws Exception;\n\n  protected abstract void cleanupTempResource(BigQueryOptions bqOptions) throws Exception;\n\n  @Override\n  public BoundedReader<T> createReader(PipelineOptions options) throws IOException {\n    throw new UnsupportedOperationException(\"BigQuery source must be split before being read\");\n  }\n\n  @Override\n  public void validate() {\n    // Do nothing, validation is done in BigQuery.Read.\n  }\n\n  @Override\n  public Coder<T> getOutputCoder() {\n    return coder;\n  }\n\n  private List<ResourceId> executeExtract(\n      String jobId,\n      TableReference table,\n      JobService jobService,\n      String executingProject,\n      String extractDestinationDir,","sourceCodeStart":171,"sourceCodeEnd":207,"githubUrl":"https://github.com/apache/beam/blob/12126d8942aaf848030c478b4c6a28c6af861c66/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQuerySourceBase.java#L171-L207","documentation":"BigQuerySourceBase.createReader unconditionally throws UnsupportedOperationException because a BigQuery source represents a whole table/export and must be split into file-based sub-sources (after the export job) before readers can be created. Attempting to read the un-split source directly is not supported by design.","triggerScenarios":"Calling BoundedSource.createReader on a BigQuerySourceBase-derived source without first calling split() — e.g. custom runner code, direct pipeline testing utilities, or Beam APIs that read sources without splitting.","commonSituations":"Custom runners or test harnesses that enumerate sources manually; old code paths using createReader directly instead of letting the runner split first; migrating code from other BoundedSource implementations.","solutions":["Call split(desiredBundleSizeBytes, options) and create readers on the resulting sub-sources","Use standard pipeline execution (p.apply(BigQueryIO.read...)) which handles splitting automatically","Refactor test/util code to go through the runner's source-reading API rather than createReader on the raw source"],"exampleFix":"// before\nBoundedReader<T> reader = bqSource.createReader(options);\n// after\nList<? extends BoundedSource<T>> splits = bqSource.split(400_000_000L, options);\nBoundedReader<T> reader = splits.get(0).createReader(options);","handlingStrategy":"fallback","validationCode":"// never call createReader on the raw BigQuery source\nif (source instanceof BigQuerySourceBase) {\n  List<? extends BoundedSource<T>> splits = source.split(desiredSizeBytes, options);\n  // read from splits\n}","typeGuard":null,"tryCatchPattern":"try {\n  reader = source.createReader(options);\n} catch (UnsupportedOperationException e) {\n  if (e.getMessage() != null && e.getMessage().contains(\"must be split before being read\")) {\n    List<? extends BoundedSource<T>> splits = source.split(400_000_000L, options);\n    reader = splits.get(0).createReader(options);\n  }\n}","preventionTips":["Read BigQuery sources through p.apply(BigQueryIO.read...) instead of raw source APIs","In custom runners/test harnesses, always split BoundedSources before reading","Don't copy createReader patterns from simple file sources to BigQuery sources"],"tags":["bigquery","gcp","unsupported-operation","bounded-source"],"backgroundTag":"unsupported-operation","analyzedSha":"12126d8942aaf848030c478b4c6a28c6af861c66","analyzedAt":"2026-09-13T01:50:10.254Z","contentChangedAt":"2026-09-13T01:50:10.254Z","schemaVersion":2},"datasetVersion":"2026-09-20T03:17:13.778Z"}