apache/beam · error · UnsupportedOperationException

BigQuery source must be split before being read

Error message

BigQuery source must be split before being read

What it means

BigQuerySourceBase.createReader unconditionally throws UnsupportedOperationException because a BigQuery source represents a whole table/export and must be split into file-based sub-sources (after the export job) before readers can be created. Attempting to read the un-split source directly is not supported by design.

Solutions

  1. Call split(desiredBundleSizeBytes, options) and create readers on the resulting sub-sources
  2. Use standard pipeline execution (p.apply(BigQueryIO.read...)) which handles splitting automatically
  3. Refactor test/util code to go through the runner's source-reading API rather than createReader on the raw source

Example fix

// before
BoundedReader<T> reader = bqSource.createReader(options);
// after
List<? extends BoundedSource<T>> splits = bqSource.split(400_000_000L, options);
BoundedReader<T> reader = splits.get(0).createReader(options);
Defensive patterns

Strategy: fallback

Validate before calling

// never call createReader on the raw BigQuery source
if (source instanceof BigQuerySourceBase) {
  List<? extends BoundedSource<T>> splits = source.split(desiredSizeBytes, options);
  // read from splits
}

Try / catch

try {
  reader = source.createReader(options);
} catch (UnsupportedOperationException e) {
  if (e.getMessage() != null && e.getMessage().contains("must be split before being read")) {
    List<? extends BoundedSource<T>> splits = source.split(400_000_000L, options);
    reader = splits.get(0).createReader(options);
  }
}

Prevention

When it happens

Trigger: Calling BoundedSource.createReader on a BigQuerySourceBase-derived source without first calling split() — e.g. custom runner code, direct pipeline testing utilities, or Beam APIs that read sources without splitting.

Common situations: Custom runners or test harnesses that enumerate sources manually; old code paths using createReader directly instead of letting the runner split first; migrating code from other BoundedSource implementations.

Understand the failure class

Background: UnsupportedOperationException and "is not supported" errors: when a library deliberately refuses a call — this error's family across 30 libraries.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/cfc98bb818d86f4e. Report an issue: GitHub.

Appendix: source

Thrown at sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQuerySourceBase.java:189

        // Match all files in the destination directory to stat them in bulk.
        List<MatchResult> matches = match(ImmutableList.of(extractDestinationDir + "*"));
        if (matches.size() > 0) {
          res.metadata = matches.get(0).metadata();
        }
      }
      cleanupTempResource(options.as(BigQueryOptions.class));
      cachedSplitResult = createSources(res.extractedFiles, res.schema, res.metadata);
    }
    return cachedSplitResult;
  }

  protected abstract TableReference getTableToExtract(BigQueryOptions bqOptions) throws Exception;

  protected abstract void cleanupTempResource(BigQueryOptions bqOptions) throws Exception;

  @Override
  public BoundedReader<T> createReader(PipelineOptions options) throws IOException {
    throw new UnsupportedOperationException("BigQuery source must be split before being read");
  }

  @Override
  public void validate() {
    // Do nothing, validation is done in BigQuery.Read.
  }

  @Override
  public Coder<T> getOutputCoder() {
    return coder;
  }

  private List<ResourceId> executeExtract(
      String jobId,
      TableReference table,
      JobService jobService,
      String executingProject,
      String extractDestinationDir,

View on GitHub (pinned to 12126d8942)