apache/beam · error · RuntimeException

Unable to split TableSource

Error message

Unable to split TableSource

What it means

Thrown when splitting a BigQuery Storage API TableSource into readable bounded sources fails. Beam estimates table size, computes a desired chunk size, and calls source.split(); any exception there is wrapped in this RuntimeException.

Source

Thrown at sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIO.java:1819

                  getProjectionPushdownApplied(),
                  getDirectReadPicosTimestampPrecision());
          List<? extends BoundedSource<T>> sources;
          try {
            // This splitting logic taken from the SDF implementation of Read
            long estimatedSize = source.getEstimatedSizeBytes(bqOptions);
            // Split into pieces as close to the default desired bundle size but if that would cause
            // too few splits then prefer to split up to the default desired number of splits.
            long desiredChunkSize;
            if (estimatedSize <= 0) {
              desiredChunkSize = 64 << 20; // 64mb
            } else {
              // 1mb --> 1 shard; 1gb --> 32 shards; 1tb --> 1000 shards, 1pb --> 32k shards
              desiredChunkSize =
                  Math.max(1 << 20, (long) (1000 * Math.sqrt((double) estimatedSize)));
            }
            sources = source.split(desiredChunkSize, bqOptions);
          } catch (Exception e) {
            throw new RuntimeException("Unable to split TableSource", e);
          }
          TupleTag<T> rowTag = new TupleTag<>();
          PCollectionTuple resultTuple =
              p.apply(Create.of(sources))
                  .apply(
                      "Read Storage Table Source",
                      ParDo.of(new ReadTableSource<T>(rowTag, parseFn, getBadRecordRouter()))
                          .withOutputTags(rowTag, TupleTagList.of(BAD_RECORD_TAG)));
          getBadRecordErrorHandler()
              .addErrorCollection(
                  resultTuple
                      .get(BAD_RECORD_TAG)
                      .setCoder(BadRecord.getCoder(input.getPipeline())));

          return resultTuple.get(rowTag).setCoder(outputCoder);
        }
      }

View on GitHub (pinned to 12126d8942)

Solutions

  1. Enable the BigQuery Storage Read API and grant storage read permissions
  2. Retry — split failures are often transient gRPC errors
  3. Fall back to the EXPORT/FILE_LOADS method if the Storage API is unavailable

Example fix

// before
BigQueryIO.read().from("proj:ds.tbl").withMethod(TypedRead.Method.DIRECT_READ);
// after
BigQueryIO.read().from("proj:ds.tbl").withMethod(TypedRead.Method.EXPORT);
Defensive patterns

Strategy: retry

Validate before calling

// Pre-check Storage API access
try (BigQueryReadClient client = BigQueryReadClient.create()) {
  // a session creation against a tiny table verifies access
} catch (Exception e) {
  throw new IllegalStateException("Storage Read API unavailable: " + e.getMessage(), e);
}

Try / catch

try {
  List<BoundedSource<T>> parts = source.split(chunkSize, options);
} catch (Exception e) {
  // transient gRPC/API error: back off and retry, or fall back to EXPORT method
  throw new RuntimeException("Unable to split TableSource", e);
}

Prevention

When it happens

Trigger: Using BigQueryIO.read with the STORAGE_API (DIRECT_READ) method and source.split(desiredChunkSize, options) throws — typically a Storage API session creation failure.

Common situations: Storage Read API not enabled on the project; insufficient permissions for storage reads; transient gRPC/API errors; table too small or estimated size issues.

Understand the failure class

Background: "API request failed": what wrapped HTTP errors from external APIs mean and how to find the real cause — this error's family across 29 libraries.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/51963f1170150856. Report an issue: GitHub.