apache/beam · error · RuntimeException

unable to confirm BigQuery table emptiness for table %s

Error message

unable to confirm BigQuery table emptiness for table %s

What it means

During pipeline validation, Beam queries the BigQuery API to confirm whether a target table exists and is empty. If that verification call fails with IOException or InterruptedException, it cannot determine table emptiness and wraps the error in a RuntimeException mentioning the table spec. The original exception is preserved as the cause.

Source

Thrown at sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryHelpers.java:676

  static String randomUUIDString() {
    return UUID.randomUUID().toString().replaceAll("-", "");
  }

  static void verifyTableNotExistOrEmpty(DatasetService datasetService, TableReference tableRef) {
    try {
      if (datasetService.getTable(
              tableRef, Collections.emptyList(), DatasetService.TableMetadataView.BASIC)
          != null) {
        checkState(
            datasetService.isTableEmpty(tableRef),
            "BigQuery table is not empty: %s.",
            toTableSpec(tableRef));
      }
    } catch (IOException | InterruptedException e) {
      if (e instanceof InterruptedException) {
        Thread.currentThread().interrupt();
      }
      throw new RuntimeException(
          "unable to confirm BigQuery table emptiness for table " + toTableSpec(tableRef), e);
    }
  }

  static void verifyDatasetPresence(DatasetService datasetService, TableReference table) {
    try {
      datasetService.getDataset(table.getProjectId(), table.getDatasetId());
    } catch (Exception e) {
      ApiErrorExtractor errorExtractor = new ApiErrorExtractor();
      if ((e instanceof IOException) && errorExtractor.itemNotFound((IOException) e)) {
        throw new IllegalArgumentException(
            String.format(RESOURCE_NOT_FOUND_ERROR, "dataset", toTableSpec(table)), e);
      } else if (e instanceof RuntimeException) {
        throw (RuntimeException) e;
      } else {
        throw new RuntimeException(
            String.format(
                UNABLE_TO_CONFIRM_PRESENCE_OF_RESOURCE_ERROR, "dataset", toTableSpec(table)),

View on GitHub (pinned to 12126d8942)

Solutions

Inspect the cause to see whether it's a network error, quota issue, or permission problem and fix that root cause.
Grant the pipeline service account BigQuery Data Viewer/Job User roles.
Disable validation with .withoutValidation() if you intentionally don't need pre-write emptiness checks.

Example fix

// before
BigQueryIO.writeTableRows().to(spec).withCreateDisposition(CREATE_IF_NEEDED); // validation on

// after
BigQueryIO.writeTableRows().to(spec)
    .withCreateDisposition(CREATE_IF_NEEDED)
    .withoutValidation();

Defensive patterns

Strategy: retry

Try / catch

try {
  pipeline.run().waitUntilFinish();
} catch (RuntimeException e) {
  if (e.getMessage() != null && e.getMessage().contains("unable to confirm BigQuery table emptiness")) {
    // inspect cause (IOException/InterruptedException) and retry after network/quota check
  }
}

Prevention

Ensure reliable network/auth from workers to bigquery.googleapis.com.
Grant read permissions on the target table's project to the service account.
Use .withoutValidation() when pre-flight checks aren't required.
Retry pipelines on transient API errors instead of failing permanently.

When it happens

Trigger: BigQueryIO write with validation enabled calling verifyTableEmptyness when the tabledata/jobs API call throws IOException (network, quota, permissions) or the thread is interrupted while checking the table.

Common situations: Transient GCP network issues or token refresh failures during pipeline startup, insufficient IAM permissions to read the table, quota/rate limiting, or Dataflow worker shutdown interrupting the check.

Understand the failure class

Background: "API request failed": what wrapped HTTP errors from external APIs mean and how to find the real cause — this error's family across 29 libraries.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/869b0529e7c3ed3d. Report an issue: GitHub.