apache/beam · error · RuntimeException

Unable to confirm BigQuery %1$s presence for table "%2$s". I

Error message

Unable to confirm BigQuery %1$s presence for table "%2$s". If the %1$s is created by an earlier stage of the pipeline, this validation can be disabled using #withoutValidation.

What it means

Also from verifyDatasetPresence(): when the datasets.get call fails but the error is neither a recognizable itemNotFound IOException nor a RuntimeException, the library can't classify it and throws a generic RuntimeException (UNABLE_TO_CONFIRM_PRESENCE_OF_RESOURCE_ERROR) preserving the cause. It indicates validation could not be completed, not that the dataset is definitely missing.

Source

Thrown at sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryHelpers.java:692

        Thread.currentThread().interrupt();
      }
      throw new RuntimeException(
          "unable to confirm BigQuery table emptiness for table " + toTableSpec(tableRef), e);
    }
  }

  static void verifyDatasetPresence(DatasetService datasetService, TableReference table) {
    try {
      datasetService.getDataset(table.getProjectId(), table.getDatasetId());
    } catch (Exception e) {
      ApiErrorExtractor errorExtractor = new ApiErrorExtractor();
      if ((e instanceof IOException) && errorExtractor.itemNotFound((IOException) e)) {
        throw new IllegalArgumentException(
            String.format(RESOURCE_NOT_FOUND_ERROR, "dataset", toTableSpec(table)), e);
      } else if (e instanceof RuntimeException) {
        throw (RuntimeException) e;
      } else {
        throw new RuntimeException(
            String.format(
                UNABLE_TO_CONFIRM_PRESENCE_OF_RESOURCE_ERROR, "dataset", toTableSpec(table)),
            e);
      }
    }
  }

  /**
   * It returns the number of rows for a given table.
   *
   * @return The number of rows in the table or null if it cannot get any estimate.
   */
  public static @Nullable BigInteger getNumRows(BigQueryOptions options, TableReference tableRef)
      throws InterruptedException, IOException {

    try (DatasetService datasetService = new BigQueryServicesImpl().getDatasetService(options)) {
      Table table = datasetService.getTable(tableRef);
      if (table == null) {

View on GitHub (pinned to 12126d8942)

Solutions

1. Inspect the cause exception for the concrete HTTP status and fix the underlying auth/network/quota issue.
2. Grant the service account BigQuery Metadata Viewer (bigquery.datasets.get) permission.
3. Use .withoutValidation() to skip pre-flight checks if your environment intentionally restricts metadata reads.

try {
  pipeline.run().waitUntilFinish();
} catch (RuntimeException e) {
  if (e.getMessage() != null && e.getMessage().contains("Unable to confirm BigQuery")) {
    // inspect cause: auth (401/403), network, quota; fix then re-run
  }
}

When it happens

Trigger: datasets.get throws a non-404, non-RuntimeException error during validation: network failures, 401/403 auth errors surfaced as checked exceptions, quota errors, or service unavailability.

Common situations: Expired/invalid credentials, IAM permission lacking bigquery.datasets.get, VPC/SCC network restrictions from the worker, or transient BigQuery outages during pipeline launch.

Related errors

• unable to confirm BigQuery table emptiness for table %s
• BigQuery %1$s not found for table "%2$s" . Please create the
• unable to obtain dataset for dataset %s in project %s
• Validation of query "%1$s" failed. If the query depends on a
• invalid table ID %s. Table IDs must be alphanumeric (plus un

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/087f81e5155f20a0. Report an issue: GitHub.