apache/beam · error · RuntimeException

Unable to confirm BigQuery %1$s presence for table "%2$s"…

Error message

Unable to confirm BigQuery %1$s presence for table "%2$s". If the %1$s is created by an earlier stage of the pipeline, this validation can be disabled using #withoutValidation.

What it means

Also from verifyDatasetPresence(): when the datasets.get call fails but the error is neither a recognizable itemNotFound IOException nor a RuntimeException, the library can't classify it and throws a generic RuntimeException (UNABLE_TO_CONFIRM_PRESENCE_OF_RESOURCE_ERROR) preserving the cause. It indicates validation could not be completed, not that the dataset is definitely missing.

Solutions

Inspect the cause exception for the concrete HTTP status and fix the underlying auth/network/quota issue.
Grant the service account BigQuery Metadata Viewer (bigquery.datasets.get) permission.
Use .withoutValidation() to skip pre-flight checks if your environment intentionally restricts metadata reads.

Defensive patterns

Strategy: try-catch

Try / catch

try {
  pipeline.run().waitUntilFinish();
} catch (RuntimeException e) {
  if (e.getMessage() != null && e.getMessage().contains("Unable to confirm BigQuery")) {
    // inspect cause: auth (401/403), network, quota; fix then re-run
  }
}

Prevention

Grant bigquery.datasets.get (BigQuery Metadata Viewer) to the service account.
Confirm workers can reach googleapis.com (VPC/SCP/firewall rules).
Refresh credentials and check token validity before long runs.
Disable validation with #withoutValidation if metadata reads are restricted.

When it happens

Trigger: datasets.get throws a non-404, non-RuntimeException error during validation: network failures, 401/403 auth errors surfaced as checked exceptions, quota errors, or service unavailability.

Common situations: Expired/invalid credentials, IAM permission lacking bigquery.datasets.get, VPC/SCC network restrictions from the worker, or transient BigQuery outages during pipeline launch.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/087f81e5155f20a0. Report an issue: GitHub.

Appendix: source

Thrown at sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryHelpers.java:692

        Thread.currentThread().interrupt();
      }
      throw new RuntimeException(
          "unable to confirm BigQuery table emptiness for table " + toTableSpec(tableRef), e);
    }
  }

  static void verifyDatasetPresence(DatasetService datasetService, TableReference table) {
    try {
      datasetService.getDataset(table.getProjectId(), table.getDatasetId());
    } catch (Exception e) {
      ApiErrorExtractor errorExtractor = new ApiErrorExtractor();
      if ((e instanceof IOException) && errorExtractor.itemNotFound((IOException) e)) {
        throw new IllegalArgumentException(
            String.format(RESOURCE_NOT_FOUND_ERROR, "dataset", toTableSpec(table)), e);
      } else if (e instanceof RuntimeException) {
        throw (RuntimeException) e;
      } else {
        throw new RuntimeException(
            String.format(
                UNABLE_TO_CONFIRM_PRESENCE_OF_RESOURCE_ERROR, "dataset", toTableSpec(table)),
            e);
      }
    }
  }

  /**
   * It returns the number of rows for a given table.
   *
   * @return The number of rows in the table or null if it cannot get any estimate.
   */
  public static @Nullable BigInteger getNumRows(BigQueryOptions options, TableReference tableRef)
      throws InterruptedException, IOException {

    try (DatasetService datasetService = new BigQueryServicesImpl().getDatasetService(options)) {
      Table table = datasetService.getTable(tableRef);
      if (table == null) {

View on GitHub (pinned to 12126d8942)