apache/beam · error · RuntimeException

unable to obtain dataset for dataset %s in project %s

Error message

unable to obtain dataset for dataset %s in project %s

What it means

While resolving a dataset's location (needed to pick the right BigQuery service endpoint), BigQueryHelpers calls datasetService.getDataset. If that call throws any Exception, it re-wraps it in a RuntimeException 'unable to obtain dataset for dataset %s in project %s' with the original as cause, and interrupts restore if the thread was interrupted.

Source

Thrown at sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryHelpers.java:741

    try (DatasetService datasetService = new BigQueryServicesImpl().getDatasetService(options)) {
      return datasetService.getTable(tableRef);
    } catch (IOException | InterruptedException e) {
      throw e;
    } catch (Exception e) {
      throw new RuntimeException(e);
    }
  }

  static String getDatasetLocation(
      DatasetService datasetService, String projectId, String datasetId) {
    Dataset dataset;
    try {
      dataset = datasetService.getDataset(projectId, datasetId);
    } catch (Exception e) {
      if (e instanceof InterruptedException) {
        Thread.currentThread().interrupt();
      }
      throw new RuntimeException(
          String.format(
              "unable to obtain dataset for dataset %s in project %s", datasetId, projectId),
          e);
    }
    return dataset.getLocation();
  }

  static void verifyTablePresence(DatasetService datasetService, TableReference table) {
    try {
      Table fetchedTable = datasetService.getTable(table);
      if (fetchedTable == null) {
        throw new IOException("Table does not exist.");
      }
    } catch (Exception e) {
      ApiErrorExtractor errorExtractor = new ApiErrorExtractor();
      if ((e instanceof IOException)
          && ("Table does not exist.".equals(e.getMessage())
              || errorExtractor.itemNotFound((IOException) e))) {

View on GitHub (pinned to 12126d8942)

Solutions

Check the cause for the real error; verify the dataset exists: bq show my-project:my_dataset.
Grant the service account bigquery.datasets.get (BigQuery Metadata Viewer) and ensure network access to bigquery.googleapis.com.
If the thread was interrupted, let shutdown proceed and re-run; avoid swallowing InterruptedException.

Example fix

// before
// RuntimeException: unable to obtain dataset for dataset analytics in project my-project

// after
// bq mk --dataset my-project:analytics  (or fix the project/dataset name), then re-run the pipeline

Defensive patterns

Strategy: try-catch

Validate before calling

Dataset ds = bigquery.getDataset(projectId, datasetId);
if (ds == null) {
  throw new IllegalStateException("dataset " + datasetId + " missing in project " + projectId);
}

Try / catch

try {
  String location = helper.resolveDatasetLocation(projectId, datasetId);
} catch (RuntimeException e) {
  // inspect cause; check dataset existence, IAM, and network before retry
}

Prevention

Verify the dataset exists in the expected project before pipeline launch.
Grant metadata-read IAM permissions to the pipeline service account.
Ensure egress to bigquery.googleapis.com from worker networks.
Never swallow InterruptedException; allow clean shutdown on cancel/drain.

When it happens

Trigger: Calling the helper that resolves dataset location (used by BigQueryIO to select regional endpoints) when datasets.get fails: network errors, quota, IAM denial, or thread interruption.

Common situations: Dataset name/project typo (dataset genuinely absent), service account without bigquery.datasets.get, firewall blocking googleapis.com from workers, or Dataflow cancel/drain interrupting the call.

Understand the failure class

Background: "API request failed": what wrapped HTTP errors from external APIs mean and how to find the real cause — this error's family across 29 libraries.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/f296d6b962d38fd0. Report an issue: GitHub.