apache/beam · error · IllegalArgumentException

BigQuery %1$s not found for table "%2$s" . Please create the

Error message

BigQuery %1$s not found for table "%2$s" . Please create the %1$s before pipeline execution. If the %1$s is created by an earlier stage of the pipeline, this validation can be disabled using #withoutValidation.

What it means

verifyDatasetPresence() checks that the dataset backing the target table exists before running the pipeline. When the BigQuery API returns an itemNotFound error for datasets.get, the library throws IllegalArgumentException with RESOURCE_NOT_FOUND_ERROR telling the developer to create the dataset first or disable validation with #withoutValidation.

Source

Thrown at sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryHelpers.java:687

            "BigQuery table is not empty: %s.",
            toTableSpec(tableRef));
      }
    } catch (IOException | InterruptedException e) {
      if (e instanceof InterruptedException) {
        Thread.currentThread().interrupt();
      }
      throw new RuntimeException(
          "unable to confirm BigQuery table emptiness for table " + toTableSpec(tableRef), e);
    }
  }

  static void verifyDatasetPresence(DatasetService datasetService, TableReference table) {
    try {
      datasetService.getDataset(table.getProjectId(), table.getDatasetId());
    } catch (Exception e) {
      ApiErrorExtractor errorExtractor = new ApiErrorExtractor();
      if ((e instanceof IOException) && errorExtractor.itemNotFound((IOException) e)) {
        throw new IllegalArgumentException(
            String.format(RESOURCE_NOT_FOUND_ERROR, "dataset", toTableSpec(table)), e);
      } else if (e instanceof RuntimeException) {
        throw (RuntimeException) e;
      } else {
        throw new RuntimeException(
            String.format(
                UNABLE_TO_CONFIRM_PRESENCE_OF_RESOURCE_ERROR, "dataset", toTableSpec(table)),
            e);
      }
    }
  }

  /**
   * It returns the number of rows for a given table.
   *
   * @return The number of rows in the table or null if it cannot get any estimate.
   */
  public static @Nullable BigInteger getNumRows(BigQueryOptions options, TableReference tableRef)

View on GitHub (pinned to 12126d8942)

Solutions

Create the dataset before running the pipeline (bq mk or Cloud Console).
Fix typos / verify the correct project and dataset in the table spec.
If the dataset is created by an earlier stage of the same pipeline, disable validation with .withoutValidation().

Example fix

// before
String spec = "my-project:nonexistent_dataset.my_table"; // dataset missing

// after
// run: bq mk --dataset my-project:nonexistent_dataset
String spec = "my-project:nonexistent_dataset.my_table";

Defensive patterns

Strategy: validation

Validate before calling

// before running the pipeline
Dataset dataset = bigquery.getDataset(projectId, datasetId);
if (dataset == null) {
  throw new IllegalStateException("Dataset " + datasetId + " does not exist in " + projectId);
}

Try / catch

try {
  pipeline.run().waitUntilFinish();
} catch (IllegalArgumentException e) {
  if (e.getMessage().contains("not found") && e.getMessage().contains("dataset")) {
    // create the dataset or switch to withoutValidation()
  }
}

Prevention

Verify dataset existence with 'bq show project:dataset' before submission.
Create datasets via IaC/CLI before deploying pipelines that depend on them.
Avoid typos by building specs from constants/config rather than ad-hoc strings.
Use #withoutValidation when resources are created mid-pipeline.

When it happens

Trigger: BigQueryIO write with validation enabled where the referenced dataset does not exist in the project at pipeline submission time (datasets.get returns 404).

Common situations: Typo in the dataset name, wrong project, dataset expected to be created by an earlier pipeline stage but validation runs before it, or environment (prod vs staging) mismatch.

Understand the failure class

Background: 'Could not be found', 'does not exist', 'not found in database': the resource-not-found family when an ID, slug, key, or URI lookup comes back empty — this error's family across 20 libraries.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/98e40140c03f1591. Report an issue: GitHub.