apache/beam · error · IOException

Unable to create dataset: %s, aborting after %d .

Error message

Unable to create dataset: %s, aborting after %d .

What it means

Thrown by BigQueryServicesImpl.createDataset after exhausting MAX_RPC_RETRIES attempts to create a BigQuery dataset via the API. Each attempt catches IOException, logs 'Ignore the error and retry', backs off, and retries; when the backoff loop ends the last exception is wrapped in this IOException. It means the dataset could not be created due to a persistent, repeated API failure.

Source

Thrown at sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryServicesImpl.java:952

      Exception lastException;
      do {
        try {
          client.datasets().insert(projectId, dataset).setPrettyPrint(false).execute();
          return; // SUCCEEDED
        } catch (GoogleJsonResponseException e) {
          if (errorExtractor.itemAlreadyExists(e)) {
            return; // SUCCEEDED
          }
          // ignore and retry
          LOG.info("Ignore the error and retry creating the dataset.", e);
          lastException = e;
        } catch (IOException e) {
          LOG.info("Ignore the error and retry creating the dataset.", e);
          lastException = e;
        }
      } while (nextBackOff(sleeper, backoff));
      throw new IOException(
          String.format(
              "Unable to create dataset: %s, aborting after %d .", datasetId, MAX_RPC_RETRIES),
          lastException);
    }

    /**
     * {@inheritDoc}
     *
     * <p>Tries executing the RPC for at most {@code MAX_RPC_RETRIES} times until it succeeds.
     *
     * @throws IOException if it exceeds {@code MAX_RPC_RETRIES} attempts.
     */
    @Override
    public void deleteDataset(String projectId, String datasetId)
        throws IOException, InterruptedException {
      executeWithRetries(
          client.datasets().delete(projectId, datasetId),
          String.format(

View on GitHub (pinned to 12126d8942)

Solutions

1. Check the resolved project/dataset IDs and confirm the dataset doesn't already exist with conflicting permissions
2. Grant the service account roles/bigquery.dataEditor (includes datasets.create) on the project
3. Verify billing is enabled and BigQuery API is enabled for the project
4. Inspect the wrapped lastException (getCause()) for the real per-attempt error and address it
5. Retry later if the underlying API was returning 5xx during an outage

Example fix

// before
pipeline.apply(BigQueryIO.writeTableRows().to("my-project:wrong_dataset.table")...);
// after
pipeline.apply(BigQueryIO.writeTableRows().to("my-project:correct_dataset.table")...);
// and ensure: bq mk --dataset my-project:correct_dataset, or fix IAM

// pre-check with bq CLI or client
BigQuery bq = options.as(BigQueryOptions.class).getBigQueryServices().getBigQuery(options.as(BigQueryOptions.class));
DatasetId dsId = DatasetId.of(projectId, datasetId);
if (bq.getDataset(dsId) == null) {
  // confirm service account has bigquery.datasets.create and billing is enabled
}

try {
  pipeline.run().waitUntilFinish();
} catch (IOException e) {
  if (e.getMessage() != null && e.getMessage().contains("Unable to create dataset")) {
    Throwable cause = e.getCause(); // real API error
    // handle permission/quota/project errors
  }
  throw e;
}

When it happens

Trigger: Calling BigQuery IO that requires the destination dataset to exist with dataset auto-creation enabled, while every create-dataset RPC fails (e.g. project not found, permission denied, quota exceeded, repeated 5xx/503 from the BigQuery API) for MAX_RPC_RETRIES consecutive attempts.

Common situations: Wrong project ID or dataset ID in the table spec; service account lacking bigquery.datasets.create permission; billing not enabled on the project; transient BigQuery API outages lasting longer than the retry budget.

Understand the failure class

Background: "API request failed": what wrapped HTTP errors from external APIs mean and how to find the real cause — this error's family across 29 libraries.

Related errors

• Unable to insert job: %s, aborting after %d .
• Unable to find BigQuery job: %s, aborting after %d retries.
• Failed to patch table schema.
• More than %d attempts to call AppendRows failed. Last encoun
• More than %d attempts to call AppendRows failed. Last encoun

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/d8fa9a68d2fda89f. Report an issue: GitHub.