{"record":{"id":"d8fa9a68d2fda89f","repo":"apache/beam","slug":"unable-to-create-dataset-s-aborting-after-d","errorCode":null,"errorMessage":"Unable to create dataset: %s, aborting after %d .","messagePattern":"Unable to create dataset: (.+?), aborting after (.+?) \\.","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryServicesImpl.java","lineNumber":952,"sourceCode":"\n      Exception lastException;\n      do {\n        try {\n          client.datasets().insert(projectId, dataset).setPrettyPrint(false).execute();\n          return; // SUCCEEDED\n        } catch (GoogleJsonResponseException e) {\n          if (errorExtractor.itemAlreadyExists(e)) {\n            return; // SUCCEEDED\n          }\n          // ignore and retry\n          LOG.info(\"Ignore the error and retry creating the dataset.\", e);\n          lastException = e;\n        } catch (IOException e) {\n          LOG.info(\"Ignore the error and retry creating the dataset.\", e);\n          lastException = e;\n        }\n      } while (nextBackOff(sleeper, backoff));\n      throw new IOException(\n          String.format(\n              \"Unable to create dataset: %s, aborting after %d .\", datasetId, MAX_RPC_RETRIES),\n          lastException);\n    }\n\n    /**\n     * {@inheritDoc}\n     *\n     * <p>Tries executing the RPC for at most {@code MAX_RPC_RETRIES} times until it succeeds.\n     *\n     * @throws IOException if it exceeds {@code MAX_RPC_RETRIES} attempts.\n     */\n    @Override\n    public void deleteDataset(String projectId, String datasetId)\n        throws IOException, InterruptedException {\n      executeWithRetries(\n          client.datasets().delete(projectId, datasetId),\n          String.format(","sourceCodeStart":934,"sourceCodeEnd":970,"githubUrl":"https://github.com/apache/beam/blob/12126d8942aaf848030c478b4c6a28c6af861c66/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryServicesImpl.java#L934-L970","documentation":"Thrown by BigQueryServicesImpl.createDataset after exhausting MAX_RPC_RETRIES attempts to create a BigQuery dataset via the API. Each attempt catches IOException, logs 'Ignore the error and retry', backs off, and retries; when the backoff loop ends the last exception is wrapped in this IOException. It means the dataset could not be created due to a persistent, repeated API failure.","triggerScenarios":"Calling BigQuery IO that requires the destination dataset to exist with dataset auto-creation enabled, while every create-dataset RPC fails (e.g. project not found, permission denied, quota exceeded, repeated 5xx/503 from the BigQuery API) for MAX_RPC_RETRIES consecutive attempts.","commonSituations":"Wrong project ID or dataset ID in the table spec; service account lacking bigquery.datasets.create permission; billing not enabled on the project; transient BigQuery API outages lasting longer than the retry budget.","solutions":["Check the resolved project/dataset IDs and confirm the dataset doesn't already exist with conflicting permissions","Grant the service account roles/bigquery.dataEditor (includes datasets.create) on the project","Verify billing is enabled and BigQuery API is enabled for the project","Inspect the wrapped lastException (getCause()) for the real per-attempt error and address it","Retry later if the underlying API was returning 5xx during an outage"],"exampleFix":"// before\npipeline.apply(BigQueryIO.writeTableRows().to(\"my-project:wrong_dataset.table\")...);\n// after\npipeline.apply(BigQueryIO.writeTableRows().to(\"my-project:correct_dataset.table\")...);\n// and ensure: bq mk --dataset my-project:correct_dataset, or fix IAM","handlingStrategy":"retry","validationCode":"// pre-check with bq CLI or client\nBigQuery bq = options.as(BigQueryOptions.class).getBigQueryServices().getBigQuery(options.as(BigQueryOptions.class));\nDatasetId dsId = DatasetId.of(projectId, datasetId);\nif (bq.getDataset(dsId) == null) {\n  // confirm service account has bigquery.datasets.create and billing is enabled\n}","typeGuard":null,"tryCatchPattern":"try {\n  pipeline.run().waitUntilFinish();\n} catch (IOException e) {\n  if (e.getMessage() != null && e.getMessage().contains(\"Unable to create dataset\")) {\n    Throwable cause = e.getCause(); // real API error\n    // handle permission/quota/project errors\n  }\n  throw e;\n}","preventionTips":["Pre-create datasets with IaTerraform/bq mk instead of relying on auto-creation","Grant roles/bigquery.dataEditor to the runner service account","Verify billing and BigQuery API are enabled in the target project","Log the resolved project/dataset before running"],"tags":["bigquery","gcp","io","retry-exhausted"],"backgroundTag":"api-request-failed","analyzedSha":"12126d8942aaf848030c478b4c6a28c6af861c66","analyzedAt":"2026-09-13T01:50:10.254Z","contentChangedAt":"2026-09-13T01:50:10.254Z","schemaVersion":2},"datasetVersion":"2026-09-14T16:17:12.679Z"}