apache/beam · error · RuntimeException

Unable to get BigQuery response after retrying %d times for

Error message

Unable to get BigQuery response after retrying %d times for dataset (%s)

What it means

createNewDataset tries to insert a new BigQuery dataset with a bounded retry loop over IOExceptions. If every attempt fails, it throws 'Unable to get BigQuery response after retrying %d times for dataset (%s)' including the last exception. This mirrors the other BigqueryClient retry helpers but for dataset creation.

Source

Thrown at sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/testing/BigqueryClient.java:467

          return;
        } else {
          lastException =
              new IOException(
                  "Expected valid response from insert dataset job, but received null.");
        }
      } catch (GoogleJsonResponseException e) {
        if (e.getStatusCode() == 409) {
          LOG.info("Dataset {} already exists, treating as success.", datasetId);
          return;
        }
        lastException = e;
      } catch (IOException e) {
        // ignore and retry
        lastException = e;
      }
    } while (BackOffUtils.next(sleeper, backoff));

    throw new RuntimeException(
        String.format(
            "Unable to get BigQuery response after retrying %d times for dataset (%s)",
            MAX_QUERY_RETRIES, datasetId),
        lastException);
  }

  public void deleteTable(String projectId, String datasetId, String tableName) {
    try {
      bqClient.tables().delete(projectId, datasetId, tableName).execute();
      LOG.info("Successfully deleted table: {}", tableName);
    } catch (Exception e) {
      LOG.debug("Exception caught when deleting table", e);
    }
  }

  public void deleteDataset(String projectId, String datasetId) {
    try {
      TableList tables = bqClient.tables().list(projectId, datasetId).execute();

View on GitHub (pinned to 12126d8942)

Solutions

Check the wrapped cause: if it's a 409/conflict, delete or use a unique datasetId.
Verify the credential has bigquery.datasets.create permission on the target project.
Confirm network connectivity to bigquery.googleapis.com from the test environment.
Check BigQuery quota usage; stagger or reduce parallel dataset creation.
Retry the run — transient Google-side failures may resolve on a later attempt.

Example fix

// before
String datasetId = projectId + "_it"; // static ID, collides across runs
client.createNewDataset(datasetId);

// after
String datasetId = projectId + "_it_" + System.currentTimeMillis();
client.createNewDataset(datasetId);

Defensive patterns

Strategy: try-catch

Validate before calling

// ensure a unique dataset name and permission beforehand
String datasetId = projectId + "_it_" + System.currentTimeMillis();
// verify caller has bigquery.datasets.create via gcloud:
// gcloud projects get-iam-policy PROJECT_ID --format=json | grep datasets.create

Try / catch

try {
  client.createNewDataset(datasetId);
} catch (RuntimeException e) {
  if (String.valueOf(e.getCause()).contains("409")) {
    LOG.warn("Dataset %s already exists; reusing", datasetId);
  } else {
    throw e;
  }
}

Prevention

Use unique, time-stamped dataset IDs per test run to avoid conflicts.
Grant the test service account bigquery.datasets.create.
Confirm CI network egress before running BigQuery ITs.
Clean up stale test datasets to avoid quota accumulation.

When it happens

Trigger: bigquery.datasets().insert failing with IOException on all attempts while creating a test dataset — persistent network errors, permission failures surfaced as IOException, duplicate-dataset or quota errors that never succeed within the retry budget.

Common situations: Test datasets with the same ID already existing (conflict retried futilely); CI runners without network egress; service account lacking bigquery.datasets.create permission; quota exhaustion during parallel ITs.

Understand the failure class

Background: "API request failed": what wrapped HTTP errors from external APIs mean and how to find the real cause — this error's family across 29 libraries.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/5b342698db0c5072. Report an issue: GitHub.