apache/beam · error · RuntimeException

Unable to get BigQuery response after retrying %d times for

Error message

Unable to get BigQuery response after retrying %d times for table (%s)

What it means

createNewTable inserts a BigQuery table with bounded retries over IOExceptions. When all attempts fail, it throws 'Unable to get BigQuery response after retrying %d times for table (%s)' with newTable.getId() in the message and the last exception attached.

Source

Thrown at sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/testing/BigqueryClient.java:535

              new IOException("Expected valid response from create table job, but received null.");
        }
      } catch (GoogleJsonResponseException e) {
        if (e.getStatusCode() == 409) {
          LOG.info(
              "Table {}:{}.{} already exists, treating as success.",
              projectId,
              datasetId,
              newTable.getTableReference().getTableId());
          return;
        }
        lastException = e;
      } catch (IOException e) {
        // ignore and retry
        lastException = e;
      }
    } while (BackOffUtils.next(sleeper, backoff));

    throw new RuntimeException(
        String.format(
            "Unable to get BigQuery response after retrying %d times for table (%s)",
            MAX_QUERY_RETRIES, newTable.getId()),
        lastException);
  }

  /** Inserts rows to a table using a BigQuery streaming write. */
  public void insertDataToTable(
      String projectId, String datasetId, String tableName, List<Map<String, Object>> rows)
      throws IOException, InterruptedException {
    Sleeper sleeper = Sleeper.DEFAULT;
    BackOff backoff = BackOffAdapter.toGcpBackOff(BACKOFF_FACTORY.backoff());
    IOException lastException = null;
    do {
      if (lastException != null) {
        LOG.warn("Retrying insert table ({}) after exception", tableName, lastException);
      }
      try {

View on GitHub (pinned to 12126d8942)

Solutions

Read the wrapped lastException for the true cause (403 vs 400 vs network).
Validate the table schema/definition is accepted by BigQuery (test a manual insert via bq CLI).
Confirm the credential can create tables in the target dataset.
Check network/proxy access to bigquery.googleapis.com.
Check quotas and reduce concurrent table creation in test harnesses.

Example fix

// before
client.createNewTable(datasetId, newTable); // throws after retries

// after
try {
  client.createNewTable(datasetId, newTable);
} catch (RuntimeException e) {
  throw new RuntimeException("Table " + newTable.getId() + " creation failed; check schema/permissions", e);
}

Defensive patterns

Strategy: validation

Validate before calling

// validate schema JSON is parseable and non-empty before insert
boolean schemaLooksValid(com.google.api.services.bigquery.model.Table t) {
  return t != null && t.getSchema() != null
      && t.getSchema().getFields() != null && !t.getSchema().getFields().isEmpty();
}

Try / catch

try {
  client.createNewTable(datasetId, newTable);
} catch (RuntimeException e) {
  LOG.error("Table creation failed for %s; cause:", newTable.getId(), e.getCause());
  throw e;
}

Prevention

Validate table schema fields before calling createNewTable.
Ensure the credential has bigquery.tables.create in the dataset.
Test the same schema manually with the bq CLI first.
Avoid hammering the same project with parallel table creation.

When it happens

Trigger: bigquery.tables().insert throwing IOException on every attempt while creating the test table — persistent network errors, invalid table schema never accepted, missing permissions, or quota limits exceeded throughout the retry loop.

Common situations: Schema JSON in test fixtures rejected server-side repeatedly; CI without egress to Google APIs; service account lacking bigquery.tables.create; parallel ITs hammering the same project's quota.

Understand the failure class

Background: "API request failed": what wrapped HTTP errors from external APIs mean and how to find the real cause — this error's family across 29 libraries.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/b201b8df9908a39c. Report an issue: GitHub.