{"record":{"id":"3b45dd559675e5e6","repo":"apache/beam","slug":"unable-to-get-bigquery-response-after-retrying-d-times-using","errorCode":null,"errorMessage":"Unable to get BigQuery response after retrying %d times using query (%s)","messagePattern":"Unable to get BigQuery response after retrying (.+?) times using query \\((.+?)\\)","errorType":"exception","errorClass":"RuntimeException","httpStatus":null,"severity":"error","filePath":"sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/testing/BigqueryClient.java","lineNumber":400,"sourceCode":"    do {\n      if (lastException != null) {\n        LOG.warn(\"Retrying query ({}) after exception\", bqQueryRequest.getQuery(), lastException);\n      }\n      try {\n        QueryResponse response = bqClient.jobs().query(projectId, bqQueryRequest).execute();\n        if (response != null) {\n          return typed ? getTypedTableRows(response) : response;\n        } else {\n          lastException =\n              new IOException(\"Expected valid response from query job, but received null.\");\n        }\n      } catch (IOException e) {\n        // ignore and retry\n        lastException = e;\n      }\n    } while (BackOffUtils.next(sleeper, backoff));\n\n    throw new RuntimeException(\n        String.format(\n            \"Unable to get BigQuery response after retrying %d times using query (%s)\",\n            MAX_QUERY_RETRIES, bqQueryRequest.getQuery()),\n        lastException);\n  }\n\n  /** Creates a new dataset. */\n  public void createNewDataset(String projectId, String datasetId)\n      throws IOException, InterruptedException {\n    createNewDataset(projectId, datasetId, null);\n  }\n\n  /** Creates a new dataset with defaultTableExpirationMs. */\n  public void createNewDataset(\n      String projectId, String datasetId, @Nullable Long defaultTableExpirationMs)\n      throws IOException, InterruptedException {\n    createNewDataset(projectId, datasetId, defaultTableExpirationMs, null);\n  }","sourceCodeStart":382,"sourceCodeEnd":418,"githubUrl":"https://github.com/apache/beam/blob/12126d8942aaf848030c478b4c6a28c6af861c66/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/testing/BigqueryClient.java#L382-L418","documentation":"queryWithRetries executes a BigQuery query with a bounded retry loop (MAX_QUERY_RETRIES) over transient IOExceptions. If all retries are exhausted, it throws a RuntimeException 'Unable to get BigQuery response after retrying %d times using query (%s)' with the last exception attached.","triggerScenarios":"bigquery.jobs().query() throwing IOException on every attempt up to MAX_QUERY_RETRIES while running queryWithRetries/queryWithRetriesUsingStandardSql — e.g. persistent network failure, quota/rate errors surfaced as IOException, or an invalid query that always fails.","commonSituations":"CI machines with flaky or blocked network access to bigquery.googleapis.com; exhausted BigQuery API quota; transient Google outages; project-level rate limits during parallel IT runs.","solutions":["Inspect the wrapped lastException in the stack trace for the root cause (quota vs network vs auth).","Verify network/proxy access to https://bigquery.googleapis.com from the test host.","Check BigQuery quotas/rate limits in Cloud Console; reduce test parallelism if rate-limited.","Re-run the tests — these are transient-retry loops, so intermittent infra issues often clear.","Increase MAX_QUERY_RETRIES or backoff for very large/slow test datasets if appropriate."],"exampleFix":"// before\nclient.queryWithRetries(query, projectId); // throws after MAX_QUERY_RETRIES\n\n// after\ntry {\n  client.queryWithRetries(query, projectId);\n} catch (RuntimeException e) {\n  LOG.warn(\"BigQuery query failed after retries; retrying once after backoff\", e);\n  Thread.sleep(10_000);\n  client.queryWithRetries(query, projectId);\n}","handlingStrategy":"retry","validationCode":"// pre-flight connectivity check\nProcess p = Runtime.getRuntime().exec(new String[]{\"curl\", \"-sSfo\", \"/dev/null\",\n    \"https://bigquery.googleapis.com/\"});\nboolean reachable = p.waitFor() == 0;","typeGuard":null,"tryCatchPattern":"try {\n  client.queryWithRetries(query, projectId);\n} catch (RuntimeException e) {\n  if (e.getCause() instanceof IOException) {\n    LOG.error(\"BigQuery unreachable or quota-exhausted; root cause:\", e.getCause());\n  }\n  throw e;\n}","preventionTips":["Check BigQuery quotas and rate limits before large parallel test runs.","Verify network egress to bigquery.googleapis.com in CI sandboxes.","Inspect the wrapped cause to classify quota vs auth vs network issues.","Throttle test concurrency that issues many simultaneous queries."],"tags":["java","gcp","bigquery","retry","network"],"backgroundTag":"api-request-failed","analyzedSha":"12126d8942aaf848030c478b4c6a28c6af861c66","analyzedAt":"2026-09-13T01:50:10.254Z","contentChangedAt":"2026-09-13T01:50:10.254Z","schemaVersion":2},"datasetVersion":"2026-09-14T16:17:12.679Z"}