{"record":{"id":"5b342698db0c5072","repo":"apache/beam","slug":"unable-to-get-bigquery-response-after-retrying-d-times-for","errorCode":null,"errorMessage":"Unable to get BigQuery response after retrying %d times for dataset (%s)","messagePattern":"Unable to get BigQuery response after retrying (.+?) times for dataset \\((.+?)\\)","errorType":"exception","errorClass":"RuntimeException","httpStatus":null,"severity":"error","filePath":"sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/testing/BigqueryClient.java","lineNumber":467,"sourceCode":"          return;\n        } else {\n          lastException =\n              new IOException(\n                  \"Expected valid response from insert dataset job, but received null.\");\n        }\n      } catch (GoogleJsonResponseException e) {\n        if (e.getStatusCode() == 409) {\n          LOG.info(\"Dataset {} already exists, treating as success.\", datasetId);\n          return;\n        }\n        lastException = e;\n      } catch (IOException e) {\n        // ignore and retry\n        lastException = e;\n      }\n    } while (BackOffUtils.next(sleeper, backoff));\n\n    throw new RuntimeException(\n        String.format(\n            \"Unable to get BigQuery response after retrying %d times for dataset (%s)\",\n            MAX_QUERY_RETRIES, datasetId),\n        lastException);\n  }\n\n  public void deleteTable(String projectId, String datasetId, String tableName) {\n    try {\n      bqClient.tables().delete(projectId, datasetId, tableName).execute();\n      LOG.info(\"Successfully deleted table: {}\", tableName);\n    } catch (Exception e) {\n      LOG.debug(\"Exception caught when deleting table\", e);\n    }\n  }\n\n  public void deleteDataset(String projectId, String datasetId) {\n    try {\n      TableList tables = bqClient.tables().list(projectId, datasetId).execute();","sourceCodeStart":449,"sourceCodeEnd":485,"githubUrl":"https://github.com/apache/beam/blob/12126d8942aaf848030c478b4c6a28c6af861c66/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/testing/BigqueryClient.java#L449-L485","documentation":"createNewDataset tries to insert a new BigQuery dataset with a bounded retry loop over IOExceptions. If every attempt fails, it throws 'Unable to get BigQuery response after retrying %d times for dataset (%s)' including the last exception. This mirrors the other BigqueryClient retry helpers but for dataset creation.","triggerScenarios":"bigquery.datasets().insert failing with IOException on all attempts while creating a test dataset — persistent network errors, permission failures surfaced as IOException, duplicate-dataset or quota errors that never succeed within the retry budget.","commonSituations":"Test datasets with the same ID already existing (conflict retried futilely); CI runners without network egress; service account lacking bigquery.datasets.create permission; quota exhaustion during parallel ITs.","solutions":["Check the wrapped cause: if it's a 409/conflict, delete or use a unique datasetId.","Verify the credential has bigquery.datasets.create permission on the target project.","Confirm network connectivity to bigquery.googleapis.com from the test environment.","Check BigQuery quota usage; stagger or reduce parallel dataset creation.","Retry the run — transient Google-side failures may resolve on a later attempt."],"exampleFix":"// before\nString datasetId = projectId + \"_it\"; // static ID, collides across runs\nclient.createNewDataset(datasetId);\n\n// after\nString datasetId = projectId + \"_it_\" + System.currentTimeMillis();\nclient.createNewDataset(datasetId);","handlingStrategy":"try-catch","validationCode":"// ensure a unique dataset name and permission beforehand\nString datasetId = projectId + \"_it_\" + System.currentTimeMillis();\n// verify caller has bigquery.datasets.create via gcloud:\n// gcloud projects get-iam-policy PROJECT_ID --format=json | grep datasets.create","typeGuard":null,"tryCatchPattern":"try {\n  client.createNewDataset(datasetId);\n} catch (RuntimeException e) {\n  if (String.valueOf(e.getCause()).contains(\"409\")) {\n    LOG.warn(\"Dataset %s already exists; reusing\", datasetId);\n  } else {\n    throw e;\n  }\n}","preventionTips":["Use unique, time-stamped dataset IDs per test run to avoid conflicts.","Grant the test service account bigquery.datasets.create.","Confirm CI network egress before running BigQuery ITs.","Clean up stale test datasets to avoid quota accumulation."],"tags":["java","gcp","bigquery","retry","dataset"],"backgroundTag":"api-request-failed","analyzedSha":"12126d8942aaf848030c478b4c6a28c6af861c66","analyzedAt":"2026-09-13T01:50:10.254Z","contentChangedAt":"2026-09-13T01:50:10.254Z","schemaVersion":2},"datasetVersion":"2026-09-14T16:17:12.679Z"}