{"record":{"id":"cbe49dea1cd6376a","repo":"apache/beam","slug":"failed-to-fetch-bigquery-data","errorCode":null,"errorMessage":"Failed to fetch BigQuery data.","messagePattern":"Failed to fetch BigQuery data\\.","errorType":"exception","errorClass":"RuntimeException","httpStatus":null,"severity":"error","filePath":"sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/testing/BigqueryMatcher.java","lineNumber":128,"sourceCode":"\n    LOG.info(\"Verifying Bigquery data\");\n\n    // execute query\n    LOG.debug(\"Executing query: {}\", tableAndQuery.getQuery());\n    try {\n      if (tableAndQuery.getUsingStandardSql()) {\n        response =\n            bigqueryClient.queryWithRetriesUsingStandardSql(\n                tableAndQuery.getQuery(), tableAndQuery.getProjectId());\n      } else {\n        response =\n            bigqueryClient.queryWithRetries(tableAndQuery.getQuery(), tableAndQuery.getProjectId());\n      }\n    } catch (IOException | InterruptedException e) {\n      if (e instanceof InterruptedIOException) {\n        Thread.currentThread().interrupt();\n      }\n      throw new RuntimeException(\"Failed to fetch BigQuery data.\", e);\n    }\n\n    if (!response.getJobComplete()) {\n      // query job not complete, verification failed\n      return false;\n    } else {\n      // compute checksum\n      actualChecksum = generateHash(response.getRows());\n      LOG.debug(\"Generated a SHA1 checksum based on queried data: {}\", actualChecksum);\n\n      return expectedChecksum.equals(actualChecksum);\n    }\n  }\n\n  private void validateArgument(String name, String value) {\n    checkArgument(!Strings.isNullOrEmpty(value), \"Expected valid %s, but was %s\", name, value);\n  }\n","sourceCodeStart":110,"sourceCodeEnd":146,"githubUrl":"https://github.com/apache/beam/blob/12126d8942aaf848030c478b4c6a28c6af861c66/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/testing/BigqueryMatcher.java#L110-L146","documentation":"BigqueryMatcher.matchesSafely verifies pipeline output by running a BigQuery query and comparing the response. If the query call throws IOException or InterruptedException, it throws 'Failed to fetch BigQuery data.' wrapping the cause (and restores the interrupt flag for InterruptedIOException). This is a Hamcrest matcher used in BigQuery integration tests.","triggerScenarios":"bigqueryClient.queryWithRetries(...) throwing IOException (network/auth/quota failure surviving retries) or InterruptedException while BigqueryMatcher waits for the pipeline output to land in BigQuery; also test-thread interruption during matcher evaluation.","commonSituations":"Pipeline not yet written results but retries exhausted due to slow/failed queries; test runner cancelling threads mid-wait; credentials/network problems inside IT JVM; quota exhaustion during assertion phase.","solutions":["Inspect the wrapped cause for whether it's an IOException (infra/quota/auth) or interruption.","Verify ADC/credentials are valid for the target project (see credential setup for BigqueryClient).","Check that the pipeline actually wrote the expected rows before matching; add waits/retries upstream.","Confirm network egress to bigquery.googleapis.com from the test machine.","If interrupted, avoid swallowing interrupts — ensure the test framework isn't cancelling threads."],"exampleFix":"// before\nassertThat(job, matchesBigQueryOutput(tableAndQuery)); // throws 'Failed to fetch BigQuery data.'\n\n// after\ntry {\n  assertThat(job, matchesBigQueryOutput(tableAndQuery));\n} catch (RuntimeException e) {\n  LOG.warn(\"BigQuery verification failed; re-checking after write lag\", e);\n  Thread.sleep(30_000);\n  assertThat(job, matchesBigQueryOutput(tableAndQuery));\n}","handlingStrategy":"try-catch","validationCode":"// pre-check credentials and connectivity before running the matcher\nGoogleCredentials.getApplicationDefault(); // throws fast if ADC missing\nboolean ok = Runtime.getRuntime().exec(new String[]{\"curl\", \"-sSfo\", \"/dev/null\",\n    \"https://bigquery.googleapis.com/\"}).waitFor() == 0;","typeGuard":null,"tryCatchPattern":"try {\n  assertThat(job, matchesBigQueryOutput(tableAndQuery));\n} catch (RuntimeException e) {\n  if (e.getCause() instanceof InterruptedIOException\n      || e.getCause() instanceof InterruptedException) {\n    Thread.currentThread().interrupt();\n  }\n  throw e;\n}","preventionTips":["Ensure the pipeline completed writing rows before evaluating the matcher.","Verify ADC/credentials in the test JVM before IT suites.","Check quotas and network access during assertion phases.","Never swallow InterruptedException — restore the interrupt flag."],"tags":["java","gcp","bigquery","testing","matcher"],"backgroundTag":"database-query-failed","analyzedSha":"12126d8942aaf848030c478b4c6a28c6af861c66","analyzedAt":"2026-09-13T01:50:10.254Z","contentChangedAt":"2026-09-13T01:50:10.254Z","schemaVersion":2},"datasetVersion":"2026-09-14T16:17:12.679Z"}