What causes "Failed to fetch BigQuery data." in apache/beam?

Inspect the wrapped cause for whether it's an IOException (infra/quota/auth) or interruption.

apache/beam · error · RuntimeException

Failed to fetch BigQuery data.

Error message

Failed to fetch BigQuery data.

What it means

BigqueryMatcher.matchesSafely verifies pipeline output by running a BigQuery query and comparing the response. If the query call throws IOException or InterruptedException, it throws 'Failed to fetch BigQuery data.' wrapping the cause (and restores the interrupt flag for InterruptedIOException). This is a Hamcrest matcher used in BigQuery integration tests.

Source

Thrown at sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/testing/BigqueryMatcher.java:128

    LOG.info("Verifying Bigquery data");

    // execute query
    LOG.debug("Executing query: {}", tableAndQuery.getQuery());
    try {
      if (tableAndQuery.getUsingStandardSql()) {
        response =
            bigqueryClient.queryWithRetriesUsingStandardSql(
                tableAndQuery.getQuery(), tableAndQuery.getProjectId());
      } else {
        response =
            bigqueryClient.queryWithRetries(tableAndQuery.getQuery(), tableAndQuery.getProjectId());
      }
    } catch (IOException | InterruptedException e) {
      if (e instanceof InterruptedIOException) {
        Thread.currentThread().interrupt();
      }
      throw new RuntimeException("Failed to fetch BigQuery data.", e);
    }

    if (!response.getJobComplete()) {
      // query job not complete, verification failed
      return false;
    } else {
      // compute checksum
      actualChecksum = generateHash(response.getRows());
      LOG.debug("Generated a SHA1 checksum based on queried data: {}", actualChecksum);

      return expectedChecksum.equals(actualChecksum);
    }
  }

  private void validateArgument(String name, String value) {
    checkArgument(!Strings.isNullOrEmpty(value), "Expected valid %s, but was %s", name, value);
  }

View on GitHub (pinned to 12126d8942)

Solutions

Inspect the wrapped cause for whether it's an IOException (infra/quota/auth) or interruption.
Verify ADC/credentials are valid for the target project (see credential setup for BigqueryClient).
Check that the pipeline actually wrote the expected rows before matching; add waits/retries upstream.
Confirm network egress to bigquery.googleapis.com from the test machine.
If interrupted, avoid swallowing interrupts — ensure the test framework isn't cancelling threads.

Example fix

// before
assertThat(job, matchesBigQueryOutput(tableAndQuery)); // throws 'Failed to fetch BigQuery data.'

// after
try {
  assertThat(job, matchesBigQueryOutput(tableAndQuery));
} catch (RuntimeException e) {
  LOG.warn("BigQuery verification failed; re-checking after write lag", e);
  Thread.sleep(30_000);
  assertThat(job, matchesBigQueryOutput(tableAndQuery));
}

Defensive patterns

Strategy: try-catch

Validate before calling

// pre-check credentials and connectivity before running the matcher
GoogleCredentials.getApplicationDefault(); // throws fast if ADC missing
boolean ok = Runtime.getRuntime().exec(new String[]{"curl", "-sSfo", "/dev/null",
    "https://bigquery.googleapis.com/"}).waitFor() == 0;

Try / catch

try {
  assertThat(job, matchesBigQueryOutput(tableAndQuery));
} catch (RuntimeException e) {
  if (e.getCause() instanceof InterruptedIOException
      || e.getCause() instanceof InterruptedException) {
    Thread.currentThread().interrupt();
  }
  throw e;
}

Prevention

Ensure the pipeline completed writing rows before evaluating the matcher.
Verify ADC/credentials in the test JVM before IT suites.
Check quotas and network access during assertion phases.
Never swallow InterruptedException — restore the interrupt flag.

When it happens

Trigger: bigqueryClient.queryWithRetries(...) throwing IOException (network/auth/quota failure surviving retries) or InterruptedException while BigqueryMatcher waits for the pipeline output to land in BigQuery; also test-thread interruption during matcher evaluation.

Common situations: Pipeline not yet written results but retries exhausted due to slow/failed queries; test runner cancelling threads mid-wait; credentials/network problems inside IT JVM; quota exhaustion during assertion phase.

Understand the failure class

Background: Database query failed: Internal Server Error 500s wrapping SQL, Prisma, and connection failures — what to check first — this error's family across 16 libraries.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/cbe49dea1cd6376a. Report an issue: GitHub.