apache/beam · error · BigQuerySchemaRetrievalException

Exception while trying to retrieve schema of query

Error message

Exception while trying to retrieve schema of query

What it means

BigQueryIO throws BigQuerySchemaRetrievalException with this message in BigQueryQuerySourceDef.getTableSchema when it cannot determine the schema of a query result. The code calls the BigQuery API (getQueryResults/dry-run style call) and any IOException, InterruptedException, or NullPointerException during schema retrieval is wrapped in this exception.

Source

Thrown at sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryQuerySourceDef.java:193

        stepUuid, this, bqServices, coder, readerFactory, useAvroLogicalTypes);
  }

  /** {@inheritDoc} */
  @Override
  public TableSchema getTableSchema(BigQueryOptions bqOptions) {
    try {
      JobStatistics stats =
          BigQueryQueryHelper.dryRunQueryIfNeeded(
              bqServices,
              bqOptions,
              dryRunJobStats,
              query.get(),
              flattenResults,
              useLegacySql,
              location);
      return stats.getQuery().getSchema();
    } catch (IOException | InterruptedException | NullPointerException e) {
      throw new BigQuerySchemaRetrievalException(
          "Exception while trying to retrieve schema of query", e);
    }
  }

  ValueProvider<String> getQuery() {
    return query;
  }
}

View on GitHub (pinned to 12126d8942)

Solutions

Log/inspect the cause (getCause()) for the actual BigQuery API error and fix the query or table references
Retry the pipeline translation if the cause was transient (rate limit or outage)
Validate the query is non-empty and runnable (e.g. dry-run it in the console or via the API) before building the pipeline
Ensure the query actually produces a schema (not e.g. a script/DDL with no result schema)
Check for NullPointerException causes indicating an unexpected null from the API and upgrade the Beam/Google client versions

Example fix

// before
pipeline.apply(BigQueryIO.readTableRows().fromQuery(unvalidatedQuery))
// after: dry-run/validate first
validateQueryRuns(query, useLegacySql); // throws early with real API error
pipeline.apply(BigQueryIO.readTableRows().fromQuery(query).withQueryPriority(QueryPriority.INTERACTIVE))

Defensive patterns

Strategy: try-catch

Validate before calling

// Dry-run the query to confirm it yields a schema before pipeline construction
com.google.api.services.bigquery.model.Job dryRunJob = bigquery.jobs()
    .query(projectId, QueryRequest.newBuilder(query).setDryRun(true).setUseLegacySql(false).build())
    .execute();
if (dryRunJob.getStatistics().getQuery().getSchema() == null) throw new IllegalStateException("Query has no result schema");

Try / catch

try {
  Schema schema = sourceDef.getTableSchema(bigquery); // or pipeline translation that resolves schema
} catch (BigQuerySchemaRetrievalException e) {
  Throwable cause = e.getCause();
  if (cause instanceof IOException) {
    LOG.warn("Transient BigQuery API failure retrieving schema, retrying", cause); // retry with backoff
  } else {
    throw new IllegalArgumentException("Query does not produce a valid schema", cause);
  }
}

Prevention

Dry-run the query to confirm schema resolution before building the pipeline
Handle InterruptedException properly (restore the interrupt flag) around schema lookups
Pin recent Beam and google-cloud-bigquery client versions to avoid null-stats issues
Validate query strings are non-null and non-empty when supplied via ValueProvider

When it happens

Trigger: Resolving the schema of BigQueryIO.readTableRows().fromQuery(...) at pipeline-construction time when the BigQuery API call to fetch query statistics fails, is interrupted, or returns a null schema (e.g. the query returned no schema in stats).

Common situations: Transient BigQuery API outages or rate limiting during pipeline translation; query references a nonexistent table so no schema is returned; invalid/empty query ValueProvider; thread interruption during schema lookup; older BigQuery client behaviors yielding unexpected null stats.

Understand the failure class

Background: Schema validation failed / invalid input schema: payload rejected because its shape doesn't match the expected schema — this error's family across 28 libraries.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/b0dc9982063856e6. Report an issue: GitHub.