apache/beam · error · BigQuerySchemaRetrievalException

Exception while trying to retrieve schema

Error message

Exception while trying to retrieve schema

What it means

BigQueryTableSourceDef.getTableSchema retrieves the table and returns its schema. Any exception during retrieval (network, permission, missing table, null table failing checkStateNotNull) is wrapped in a BigQuerySchemaRetrievalException with message 'Exception while trying to retrieve schema' and the original cause attached. The real cause is always in getCause().

Source

Thrown at sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryTableSourceDef.java:112

      String stepUuid,
      Coder<T> coder,
      SerializableFunction<TableSchema, AvroSource.DatumReaderFactory<T>> readerFactory,
      boolean useAvroLogicalTypes) {
    return BigQueryTableSource.create(
        stepUuid, this, bqServices, coder, readerFactory, useAvroLogicalTypes);
  }

  /** {@inheritDoc} */
  @Override
  public TableSchema getTableSchema(BigQueryOptions bqOptions) {
    try {
      try (DatasetService datasetService = bqServices.getDatasetService(bqOptions)) {
        TableReference tableRef = getTableReference(bqOptions);
        Table table = datasetService.getTable(tableRef);
        return Preconditions.checkStateNotNull(table).getSchema();
      }
    } catch (Exception e) {
      throw new BigQuerySchemaRetrievalException("Exception while trying to retrieve schema", e);
    }
  }
}

View on GitHub (pinned to 12126d8942)

Solutions

1. Inspect the cause chain (e.getCause()) to see the underlying BigQuery error and fix that specifically
2. Verify table existence and access with `bq show --schema project:dataset.table` using pipeline credentials
3. Add retry with backoff around schema retrieval for transient BigQuery errors
4. Grant roles/bigquery.metadataViewer to the runner service account on the dataset

Example fix

// before
Schema s = tableSourceDef.getTableSchema(); // wraps any failure
// after
try {
  Schema s = tableSourceDef.getTableSchema();
} catch (BigQuerySchemaRetrievalException e) {
  Throwable cause = e.getCause(); // e.g. permission-denied / not-found
  throw new IllegalStateException("Fix BQ schema access: " + cause.getMessage(), e);
}

// Pre-check retrievability before asking the source for its schema:
BigQuery bq = BigQueryOptions.getDefaultInstance().getService();
Table t = bq.getTable("proj", "ds", "tbl");
if (t == null || t.getSchema() == null) throw new IllegalStateException("Schema not retrievable");

try {
  Schema schema = tableSourceDef.getTableSchema();
} catch (BigQuerySchemaRetrievalException e) {
  // always inspect the wrapped cause for the real BigQuery error
  throw new IllegalStateException("Schema retrieval failed: " + e.getCause(), e);
}

When it happens

Trigger: Calling getTableSchema (directly or via schema-aware BigQueryIO reads) when the DatasetService.getTable call fails: table missing, permissions denied, transient BigQuery API errors, or a null table tripping checkStateNotNull.

Common situations: Service account lacks bigquery.tables.get on the dataset; table dropped/renamed between pipeline validation and execution; transient 429/5xx from BigQuery; cross-project access without proper ACLs.

Understand the failure class

Background: Schema validation failed / invalid input schema: payload rejected because its shape doesn't match the expected schema — this error's family across 28 libraries.

Related errors

• We have observed a row of size %s bytes exceeding the BigQue
• Failed to patch table schema.
• Schema must be set for table " + tableReference + " when wri
• Table %s:%s.%s requires a schema. None can be inferred becau
• Reserved field name <field.name()> in user schema.

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/a722928072b809f7. Report an issue: GitHub.