{"record":{"id":"b0dc9982063856e6","repo":"apache/beam","slug":"exception-while-trying-to-retrieve-schema-of-query","errorCode":null,"errorMessage":"Exception while trying to retrieve schema of query","messagePattern":"Exception while trying to retrieve schema of query","errorType":"exception","errorClass":"BigQuerySchemaRetrievalException","httpStatus":null,"severity":"error","filePath":"sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryQuerySourceDef.java","lineNumber":193,"sourceCode":"        stepUuid, this, bqServices, coder, readerFactory, useAvroLogicalTypes);\n  }\n\n  /** {@inheritDoc} */\n  @Override\n  public TableSchema getTableSchema(BigQueryOptions bqOptions) {\n    try {\n      JobStatistics stats =\n          BigQueryQueryHelper.dryRunQueryIfNeeded(\n              bqServices,\n              bqOptions,\n              dryRunJobStats,\n              query.get(),\n              flattenResults,\n              useLegacySql,\n              location);\n      return stats.getQuery().getSchema();\n    } catch (IOException | InterruptedException | NullPointerException e) {\n      throw new BigQuerySchemaRetrievalException(\n          \"Exception while trying to retrieve schema of query\", e);\n    }\n  }\n\n  ValueProvider<String> getQuery() {\n    return query;\n  }\n}\n","sourceCodeStart":175,"sourceCodeEnd":202,"githubUrl":"https://github.com/apache/beam/blob/12126d8942aaf848030c478b4c6a28c6af861c66/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryQuerySourceDef.java#L175-L202","documentation":"BigQueryIO throws BigQuerySchemaRetrievalException with this message in BigQueryQuerySourceDef.getTableSchema when it cannot determine the schema of a query result. The code calls the BigQuery API (getQueryResults/dry-run style call) and any IOException, InterruptedException, or NullPointerException during schema retrieval is wrapped in this exception.","triggerScenarios":"Resolving the schema of BigQueryIO.readTableRows().fromQuery(...) at pipeline-construction time when the BigQuery API call to fetch query statistics fails, is interrupted, or returns a null schema (e.g. the query returned no schema in stats).","commonSituations":"Transient BigQuery API outages or rate limiting during pipeline translation; query references a nonexistent table so no schema is returned; invalid/empty query ValueProvider; thread interruption during schema lookup; older BigQuery client behaviors yielding unexpected null stats.","solutions":["Log/inspect the cause (getCause()) for the actual BigQuery API error and fix the query or table references","Retry the pipeline translation if the cause was transient (rate limit or outage)","Validate the query is non-empty and runnable (e.g. dry-run it in the console or via the API) before building the pipeline","Ensure the query actually produces a schema (not e.g. a script/DDL with no result schema)","Check for NullPointerException causes indicating an unexpected null from the API and upgrade the Beam/Google client versions"],"exampleFix":"// before\npipeline.apply(BigQueryIO.readTableRows().fromQuery(unvalidatedQuery))\n// after: dry-run/validate first\nvalidateQueryRuns(query, useLegacySql); // throws early with real API error\npipeline.apply(BigQueryIO.readTableRows().fromQuery(query).withQueryPriority(QueryPriority.INTERACTIVE))","handlingStrategy":"try-catch","validationCode":"// Dry-run the query to confirm it yields a schema before pipeline construction\ncom.google.api.services.bigquery.model.Job dryRunJob = bigquery.jobs()\n    .query(projectId, QueryRequest.newBuilder(query).setDryRun(true).setUseLegacySql(false).build())\n    .execute();\nif (dryRunJob.getStatistics().getQuery().getSchema() == null) throw new IllegalStateException(\"Query has no result schema\");","typeGuard":null,"tryCatchPattern":"try {\n  Schema schema = sourceDef.getTableSchema(bigquery); // or pipeline translation that resolves schema\n} catch (BigQuerySchemaRetrievalException e) {\n  Throwable cause = e.getCause();\n  if (cause instanceof IOException) {\n    LOG.warn(\"Transient BigQuery API failure retrieving schema, retrying\", cause); // retry with backoff\n  } else {\n    throw new IllegalArgumentException(\"Query does not produce a valid schema\", cause);\n  }\n}","preventionTips":["Dry-run the query to confirm schema resolution before building the pipeline","Handle InterruptedException properly (restore the interrupt flag) around schema lookups","Pin recent Beam and google-cloud-bigquery client versions to avoid null-stats issues","Validate query strings are non-null and non-empty when supplied via ValueProvider"],"tags":["bigquery","gcp","schema-retrieval"],"backgroundTag":"schema-validation-failed","analyzedSha":"12126d8942aaf848030c478b4c6a28c6af861c66","analyzedAt":"2026-09-13T01:50:10.254Z","contentChangedAt":"2026-09-13T01:50:10.254Z","schemaVersion":2},"datasetVersion":"2026-09-14T16:17:12.679Z"}