What causes "Exception while trying to retrieve schema" in apache/beam?

Inspect the cause chain of SpannerSchemaRetrievalException for the underlying SpannerException.

apache/beam · error · SpannerSchemaRetrievalException

Exception while trying to retrieve schema

Error message

Exception while trying to retrieve schema

What it means

SpannerQuerySourceDef.getBeamSchema() analyzes the query with QueryAnalyzeMode.PLAN to derive a Beam schema; any exception during that round trip is wrapped in SpannerSchemaRetrievalException("Exception while trying to retrieve schema", e). The cause (Spanner API error, bad SQL, permissions) is attached as the cause.

Source

Thrown at sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/spanner/SpannerQuerySourceDef.java:52

  }

  private SpannerQuerySourceDef(SpannerConfig config, Statement query) {
    this.config = config;
    this.query = query;
  }

  /** {@inheritDoc} */
  @Override
  public Schema getBeamSchema() {
    Schema beamSchema;
    try (SpannerAccessor spannerAccessor = SpannerAccessor.getOrCreate(config)) {
      try (ReadContext readContext = spannerAccessor.getDatabaseClient().singleUse()) {
        ResultSet result = readContext.analyzeQuery(query, ReadContext.QueryAnalyzeMode.PLAN);
        result.next();
        beamSchema = structTypeToBeamRowSchema(result.getMetadata().getRowType(), true);
      }
    } catch (Exception e) {
      throw new SpannerSchemaRetrievalException("Exception while trying to retrieve schema", e);
    }
    return beamSchema;
  }
}

View on GitHub (pinned to 12126d8942)

Solutions

Inspect the cause chain of SpannerSchemaRetrievalException for the underlying SpannerException.
Run the SQL directly in the Spanner console / gcloud to reproduce the planning error.
Verify IAM permissions (spanner.databases.read, and read metadata) for the credentials in SpannerConfig.
Check network/connectivity and Spanner service status; retry on transient errors.

Example fix

// before
PCollection<Row> rows = p.apply(SqlTransform.query("SELEC * FROM users"));
// after
PCollection<Row> rows = p.apply(SqlTransform.query("SELECT * FROM users"));

Defensive patterns

Strategy: try-catch

Validate before calling

// Pre-flight: validate SQL and permissions by running a cheap PLAN analysis yourself
try (ReadContext ctx = dbClient.singleUse()) { ctx.analyzeQuery(sql, ReadContext.QueryAnalyzeMode.PLAN); }

Try / catch

try { rows = p.apply(SqlTransform.query(sql)); } catch (SpannerSchemaRetrievalException e) { LOG.error("Schema retrieval failed", e.getCause()); throw new RuntimeException("Fix SQL/permissions for: " + sql, e); }

Prevention

Test the query in the Spanner console before wiring it into Beam SQL.
Grant spanner.databases.read to the pipeline's service account.
Always inspect getCause() of SpannerSchemaRetrievalException.

When it happens

Trigger: Calling getBeamSchema() (e.g., during SqlTransform resolution on a Spanner query source) when the single-use ReadContext analyzeQuery(PLAN) call fails: invalid SQL, missing Spanner permissions, network/API errors, or an empty/invalid query result metadata.

Common situations: SQL syntax/typing errors that break query planning; service account lacking spanner.databasees.read; transient Spanner unavailability; querying a database that no longer exists; Beam SQL join against a Spanner table whose schema can't be inferred.

Understand the failure class

Background: Schema validation failed / invalid input schema: payload rejected because its shape doesn't match the expected schema — this error's family across 28 libraries.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/a425bbcfa00adfc1. Report an issue: GitHub.