apache/beam · error · InvalidConfigurationException

Configuration schema provided does not match expected

Error message

Configuration schema provided does not match expected

What it means

PubsubSchemaIOProvider.validateConfigurationSchema() checks that the configuration Row passed to from() has exactly the schema returned by configurationSchema(). Any mismatch in field names, types, or order throws an InvalidConfigurationException.

Solutions

Build the configuration Row using the provider's own configurationSchema() (or the SchemaTransform providers' configuration Row builders) instead of hand-rolling a Row.
Verify the Row's schema equals PubsubSchemaIOProvider.configurationSchema() before calling from().
Upgrade/refresh code that hard-codes the config Row to match the SDK version's expected fields.

Example fix

// before
Row config = Row.withSchema(myAdHocSchema).addValues(topic).build();
// after
Row config = Row.withSchema(new PubsubSchemaIOProvider().configurationSchema()).addValues(topic, dlq, ...).build();

Defensive patterns

Strategy: validation

Validate before calling

if (!configRow.getSchema().equals(new PubsubSchemaIOProvider().configurationSchema())) { throw new IllegalArgumentException("Configuration Row schema does not match provider's configurationSchema()"); }

Try / catch

try { ioProvider.from(configRow); } catch (InvalidConfigurationException e) { if (e.getMessage().contains("does not match expected")) { /* rebuild Row using configurationSchema() */ } throw e; }

Prevention

Never hand-build the configuration Row; construct it from the provider's configurationSchema().
Re-check config Row construction after Beam upgrades, since the expected schema may change.
Add an equality check against configurationSchema() in pipeline setup tests.

When it happens

Trigger: Calling PubsubSchemaIOProvider.from(configRow) with a hand-built or stale Row whose schema differs from the provider's expected configurationSchema() — e.g. after upgrading Beam where the config schema gained/renamed fields.

Common situations: Manually constructing the configuration Row instead of using the provider's schema/builders; cached config Rows built against an older Beam version; fields added or reordered in a newer SDK release.

Understand the failure class

Background: Schema validation failed / invalid input schema: payload rejected because its shape doesn't match the expected schema — this error's family across 28 libraries.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/77da82eb5af6083a. Report an issue: GitHub.

Appendix: source

Thrown at sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsub/PubsubSchemaIOProvider.java:178

              + "CREATE TABLE for Pubsub topic must not be null");
    }
    if (!PubsubSchemaIO.fieldPresent(schema, TIMESTAMP_FIELD, FieldType.DATETIME)) {
      throw new InvalidSchemaException(
          "Unsupported schema specified for Pubsub source in CREATE TABLE."
              + "CREATE TABLE for Pubsub topic must include at least 'event_timestamp' field of "
              + "type 'TIMESTAMP'");
    }
  }

  private void validateDlq(@Nullable String deadLetterQueue) {
    if (deadLetterQueue != null && deadLetterQueue.isEmpty()) {
      throw new InvalidConfigurationException("Dead letter queue topic name is not specified");
    }
  }

  private void validateConfigurationSchema(Row configuration) {
    if (!configuration.getSchema().equals(configurationSchema())) {
      throw new InvalidConfigurationException(
          "Configuration schema provided does not match expected");
    }
  }

  /** An abstraction to create schema aware IOs. */
  private static class PubsubSchemaIO implements SchemaIO, Serializable {
    protected final Schema dataSchema;
    protected final String location;
    protected final boolean useFlatSchema;
    protected final Config config;

    private PubsubSchemaIO(String location, Row config, Schema dataSchema) {
      this.dataSchema = dataSchema;
      this.location = location;
      this.useFlatSchema = !shouldUseNestedSchema(dataSchema);
      this.config =
          new AutoValueSchema().fromRowFunction(TypeDescriptor.of(Config.class)).apply(config);
    }

View on GitHub (pinned to 12126d8942)