apache/beam · error · InvalidSchemaException

Unsupported schema specified for Pubsub source in CREATE TAB

Error message

Unsupported schema specified for Pubsub source in CREATE TABLE.CREATE TABLE for Pubsub topic must include at least 'event_timestamp' field of type 'TIMESTAMP'

What it means

PubsubSchemaIOProvider.validateDataSchema() requires the CREATE TABLE schema for a Pubsub source to contain an 'event_timestamp' field of type TIMESTAMP/DATETIME. Without it, messages cannot be assigned event timestamps, so an InvalidSchemaException is thrown.

Source

Thrown at sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsub/PubsubSchemaIOProvider.java:163

  @Override
  public boolean requiresDataSchema() {
    return true;
  }

  @Override
  public PCollection.IsBounded isBounded() {
    return PCollection.IsBounded.UNBOUNDED;
  }

  private void validateDataSchema(@Nullable Schema schema) {
    if (schema == null) {
      throw new InvalidSchemaException(
          "Unsupported schema specified for Pubsub source in CREATE TABLE."
              + "CREATE TABLE for Pubsub topic must not be null");
    }
    if (!PubsubSchemaIO.fieldPresent(schema, TIMESTAMP_FIELD, FieldType.DATETIME)) {
      throw new InvalidSchemaException(
          "Unsupported schema specified for Pubsub source in CREATE TABLE."
              + "CREATE TABLE for Pubsub topic must include at least 'event_timestamp' field of "
              + "type 'TIMESTAMP'");
    }
  }

  private void validateDlq(@Nullable String deadLetterQueue) {
    if (deadLetterQueue != null && deadLetterQueue.isEmpty()) {
      throw new InvalidConfigurationException("Dead letter queue topic name is not specified");
    }
  }

  private void validateConfigurationSchema(Row configuration) {
    if (!configuration.getSchema().equals(configurationSchema())) {
      throw new InvalidConfigurationException(
          "Configuration schema provided does not match expected");
    }
  }

View on GitHub (pinned to 12126d8942)

Solutions

Add a column named 'event_timestamp' of type TIMESTAMP to the CREATE TABLE schema.
Rename the existing timestamp column to 'event_timestamp' if it exists under another name.
Cast/declare the field as TIMESTAMP (DATETIME) rather than a numeric or string type.

Example fix

-- before
CREATE TABLE t (payload VARCHAR) ...;
-- after
CREATE TABLE t (payload VARCHAR, event_timestamp TIMESTAMP) ...;

Defensive patterns

Strategy: validation

Validate before calling

boolean hasTs = java.util.stream.Stream.of(tableSchema.getFields()).anyMatch(f -> f.getName().equals("event_timestamp") && f.getType().getTypeName() == org.apache.beam.sdk.schemas.Schema.TypeName.DATETIME);
if (!hasTs) { throw new IllegalArgumentException("Pubsub CREATE TABLE schema must include event_timestamp TIMESTAMP"); }

Try / catch

try { ioProvider.from(tableConfig); } catch (InvalidSchemaException e) { if (e.getMessage().contains("event_timestamp")) { /* add the event_timestamp TIMESTAMP column */ } throw e; }

Prevention

Include event_timestamp TIMESTAMP in every Pubsub CREATE TABLE template.
Do not rename or retype the event_timestamp column; other timestamp names are not accepted.

When it happens

Trigger: CREATE TABLE for a Pubsub topic whose column list omits 'event_timestamp' or declares it with a type other than TIMESTAMP (FieldType.DATETIME), triggering fieldPresent(schema, TIMESTAMP_FIELD, DATETIME) == false.

Common situations: Users defining only payload columns and forgetting the event timestamp column; naming the column differently (e.g. 'timestamp', 'ts'); declaring it as BIGINT or VARCHAR instead of TIMESTAMP.

Understand the failure class

Background: Schema validation failed / invalid input schema: payload rejected because its shape doesn't match the expected schema — this error's family across 28 libraries.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/119bd9a9c5427ffe. Report an issue: GitHub.