apache/beam · error · IllegalArgumentException

The specified 'event-time.timestamp-column' ('%s') does not

Error message

The specified 'event-time.timestamp-column' ('%s') does not exist in the table schema.

What it means

In event-time mode the datagen transform validates that the column named by 'event-time.timestamp-column' actually exists in the table schema (and, next, that it is TIMESTAMP). expand throws IllegalArgumentException when the named column is absent from the schema.

Source

Thrown at sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/datagen/DataGeneratorPTransform.java:72

          "A 'datagen' table requires either 'rows-per-second' (for unbounded) or "
              + "'number-of-rows' (for bounded) in TBLPROPERTIES.");
    }

    String behavior = properties.path("timestamp.behavior").asText("processing-time");
    @Nullable String eventTimeColumn = null;

    if ("event-time".equalsIgnoreCase(behavior)) {
      JsonNode columnNode = properties.path("event-time.timestamp-column");

      if (columnNode.isMissingNode() || columnNode.isNull()) {
        throw new IllegalArgumentException(
            "For 'event-time' behavior, 'event-time.timestamp-column' must be specified.");
      }
      eventTimeColumn = columnNode.asText();

      // Validate that the specified column exists and is of type TIMESTAMP.
      if (!schema.hasField(eventTimeColumn)) {
        throw new IllegalArgumentException(
            String.format(
                "The specified 'event-time.timestamp-column' ('%s') does not exist in the table schema.",
                eventTimeColumn));
      }

      Schema.Field eventTimeField = schema.getField(eventTimeColumn);
      if (!Schema.TypeName.DATETIME.equals(eventTimeField.getType().getTypeName())) {
        throw new IllegalArgumentException(
            String.format(
                "The specified 'event-time.timestamp-column' ('%s') must be of type TIMESTAMP, but was '%s'.",
                eventTimeColumn, eventTimeField.getType()));
      }

      long maxOutOfOrdernessMs = properties.path("event_time.max-out-of-orderness").asLong(0L);
      generator = generator.withTimestampFn(new AdvancingTimestampFn(maxOutOfOrdernessMs));
    }

    return input

View on GitHub (pinned to 12126d8942)

Solutions

Set 'event-time.timestamp-column' to the exact (case-sensitive) name of a field present in the schema.
Add a TIMESTAMP column to the schema if none exists, e.g. (event_ts TIMESTAMP).
Verify with the subsequent check too: the referenced column must be of TIMESTAMP type or the next validation will fail.

Example fix

-- before
CREATE EXTERNAL TABLE dg (id BIGINT) TYPE 'datagen'
TBLPROPERTIES '{"timestamp.behavior": "event-time", "event-time.timestamp-column": "ts"}';

-- after
CREATE EXTERNAL TABLE dg (id BIGINT, ts TIMESTAMP) TYPE 'datagen'
TBLPROPERTIES '{"timestamp.behavior": "event-time", "event-time.timestamp-column": "ts"}';

Defensive patterns

Strategy: validation

Validate before calling

String col = properties.path("event-time.timestamp-column").asText(null);
if (col == null || !schema.hasField(col)) {
  throw new IllegalArgumentException("'event-time.timestamp-column' must name an existing schema field");
}
if (schema.getField(col).getType().getTypeName() != Schema.TypeName.TIMESTAMP) {
  throw new IllegalArgumentException("'event-time.timestamp-column' must be a TIMESTAMP field");
}

Type guard

boolean hasTimestampField(Schema s, String name) {
  Schema.Field f = s.getField(name);
  return f != null && Schema.TypeName.TIMESTAMP == f.getType().getTypeName();
}

Try / catch

try {
  PCollection<Row> rows = datagenTable.expand(input);
} catch (IllegalArgumentException e) {
  if (e.getMessage().contains("does not exist in the table schema")) {
    // fix the column name in TBLPROPERTIES to match the schema exactly
  }
}

Prevention

Use the exact schema field name (case-sensitive) in event-time.timestamp-column.
Update TBLPROPERTIES whenever the schema column is renamed.
Ensure the referenced field is TIMESTAMP, not DATE or BIGINT, to pass the follow-up type check.

When it happens

Trigger: TBLPROPERTIES has "event-time.timestamp-column": "ts" but the CREATE TABLE schema has no field named 'ts' — due to a typo, case mismatch, or renaming the column without updating TBLPROPERTIES.

Common situations: Renaming the timestamp column in the schema but leaving the old name in TBLPROPERTIES; referencing a column from another table; case-sensitive mismatch like 'EventTime' vs 'eventtime'.

Understand the failure class

Background: 'Could not be found', 'does not exist', 'not found in database': the resource-not-found family when an ID, slug, key, or URI lookup comes back empty — this error's family across 20 libraries.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/7bdc47a1283d8854. Report an issue: GitHub.