apache/beam · error · IllegalArgumentException

A 'datagen' table requires either 'rows-per-second' (for unb

Error message

A 'datagen' table requires either 'rows-per-second' (for unbounded) or 'number-of-rows' (for bounded) in TBLPROPERTIES.

What it means

A 'datagen' test-data table in Beam SQL needs to know how to generate rows: 'rows-per-second' for an unbounded source or 'number-of-rows' for a bounded source, both given in TBLPROPERTIES. expand throws IllegalArgumentException when neither property is present, since there is no default generation rate or row count.

Source

Thrown at sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/datagen/DataGeneratorPTransform.java:53

  private final ObjectNode properties;

  public DataGeneratorPTransform(Schema schema, ObjectNode properties) {
    this.schema = schema;
    this.properties = properties;
  }

  @Override
  public PCollection<Row> expand(PBegin input) {
    GenerateSequence generator;
    JsonNode rpsNode = properties.path("rows-per-second");
    JsonNode numRowsNode = properties.path("number-of-rows");

    if (!rpsNode.isMissingNode()) {
      generator = GenerateSequence.from(0).withRate(rpsNode.asLong(), Duration.standardSeconds(1));
    } else if (!numRowsNode.isMissingNode()) {
      generator = GenerateSequence.from(0).to(numRowsNode.asLong());
    } else {
      throw new IllegalArgumentException(
          "A 'datagen' table requires either 'rows-per-second' (for unbounded) or "
              + "'number-of-rows' (for bounded) in TBLPROPERTIES.");
    }

    String behavior = properties.path("timestamp.behavior").asText("processing-time");
    @Nullable String eventTimeColumn = null;

    if ("event-time".equalsIgnoreCase(behavior)) {
      JsonNode columnNode = properties.path("event-time.timestamp-column");

      if (columnNode.isMissingNode() || columnNode.isNull()) {
        throw new IllegalArgumentException(
            "For 'event-time' behavior, 'event-time.timestamp-column' must be specified.");
      }
      eventTimeColumn = columnNode.asText();

      // Validate that the specified column exists and is of type TIMESTAMP.
      if (!schema.hasField(eventTimeColumn)) {

View on GitHub (pinned to 12126d8942)

Solutions

Add 'rows-per-second': <n> to TBLPROPERTIES for an unbounded streaming source.
Add 'number-of-rows': <n> to TBLPROPERTIES for a bounded batch source.
Verify exact key spelling: 'rows-per-second' and 'number-of-rows' (hyphenated, lowercase) in the TBLPROPERTIES JSON.

Example fix

-- before
CREATE EXTERNAL TABLE dg (id BIGINT) TYPE 'datagen' TBLPROPERTIES '{}';

-- after
CREATE EXTERNAL TABLE dg (id BIGINT) TYPE 'datagen' TBLPROPERTIES '{"rows-per-second": 10}';

Defensive patterns

Strategy: validation

Validate before calling

JsonNode props = properties;
if (props.path("rows-per-second").isMissingNode() && props.path("number-of-rows").isMissingNode()) {
  throw new IllegalArgumentException("datagen TBLPROPERTIES must set 'rows-per-second' or 'number-of-rows'");
}

Try / catch

try {
  PCollection<Row> rows = datagenTable.expand(input);
} catch (IllegalArgumentException e) {
  if (e.getMessage().contains("requires either 'rows-per-second'")) {
    // add one of the two properties to TBLPROPERTIES
  }
}

Prevention

Decide bounded vs unbounded up front and set 'number-of-rows' or 'rows-per-second' accordingly.
Use the exact hyphenated lowercase property names; camelCase will be silently ignored (missing node).
Keep a canonical datagen DDL template in your test utilities.

When it happens

Trigger: CREATE EXTERNAL TABLE ... TYPE 'datagen' (or provider 'datagen') whose TBLPROPERTIES JSON lacks both 'rows-per-second' and 'number-of-rows', then running a query that expands the table's PTransform.

Common situations: Copy-pasting datagen DDL examples and dropping the TBLPROPERTIES; misspelling the keys (e.g. 'rowsPerSecond' — property lookup is via exact JSON path); providing the property at query time instead of in TBLPROPERTIES.

Understand the failure class

Background: "is required", "must be set", "missing required field": configuration validation errors across open-source libraries — this error's family across 36 libraries.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/7d89afab5649557b. Report an issue: GitHub.