apache/beam · error · InvalidTableException

Table with type 'text' and format 'lines' must have exactly

Error message

Table with type 'text' and format 'lines' must have exactly one STRING/VARCHAR/CHAR column

What it means

TextTableProvider.buildBeamSqlTable enforces that a text table with format 'lines' has exactly one field of type STRING (VARCHAR/CHAR). If the schema has a different field count or a non-STRING type, InvalidTableException is thrown at table creation because each line can only map to one string value.

Source

Thrown at sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/text/TextTableProvider.java:120

    switch (format) {
      case "csv":
        String specifiedCsvFormat = properties.path("csvformat").asText(null);
        CSVFormat csvFormat =
            specifiedCsvFormat != null
                ? CSVFormat.valueOf(specifiedCsvFormat)
                : (legacyCsvFormat != null
                    ? CSVFormat.valueOf(legacyCsvFormat)
                    : CSVFormat.DEFAULT);
        return new TextTable(
            schema, filePattern, new CsvToRow(schema, csvFormat), new RowToCsv(csvFormat));
      case "json":
        return new TextJsonTable(
            schema, filePattern, JsonToRow.create(schema, deadLetterFile), RowToJson.create());
      case "lines":
        if (!(schema.getFieldCount() == 1
            && schema.getField(0).getType().getTypeName().equals(TypeName.STRING))) {
          throw new InvalidTableException(
              "Table with type 'text' and format 'lines' "
                  + "must have exactly one STRING/VARCHAR/CHAR column ");
        }
        return new TextTable(
            schema, filePattern, new LinesReadConverter(), new LinesWriteConverter());
      default:
        throw new InvalidTableException(
            "Table with type 'text' must have format 'csv' or 'lines' or 'json'");
    }
  }

  /** Write-side converter for for {@link TextTable} with format {@code 'lines'}. */
  public static class LinesWriteConverter extends PTransform<PCollection<Row>, PCollection<String>>
      implements Serializable {

    public LinesWriteConverter() {}

    @Override

View on GitHub (pinned to 12126d8942)

Solutions

Define exactly one column of type STRING (or VARCHAR/CHAR) for the lines table.
Switch the format to 'csv' or 'json' if multiple/typed columns are needed.
Parse extra fields out of the single STRING column afterwards (e.g. via UDF or subsequent SELECT with string functions).

Example fix

// before
CREATE EXTERNAL TABLE t (a INT, b INT) TYPE 'text' LOCATION 'path' TBLPROPERTIES {'format':'lines'}
// after
CREATE EXTERNAL TABLE t (line STRING) TYPE 'text' LOCATION 'path' TBLPROPERTIES {'format':'lines'}

Defensive patterns

Strategy: validation

Validate before calling

boolean validLines = schema.getFieldCount() == 1 && schema.getField(0).getType().getTypeName().equals(TypeName.STRING);
if (!validLines) throw new IllegalArgumentException("text/lines requires exactly one STRING column");

Try / catch

try { table = provider.buildBeamSqlTable(table); } catch (InvalidTableException e) { /* use csv/json format or fix schema */ }

Prevention

Use 'lines' only for single-STRING schemas
Choose csv/json for multi-column data
Validate schema before CREATE EXTERNAL TABLE

When it happens

Trigger: Declaring a text/lines table with zero, two, or more columns; or with a single column typed as BYTES, INT, etc. — the check 'schema.getFieldCount() == 1 && type == STRING' fails.

Common situations: Copying a multi-column schema from a csv/json table to a lines table; using an INT column assuming auto-conversion of each line; forgetting that 'lines' maps each raw line to one VARCHAR.

Understand the failure class

Background: Schema validation failed / invalid input schema: payload rejected because its shape doesn't match the expected schema — this error's family across 28 libraries.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/40a5256d044c0e09. Report an issue: GitHub.