apache/beam · error · java.lang.IllegalArgumentException

Expect fields, but actually

Error message

Expect %d fields, but actually %d

What it means

When converting CSV lines to Beam Rows, BeamTableUtils validates that each parsed CSV record has exactly as many fields as the target schema. A mismatch throws IllegalArgumentException with the expected and actual counts.

Solutions

1. Fix the source CSV so each row has the schema's field count
2. Set the correct delimiter/quote/escape in the CSVFormat to match the data
3. Skip header rows or filter malformed lines before csvLines2BeamRows
4. Adjust the schema to match the actual data layout

Example fix

// before
CSVFormat.DEFAULT // data is semicolon-separated
// after
CSVFormat.DEFAULT.withDelimiter(';')

long expected = schema.getFieldCount();
long actual = line.chars().filter(c -> c == ',').count() + 1;
if (actual != expected) throw new IllegalArgumentException("field count mismatch");

try { rows = BeamTableUtils.csvLines2BeamRows(line, schema, format); } catch (IllegalArgumentException e) { log.warn("skipping malformed row"); }

When it happens

Trigger: Parsing a CSV line whose number of columns differs from schema.getFieldCount() — extra or missing delimiters, ragged rows, wrong delimiter configured in CSVFormat, or a trailing separator producing an empty extra field.

Common situations: Malformed input files with unquoted embedded commas; users changing CSVFormat delimiter without updating the schema; header rows accidentally parsed as data (or vice versa).

Understand the failure class

Background: Schema validation failed / invalid input schema: payload rejected because its shape doesn't match the expected schema — this error's family across 28 libraries.

Related errors

• Column type is not supported yet!
• Could not parse CSV records from
• Field not nullable
• malformed input for decoding
• requires an input Schema. Note that only Row or user…

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/45dfea4833f4be31. Report an issue: GitHub.

Appendix: source

Thrown at sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/schema/BeamTableUtils.java:75

  /**
   * Decode zero or more CSV records from the given string, according to the specified {@link
   * CSVFormat}, and converts them to {@link Row Rows} with the specified {@link Schema}.
   *
   * <p>A single "line" read from e.g. {@link TextIO} can have zero or more records, depending on
   * whether the line was split on the same characters that delimite CSV records, and whether the
   * {@link CSVFormat} ignores blank lines.
   */
  public static Iterable<Row> csvLines2BeamRows(CSVFormat csvFormat, String line, Schema schema) {
    // Empty lines can result in empty strings after Beam splits the file,
    // which are not empty records to CSVParser unless they have a record terminator.
    if (!line.endsWith(csvFormat.getRecordSeparator())) {
      line += csvFormat.getRecordSeparator();
    }
    try (CSVParser parser = CSVParser.parse(line, csvFormat)) {
      List<Row> rows = new ArrayList<>();
      for (CSVRecord rawRecord : parser.getRecords()) {
        if (rawRecord.size() != schema.getFieldCount()) {
          throw new IllegalArgumentException(
              String.format(
                  "Expect %d fields, but actually %d", schema.getFieldCount(), rawRecord.size()));
        }
        rows.add(
            IntStream.range(0, schema.getFieldCount())
                .mapToObj(idx -> autoCastField(schema.getField(idx), rawRecord.get(idx)))
                .collect(toRow(schema)));
      }
      return rows;
    } catch (IOException e) {
      throw new IllegalArgumentException(
          String.format("Could not parse CSV records from %s with format %s", line, csvFormat), e);
    }
  }

  public static String beamRow2CsvLine(Row row, CSVFormat csvFormat) {
    StringWriter writer = new StringWriter();
    try (CSVPrinter printer = csvFormat.print(writer)) {

View on GitHub (pinned to 12126d8942)