apache/beam · error · IllegalArgumentException

header does not contain required %s field: %s

Error message

header does not contain required %s field: %s

What it means

CsvIOParseHelpers.getIndex() locates a schema field's column position in the CSV header. When the header doesn't contain the field's name and the field is not nullable, it throws IllegalArgumentException, since a required field cannot be populated from a missing column. Nullable fields just return -1 instead.

Source

Thrown at sdks/java/io/csv/src/main/java/org/apache/beam/sdk/io/csv/CsvIOParseHelpers.java:120

    }
    return indexToFieldMap;
  }

  /**
   * Attains expected index from {@link CSVFormat's} header matching a given {@link Schema.Field}.
   */
  private static int getIndex(List<String> header, Schema.Field field) {
    String fieldName = field.getName();
    boolean presentInHeader = header.contains(fieldName);
    boolean isNullable = field.getType().getNullable();
    if (presentInHeader) {
      return header.indexOf(fieldName);
    }
    if (isNullable) {
      return -1;
    }

    throw new IllegalArgumentException(
        String.format("header does not contain required %s field: %s", Schema.class, fieldName));
  }

  /**
   * Parse the given {@link String} cell of the CSV record based on the given field's {@link
   * Schema.FieldType}.
   */
  static Object parseCell(String cell, Schema.Field field) {
    Schema.FieldType fieldType = field.getType();
    try {
      switch (fieldType.getTypeName()) {
        case STRING:
          return cell;
        case INT16:
          return Short.parseShort(cell);
        case INT32:
          return Integer.parseInt(cell);
        case INT64:

View on GitHub (pinned to 12126d8942)

Solutions

1. Make the header match the schema: add the missing column (with values) to the CSV file.
2. Mark the field nullable in the schema if the column is genuinely optional.
3. Verify withHeader()/auto-detect configuration and delimiter so the header row is parsed correctly.

Example fix

// before (schema requires 'email', header: id,name)
// after
// option A: fix file header -> id,name,email
// option B: schema field
Schema.Field.of("email", FieldType.STRING.withNullable(true));

// compare header to required schema fields before reading
List<String> required = schema.getFields().stream()
    .filter(f -> !f.getType().getNullable())
    .map(Schema.Field::getName).collect(Collectors.toList());
List<String> header = Arrays.asList(firstLine.split(","));
required.forEach(r -> { if (!header.contains(r)) throw new IllegalStateException("missing required column: " + r); });

try { pipeline.apply(CsvIO.read(path)); } catch (IllegalArgumentException e) { /* log missing header field, fix file or schema */ }

When it happens

Trigger: Reading a CSV file whose header row is missing a column that maps to a required (non-nullable) schema field, e.g. header 'id,name' while the record schema requires field 'email'.

Common situations: Schema and file drifted apart after adding a new required field; using the wrong delimiter so the header parses into fewer columns; pointing CsvIO at a headerless or differently-named file version.

Understand the failure class

Background: Schema validation failed / invalid input schema: payload rejected because its shape doesn't match the expected schema — this error's family across 28 libraries.

Related errors

• requires an input Schema. Note that only Row or user classe
• Unsupported type: , consider using withCustomRecordParsing
• field was received -- type mismatch
• Required org.apache.beam.sdk.schemas.Schema field has null
• Column delimiter should be set if headers are present.

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/e1b797aef55eb7d4. Report an issue: GitHub.