{"record":{"id":"e1b797aef55eb7d4","repo":"apache/beam","slug":"header-does-not-contain-required-s-field-s","errorCode":null,"errorMessage":"header does not contain required %s field: %s","messagePattern":"header does not contain required (.+?) field: (.+?)","errorType":"validation","errorClass":"IllegalArgumentException","httpStatus":null,"severity":"error","filePath":"sdks/java/io/csv/src/main/java/org/apache/beam/sdk/io/csv/CsvIOParseHelpers.java","lineNumber":120,"sourceCode":"    }\n    return indexToFieldMap;\n  }\n\n  /**\n   * Attains expected index from {@link CSVFormat's} header matching a given {@link Schema.Field}.\n   */\n  private static int getIndex(List<String> header, Schema.Field field) {\n    String fieldName = field.getName();\n    boolean presentInHeader = header.contains(fieldName);\n    boolean isNullable = field.getType().getNullable();\n    if (presentInHeader) {\n      return header.indexOf(fieldName);\n    }\n    if (isNullable) {\n      return -1;\n    }\n\n    throw new IllegalArgumentException(\n        String.format(\"header does not contain required %s field: %s\", Schema.class, fieldName));\n  }\n\n  /**\n   * Parse the given {@link String} cell of the CSV record based on the given field's {@link\n   * Schema.FieldType}.\n   */\n  static Object parseCell(String cell, Schema.Field field) {\n    Schema.FieldType fieldType = field.getType();\n    try {\n      switch (fieldType.getTypeName()) {\n        case STRING:\n          return cell;\n        case INT16:\n          return Short.parseShort(cell);\n        case INT32:\n          return Integer.parseInt(cell);\n        case INT64:","sourceCodeStart":102,"sourceCodeEnd":138,"githubUrl":"https://github.com/apache/beam/blob/12126d8942aaf848030c478b4c6a28c6af861c66/sdks/java/io/csv/src/main/java/org/apache/beam/sdk/io/csv/CsvIOParseHelpers.java#L102-L138","documentation":"CsvIOParseHelpers.getIndex() locates a schema field's column position in the CSV header. When the header doesn't contain the field's name and the field is not nullable, it throws IllegalArgumentException, since a required field cannot be populated from a missing column. Nullable fields just return -1 instead.","triggerScenarios":"Reading a CSV file whose header row is missing a column that maps to a required (non-nullable) schema field, e.g. header 'id,name' while the record schema requires field 'email'.","commonSituations":"Schema and file drifted apart after adding a new required field; using the wrong delimiter so the header parses into fewer columns; pointing CsvIO at a headerless or differently-named file version.","solutions":["Make the header match the schema: add the missing column (with values) to the CSV file.","Mark the field nullable in the schema if the column is genuinely optional.","Verify withHeader()/auto-detect configuration and delimiter so the header row is parsed correctly."],"exampleFix":"// before (schema requires 'email', header: id,name)\n// after\n// option A: fix file header -> id,name,email\n// option B: schema field\nSchema.Field.of(\"email\", FieldType.STRING.withNullable(true));","handlingStrategy":"validation","validationCode":"// compare header to required schema fields before reading\nList<String> required = schema.getFields().stream()\n    .filter(f -> !f.getType().getNullable())\n    .map(Schema.Field::getName).collect(Collectors.toList());\nList<String> header = Arrays.asList(firstLine.split(\",\"));\nrequired.forEach(r -> { if (!header.contains(r)) throw new IllegalStateException(\"missing required column: \" + r); });","typeGuard":null,"tryCatchPattern":"try { pipeline.apply(CsvIO.read(path)); } catch (IllegalArgumentException e) { /* log missing header field, fix file or schema */ }","preventionTips":["Keep header names and schema field names in lockstep; version schemas alongside data files.","Declare optional columns nullable in the schema.","Dry-run the header against the schema on a sample file before full pipeline runs."],"tags":["java","beam-io","csv","header-mismatch"],"backgroundTag":"schema-validation-failed","analyzedSha":"12126d8942aaf848030c478b4c6a28c6af861c66","analyzedAt":"2026-09-13T01:50:10.254Z","contentChangedAt":"2026-09-13T01:50:10.254Z","schemaVersion":2},"datasetVersion":"2026-09-14T16:17:12.679Z"}