apache/beam · error · IllegalArgumentException

Required org.apache.beam.sdk.schemas.Schema field has null

Error message

Required org.apache.beam.sdk.schemas.Schema field  has null value

What it means

CsvIORecordToObjects.parseCell() throws IllegalArgumentException when a CSV cell is null but the corresponding Schema.Field is declared non-nullable. In CSV parsing a null cell typically comes from a missing/short record, and Beam enforces the schema's nullability contract at row-parse time.

Source

Thrown at sdks/java/io/csv/src/main/java/org/apache/beam/sdk/io/csv/CsvIORecordToObjects.java:117

      } catch (RuntimeException e) {
        receiver
            .get(errorTag)
            .output(
                CsvIOParseError.builder()
                    .setCsvRecord(record.toString())
                    .setMessage(Optional.ofNullable(e.getMessage()).orElse(""))
                    .setStackTrace(Throwables.getStackTraceAsString(e))
                    .setObservedTimestamp(Instant.now())
                    .build());
      }
    }
  }

  /** Parses cell to emit the value, as well as potential errors with filename. */
  Object parseCell(String cell, Schema.Field field) {
    if (cell == null) {
      if (!field.getType().getNullable()) {
        throw new IllegalArgumentException(
            "Required org.apache.beam.sdk.schemas.Schema field "
                + field.getName()
                + " has null value");
      }
      return cell;
    }
    if (customProcessingMap.containsKey(field.getName())) {
      return customProcessingMap.get(field.getName()).apply(cell);
    }
    return CsvIOParseHelpers.parseCell(cell, field);
  }
}

View on GitHub (pinned to 12126d8942)

Solutions

Declare the field nullable: FieldType.STRING.withNullable(true), if missing values are acceptable.
Repair or filter short/malformed rows before parsing.
Use a custom record parser to supply a default value for missing cells.

Example fix

// before: non-nullable field receives null from a short row
// after
Schema.Field.of("optionalNote", FieldType.STRING.withNullable(true));

Defensive patterns

Strategy: try-catch

Validate before calling

// ensure every row has as many columns as the header
if (rowValues.length != header.size()) throw new IllegalStateException("short row: " + Arrays.toString(rowValues));

Try / catch

try { parseRow(cells, schema); } catch (IllegalArgumentException e) { /* send row to dead-letter, continue */ }

Prevention

Mark fields nullable when data files may legitimately omit them.
Filter or repair ragged rows (wrong column counts) before parsing.
Add a data-quality check step that reports short/empty rows and counts.

When it happens

Trigger: Parsing a CSV record whose row has fewer columns than the header (null padding) while the field at that position is non-nullable in the schema.

Common situations: Truncated or ragged CSV lines from manual edits or partial exports; nullable flag removed from the schema after data files were already produced with missing values.

Understand the failure class

Background: Schema validation failed / invalid input schema: payload rejected because its shape doesn't match the expected schema — this error's family across 28 libraries.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/e42f129cf5d657a4. Report an issue: GitHub.