apache/beam · error · IllegalArgumentException

Received null value for non-nullable field ""

Error message

Received null value for non-nullable field ""

What it means

Thrown during BigQuery JSON -> Beam Row conversion (fromJsonToBeamField path in BigQueryUtils) when the JSON value for a field is null but the Beam FieldType is non-nullable. Collection-typed fields are converted to an empty list instead, but scalar required fields cause an immediate IllegalArgumentException naming the offending field. The library refuses to invent a value for a required scalar.

Solutions

1. Align the Beam schema with the actual BigQuery table schema: mark the field nullable (FieldType.withNullable(true)).
2. Use toBeamRow with a schema generated from the table via BigQueryUtils.fromTableSchema / fromBeamType so nullability matches.
3. Coalesce nulls upstream in the query (IFNULL(col, default)) or fill defaults in the TableRow before conversion.
4. Drop or filter rows containing nulls in required columns before calling toBeamRow.

Example fix

// before
Field f = Field.of("amount", FieldType.DOUBLE); // required, but table column NULLABLE

// after
Field f = Field.of("amount", FieldType.DOUBLE.withNullable(true));
// or: row.set("amount", row.get("amount") == null ? 0.0 : row.get("amount"));

// Java: check TableRow against Beam schema before toBeamRow
schema.getFields().forEach(f -> {
  if (!f.getType().getNullable()
      && !f.getType().getTypeName().isCollectionType()
      && tableRow.get(f.getName()) == null) {
    throw new IllegalArgumentException("Missing required field: " + f.getName());
  }
});

// Java
static boolean requiredFieldsPresent(TableRow tr, Schema schema) {
  return schema.getFields().stream()
      .filter(f -> !f.getType().getNullable())
      .allMatch(f -> tr.containsKey(f.getName()) && tr.get(f.getName()) != null);
}

try {
  Row row = BigQueryUtils.toBeamRow(schema, tableRow);
} catch (IllegalArgumentException e) {
  // dead-letter tableRow with the exception message
}

When it happens

Trigger: Calling BigQueryUtils.toBeamRow(schema, tableRow) (or the JSON variant) where TableRow.get(field) returns null for a field whose Beam FieldType has nullable=false and is not a collection type.

Common situations: Reading from a BigQuery table whose column is NULLABLE/MODE NULLABLE while the Beam schema was generated as REQUIRED (or vice versa after a table schema change); querying with SELECT that omits a column expected by the schema; stale table schemas.

Understand the failure class

Background: Schema validation failed / invalid input schema: payload rejected because its shape doesn't match the expected schema — this error's family across 28 libraries.

Related errors

• Field is not nullable.
• Field not nullable
• Received null value for non-nullable field " +…
• A function must be provided to convert the input type into…
• BigQueryIO.Write transforms cannot be converted to a…

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/a3ca9fefb616bb05. Report an issue: GitHub.

Appendix: source

Thrown at sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryUtils.java:904

    return IntStream.range(0, rowSchema.getFieldCount())
        .boxed()
        .<@Nullable Object>map(
            index -> toBeamValue(rowSchema.getField(index), rawJsonValues.get(index)))
        .collect(toRow(rowSchema));
  }

  private static @Nullable Object toBeamValue(Field field, @Nullable Object jsonBQValue) {
    FieldType fieldType = field.getType();

    if (jsonBQValue == null) {
      if (fieldType.getNullable()) {
        return null;
      } else {
        if (fieldType.getTypeName().isCollectionType()) {
          return Collections.emptyList();
        }
        throw new IllegalArgumentException(
            "Received null value for non-nullable field \"" + field.getName() + "\"");
      }
    }

    if (jsonBQValue instanceof String
        || jsonBQValue instanceof Number
        || jsonBQValue instanceof Boolean) {
      String jsonBQString = jsonBQValue.toString();
      if (JSON_VALUE_PARSERS.containsKey(fieldType.getTypeName())) {
        return JSON_VALUE_PARSERS.get(fieldType.getTypeName()).apply(jsonBQString);
      } else if (fieldType.isLogicalType(SqlTypes.DATETIME.getIdentifier())) {
        try {
          // Handle if datetime value is in micros ie. 123456789
          Long value = Long.parseLong(jsonBQString);
          return CivilTimeEncoder.decodePacked64DatetimeMicrosAsJavaTime(value);
        } catch (NumberFormatException e) {
          // Handle as a String, ie. "2023-02-16 12:00:00"
          try {

View on GitHub (pinned to 12126d8942)