apache/beam · error · RuntimeException

Received null value for required field '{fieldName}'.

Error message

Received null value for required field '{fieldName}'.

What it means

addIcebergValue converts Iceberg record values into a Beam Row builder. When the source Iceberg value is null but the corresponding Beam field type is non-nullable, a Row cannot be built, so a RuntimeException naming the field is thrown.

Source

Thrown at sdks/java/io/iceberg/src/main/java/org/apache/beam/sdk/io/iceberg/IcebergUtils.java:545

  /** Converts an Iceberg {@link Record} to a Beam {@link Row}. */
  public static Row icebergRecordToBeamRow(Schema schema, Record record) {
    Row.Builder rowBuilder = Row.withSchema(schema);
    for (Schema.Field field : schema.getFields()) {
      @Nullable Object icebergValue = record.getField(field.getName());
      addIcebergValue(rowBuilder, field, icebergValue);
    }
    return rowBuilder.build();
  }

  private static void addIcebergValue(
      Row.Builder rowBuilder, Schema.Field field, @Nullable Object icebergValue) {
    boolean isNullable = field.getType().getNullable();
    if (icebergValue == null) {
      if (isNullable) {
        rowBuilder.addValue(null);
        return;
      }
      throw new RuntimeException(
          String.format("Received null value for required field '%s'.", field.getName()));
    }
    switch (field.getType().getTypeName()) {
      case BYTE:
      case INT16:
      case INT32:
      case INT64:
      case DECIMAL: // Iceberg and Beam both use BigDecimal
      case FLOAT: // Iceberg and Beam both use float
      case DOUBLE: // Iceberg and Beam both use double
      case STRING: // Iceberg and Beam both use String
      case BOOLEAN: // Iceberg and Beam both use boolean
        rowBuilder.addValue(icebergValue);
        break;
      case ARRAY:
        checkState(
            icebergValue instanceof List,
            "Expected List type for field '%s' but received %s",

View on GitHub (pinned to 12126d8942)

Solutions

1. Align nullability: declare the Beam field as nullable (FieldType.withNullable(true)) to tolerate nulls
2. Fix the data: filter or backfill Iceberg rows containing nulls in required columns before converting
3. If the column is truly required, validate the Iceberg table schema (icebergTable.schema().findField(name).isOptional()) before reading
4. Wrap record conversion and fall back to a default value for the field when null is encountered

Example fix

// before
FieldType ft = FieldType.STRING; // non-nullable by default
// after
FieldType ft = FieldType.STRING.withNullable(true);

if (v == null && !field.getType().getNullable()) {
  throw new IllegalArgumentException("Field '" + field.getName() + "' is required but value is null");
}

boolean safeForRequired(Object v, Schema.Field f) {
  return f.getType().getNullable() || v != null;
}

try {
  row = icebergRecordToBeamRow(schema, record);
} catch (RuntimeException e) {
  if (e.getMessage() != null && e.getMessage().startsWith("Received null value")) {
    LOG.warn("Null in required field, routing record to dead-letter");
  } else { throw e; }
}

When it happens

Trigger: structToRow or icebergRecordToBeamRow processes a required (non-optional) Beam field whose Iceberg value is null — e.g. the Iceberg table column was made required after old rows were written with nulls, or the Beam schema was declared with nullable=false while the underlying data is nullable.

Common situations: Schema evolution in Iceberg where a column changed from optional to required; Beam schema derived from a subset of columns with wrong nullability; reading legacy Iceberg data written before a NOT NULL constraint was added.

Understand the failure class

Background: Schema validation failed / invalid input schema: payload rejected because its shape doesn't match the expected schema — this error's family across 28 libraries.

Related errors

• Null collection element type at field {}
• Unsupported Beam type: {typeName}
• Unknown File Format: {}
• cannot encode a null Byte
• cannot estimate size for unsupported null value

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/d056f3c97da2b321. Report an issue: GitHub.