apache/iceberg · error · IllegalArgumentException

Field ${fieldIndex} in target schema ${targetType} is non-nu

Error message

Field ${fieldIndex} in target schema ${targetType} is non-nullable but does not exist in source schema.

What it means

RowDataConverter matches target schema fields to source fields by name. If a target field is absent from the source schema AND the target field is non-nullable, there is no valid value to produce, so it throws IllegalArgumentException. Nullable missing fields are tolerated (filled with null).

Source

Thrown at flink/v2.1/flink/src/main/java/org/apache/iceberg/flink/sink/dynamic/DataConverter.java:153

  class RowDataConverter implements DataConverter {
    private final RowData.FieldGetter[] fieldGetters;
    private final DataConverter[] dataConverters;

    RowDataConverter(RowType sourceType, RowType targetType) {
      this.fieldGetters = new RowData.FieldGetter[targetType.getFields().size()];
      this.dataConverters = new DataConverter[targetType.getFields().size()];

      for (int i = 0; i < targetType.getFields().size(); i++) {
        RowData.FieldGetter fieldGetter;
        DataConverter dataConverter;
        RowType.RowField targetField = targetType.getFields().get(i);
        int sourceFieldIndex = sourceType.getFieldIndex(targetField.getName());
        if (sourceFieldIndex == -1) {
          if (targetField.getType().isNullable()) {
            fieldGetter = row -> null;
            dataConverter = value -> null;
          } else {
            throw new IllegalArgumentException(
                String.format(
                    "Field %s in target schema %s is non-nullable but does not exist in source schema.",
                    i + 1, targetType));
          }
        } else {
          RowType.RowField sourceField = sourceType.getFields().get(sourceFieldIndex);
          fieldGetter = RowData.createFieldGetter(sourceField.getType(), sourceFieldIndex);
          dataConverter = DataConverter.getNullable(sourceField.getType(), targetField.getType());
        }

        this.fieldGetters[i] = fieldGetter;
        this.dataConverters[i] = dataConverter;
      }
    }

    @Override
    public RowData convert(Object object) {
      RowData sourceData = (RowData) object;

View on GitHub (pinned to 86d9c8fc54)

Solutions

Add the new column as nullable, or with a write default, instead of required (ALTER TABLE ... ADD COLUMN c X -- nullable).
Provide a default value for the new field so old data can be converted (ADD COLUMN ... AFTER ... with a write-default).
Backfill the source data so every record contains the new field before switching the sink to the evolved schema.

Example fix

// before
ALTER TABLE db.t ADD COLUMN region STRING NOT NULL;
// after
ALTER TABLE db.t ADD COLUMN region STRING; -- nullable, or provide write-default

Defensive patterns

Strategy: validation

Validate before calling

for (Types.NestedField f : targetSchema.columns()) {
  if (f.isRequired() && sourceSchema.findField(f.name()) == null) {
    throw new IllegalArgumentException("New required field without default: " + f.name());
  }
}

Prevention

Add columns as nullable or with write defaults when evolving schemas.
Backfill source data before making fields required.
Run schema-diff checks between source and target schemas before writing.

When it happens

Trigger: Schema evolution where the target schema adds a required (non-nullable, no default) field that doesn't exist in the source RowData schema being converted.

Common situations: Evolving an Iceberg table by adding a required column with required=true and no write default, then writing old records through the Flink dynamic sink; mismatched schemas passed to the dynamic sink.

Understand the failure class

Background: Schema validation failed / invalid input schema: payload rejected because its shape doesn't match the expected schema — this error's family across 28 libraries.

Related errors

AI-assisted analysis of apache/iceberg@86d9c8fc54 (2026-09-12). Data as JSON: /api/errors/3d00d11a8e1b234c. Report an issue: GitHub.