apache/iceberg · error · IllegalArgumentException

Field %s in target schema %s is non-nullable but does not ex

Error message

Field %s in target schema %s is non-nullable but does not exist in source schema.

What it means

RowDataConverter maps target schema fields to source schema fields by name. If a target field is missing from the source and the target field is nullable, the converter substitutes null; but if the target field is non-nullable, it throws this IllegalArgumentException because a required value cannot be produced. This enforces schema evolution safety: non-nullable target columns must exist in the incoming data.

Source

Thrown at flink/v1.20/flink/src/main/java/org/apache/iceberg/flink/sink/dynamic/DataConverter.java:153

  class RowDataConverter implements DataConverter {
    private final RowData.FieldGetter[] fieldGetters;
    private final DataConverter[] dataConverters;

    RowDataConverter(RowType sourceType, RowType targetType) {
      this.fieldGetters = new RowData.FieldGetter[targetType.getFields().size()];
      this.dataConverters = new DataConverter[targetType.getFields().size()];

      for (int i = 0; i < targetType.getFields().size(); i++) {
        RowData.FieldGetter fieldGetter;
        DataConverter dataConverter;
        RowType.RowField targetField = targetType.getFields().get(i);
        int sourceFieldIndex = sourceType.getFieldIndex(targetField.getName());
        if (sourceFieldIndex == -1) {
          if (targetField.getType().isNullable()) {
            fieldGetter = row -> null;
            dataConverter = value -> null;
          } else {
            throw new IllegalArgumentException(
                String.format(
                    "Field %s in target schema %s is non-nullable but does not exist in source schema.",
                    i + 1, targetType));
          }
        } else {
          RowType.RowField sourceField = sourceType.getFields().get(sourceFieldIndex);
          fieldGetter = RowData.createFieldGetter(sourceField.getType(), sourceFieldIndex);
          dataConverter = DataConverter.getNullable(sourceField.getType(), targetField.getType());
        }

        this.fieldGetters[i] = fieldGetter;
        this.dataConverters[i] = dataConverter;
      }
    }

    @Override
    public RowData convert(Object object) {
      RowData sourceData = (RowData) object;

View on GitHub (pinned to 86d9c8fc54)

Solutions

Make the new target column nullable so missing source values map to null
Update the source RowData schema/data to include the new field with a value
Use a schema evolution with a default value (Iceberg add-column with write-default/initial-default) if the table format supports it

Example fix

// before
// target schema: RequiredField LONG (required), absent in source
// after
// evolve target so the new field is nullable, or supply the field in the source RowType

Defensive patterns

Strategy: validation

Validate before calling

for (RowType.RowField targetField : targetRowType.getFields()) {
  if (!targetField.getType().isNullable()
      && sourceRowType.getFieldIndex(targetField.getName()) == -1) {
    throw new IllegalArgumentException(
        "Non-nullable target field missing in source: " + targetField.getName());
  }
}

Try / catch

try {
  converter = new RowDataConverter(sourceRowType, targetRowType);
} catch (IllegalArgumentException e) {
  // evolve schema or update source before writing
}

Prevention

Make all ALTER TABLE ADD COLUMN targets nullable unless the pipeline guarantees values
Keep source stream schema and target table schema in sync through schema evolution
Validate schema compatibility at job startup before writing

When it happens

Trigger: Evolving the target table by adding a REQUIRED (non-nullable) column that does not exist in the incoming Flink RowData schema, then writing through the dynamic sink.

Common situations: ALTER TABLE ADD COLUMN without making the new column nullable; source stream schema not updated after a target schema migration; backfilling old records against an evolved target schema.

Understand the failure class

Background: Schema validation failed / invalid input schema: payload rejected because its shape doesn't match the expected schema — this error's family across 28 libraries.

Related errors

AI-assisted analysis of apache/iceberg@86d9c8fc54 (2026-09-12). Data as JSON: /api/errors/f41800bf5f6bf3b9. Report an issue: GitHub.