apache/iceberg · error · IllegalArgumentException

Field %s in target schema %s is non-nullable but does not ex

Error message

Field %s in target schema %s is non-nullable but does not exist in source schema.

What it means

RowDataConverter maps source rows to a target schema. When a target field is missing from the source schema, it can only be filled with null — which requires the target field to be nullable. A non-nullable target field absent from the source makes the conversion impossible, so IllegalArgumentException is thrown naming the field index and target schema.

Source

Thrown at flink/v2.2/flink/src/main/java/org/apache/iceberg/flink/sink/dynamic/DataConverter.java:153

  class RowDataConverter implements DataConverter {
    private final RowData.FieldGetter[] fieldGetters;
    private final DataConverter[] dataConverters;

    RowDataConverter(RowType sourceType, RowType targetType) {
      this.fieldGetters = new RowData.FieldGetter[targetType.getFields().size()];
      this.dataConverters = new DataConverter[targetType.getFields().size()];

      for (int i = 0; i < targetType.getFields().size(); i++) {
        RowData.FieldGetter fieldGetter;
        DataConverter dataConverter;
        RowType.RowField targetField = targetType.getFields().get(i);
        int sourceFieldIndex = sourceType.getFieldIndex(targetField.getName());
        if (sourceFieldIndex == -1) {
          if (targetField.getType().isNullable()) {
            fieldGetter = row -> null;
            dataConverter = value -> null;
          } else {
            throw new IllegalArgumentException(
                String.format(
                    "Field %s in target schema %s is non-nullable but does not exist in source schema.",
                    i + 1, targetType));
          }
        } else {
          RowType.RowField sourceField = sourceType.getFields().get(sourceFieldIndex);
          fieldGetter = RowData.createFieldGetter(sourceField.getType(), sourceFieldIndex);
          dataConverter = DataConverter.getNullable(sourceField.getType(), targetField.getType());
        }

        this.fieldGetters[i] = fieldGetter;
        this.dataConverters[i] = dataConverter;
      }
    }

    @Override
    public RowData convert(Object object) {
      RowData sourceData = (RowData) object;

View on GitHub (pinned to 86d9c8fc54)

Solutions

1. Make the new target field optional (nullable) in the Iceberg schema, or provide a default value.
2. Add the field to the source schema so the converter can map real values.
3. Fix the field name if it was a typo and should match an existing source field.

Example fix

// before: adding a required column the source lacks
Schema schema = new Schema(Types.NestedField.required(1, "new_col", Types.IntegerType.get()));
// after: add it as optional
Schema schema = new Schema(Types.NestedField.optional(1, "new_col", Types.IntegerType.get()));

for (RowType.RowField f : targetRowType.getFields()) { if (!f.getType().isNullable() && sourceRowType.getFieldIndex(f.getName()) == -1) { throw new IllegalArgumentException("Source lacks required field: " + f.getName()); } }

try { converter = new RowDataConverter(source, target); } catch (IllegalArgumentException e) { /* make the field optional in the target schema or add it to source */ throw e; }

When it happens

Trigger: Schema evolution where a new REQUIRED (non-nullable) column is added to the target table schema while source rows do not contain that field; constructing RowDataConverter with such source/target RowTypes.

Common situations: Adding a required column to the Iceberg table that the upstream Flink source does not produce; typo in the new field name so it never matches a source field.

Understand the failure class

Background: Schema validation failed / invalid input schema: payload rejected because its shape doesn't match the expected schema — this error's family across 28 libraries.

Related errors

• Field %s in target schema %s is non-nullable but does not ex
• Altering schema is not supported in the old alterTable API.
• Altering schema is not supported in the old alterTable API.
• Invalid primary key '%s'. Column '%s' is nullable.
• Field ${fieldIndex} in target schema ${targetType} is non-nu

AI-assisted analysis of apache/iceberg@86d9c8fc54 (2026-09-12). Data as JSON: /api/errors/e0c0ab1b62d1072f. Report an issue: GitHub.