{"record":{"id":"e0c0ab1b62d1072f","repo":"apache/iceberg","slug":"field-s-in-target-schema-s-is-non-nullable-but-d-e0c0ab","errorCode":null,"errorMessage":"Field %s in target schema %s is non-nullable but does not exist in source schema.","messagePattern":"Field (.+?) in target schema (.+?) is non-nullable but does not exist in source schema\\.","errorType":"validation","errorClass":"IllegalArgumentException","httpStatus":null,"severity":"error","filePath":"flink/v2.2/flink/src/main/java/org/apache/iceberg/flink/sink/dynamic/DataConverter.java","lineNumber":153,"sourceCode":"  class RowDataConverter implements DataConverter {\n    private final RowData.FieldGetter[] fieldGetters;\n    private final DataConverter[] dataConverters;\n\n    RowDataConverter(RowType sourceType, RowType targetType) {\n      this.fieldGetters = new RowData.FieldGetter[targetType.getFields().size()];\n      this.dataConverters = new DataConverter[targetType.getFields().size()];\n\n      for (int i = 0; i < targetType.getFields().size(); i++) {\n        RowData.FieldGetter fieldGetter;\n        DataConverter dataConverter;\n        RowType.RowField targetField = targetType.getFields().get(i);\n        int sourceFieldIndex = sourceType.getFieldIndex(targetField.getName());\n        if (sourceFieldIndex == -1) {\n          if (targetField.getType().isNullable()) {\n            fieldGetter = row -> null;\n            dataConverter = value -> null;\n          } else {\n            throw new IllegalArgumentException(\n                String.format(\n                    \"Field %s in target schema %s is non-nullable but does not exist in source schema.\",\n                    i + 1, targetType));\n          }\n        } else {\n          RowType.RowField sourceField = sourceType.getFields().get(sourceFieldIndex);\n          fieldGetter = RowData.createFieldGetter(sourceField.getType(), sourceFieldIndex);\n          dataConverter = DataConverter.getNullable(sourceField.getType(), targetField.getType());\n        }\n\n        this.fieldGetters[i] = fieldGetter;\n        this.dataConverters[i] = dataConverter;\n      }\n    }\n\n    @Override\n    public RowData convert(Object object) {\n      RowData sourceData = (RowData) object;","sourceCodeStart":135,"sourceCodeEnd":171,"githubUrl":"https://github.com/apache/iceberg/blob/86d9c8fc543e7c56c9f624eb725f76c9baff9570/flink/v2.2/flink/src/main/java/org/apache/iceberg/flink/sink/dynamic/DataConverter.java#L135-L171","documentation":"RowDataConverter maps source rows to a target schema. When a target field is missing from the source schema, it can only be filled with null — which requires the target field to be nullable. A non-nullable target field absent from the source makes the conversion impossible, so IllegalArgumentException is thrown naming the field index and target schema.","triggerScenarios":"Schema evolution where a new REQUIRED (non-nullable) column is added to the target table schema while source rows do not contain that field; constructing RowDataConverter with such source/target RowTypes.","commonSituations":"Adding a required column to the Iceberg table that the upstream Flink source does not produce; typo in the new field name so it never matches a source field.","solutions":["Make the new target field optional (nullable) in the Iceberg schema, or provide a default value.","Add the field to the source schema so the converter can map real values.","Fix the field name if it was a typo and should match an existing source field."],"exampleFix":"// before: adding a required column the source lacks\nSchema schema = new Schema(Types.NestedField.required(1, \"new_col\", Types.IntegerType.get()));\n// after: add it as optional\nSchema schema = new Schema(Types.NestedField.optional(1, \"new_col\", Types.IntegerType.get()));","handlingStrategy":"validation","validationCode":"for (RowType.RowField f : targetRowType.getFields()) { if (!f.getType().isNullable() && sourceRowType.getFieldIndex(f.getName()) == -1) { throw new IllegalArgumentException(\"Source lacks required field: \" + f.getName()); } }","typeGuard":null,"tryCatchPattern":"try { converter = new RowDataConverter(source, target); } catch (IllegalArgumentException e) { /* make the field optional in the target schema or add it to source */ throw e; }","preventionTips":["Only add optional (nullable) columns via schema evolution unless the source also produces them","Cross-check new field names against the source schema before committing the table schema","Unit-test schema evolution paths with the real source RowType"],"tags":["flink","schema-evolution","nullable"],"backgroundTag":"schema-validation-failed","analyzedSha":"86d9c8fc543e7c56c9f624eb725f76c9baff9570","analyzedAt":"2026-09-12T00:46:39.097Z","contentChangedAt":"2026-09-12T00:46:39.097Z","schemaVersion":2},"datasetVersion":"2026-09-14T16:17:12.679Z"}