{"record":{"id":"fac9c2abd075f635","repo":"apache/iceberg","slug":"missing-required-field-s-fac9c2","errorCode":null,"errorMessage":"Missing required field: %s","messagePattern":"Missing required field: (.+?)","errorType":"exception","errorClass":"IllegalArgumentException","httpStatus":null,"severity":"error","filePath":"parquet/src/main/java/org/apache/iceberg/parquet/ParquetValueReaders.java","lineNumber":286,"sourceCode":"        ? reader\n        : new PresenceReader<>(reader, presence, fileSchema.getMaxRepetitionLevel(structPath));\n  }\n\n  private static ParquetValueReader<?> defaultReader(\n      Types.NestedField field,\n      ParquetValueReader<?> reader,\n      int constantDefinitionLevel,\n      BiFunction<org.apache.iceberg.types.Type, Object, Object> convertConstant) {\n    if (reader != null) {\n      return reader;\n    } else if (field.initialDefault() != null) {\n      Object value = convertConstant.apply(field.type(), field.initialDefault());\n      return constant(value, constantDefinitionLevel);\n    } else if (field.isOptional()) {\n      return nulls();\n    }\n\n    throw new IllegalArgumentException(String.format(\"Missing required field: %s\", field.name()));\n  }\n\n  /**\n   * Returns the column whose definition level shows whether the struct is present on each row, or\n   * null if a field reader already reads one or the struct can never be null.\n   */\n  private static ColumnDescriptor presenceColumn(\n      MessageType fileSchema,\n      String[] structPath,\n      int structDefinitionLevel,\n      List<Types.NestedField> expectedFields,\n      Map<Integer, ParquetValueReader<?>> readersById) {\n    boolean hasRealFieldReader =\n        expectedFields.stream().anyMatch(field -> readersById.containsKey(field.fieldId()));\n    if (hasRealFieldReader || structDefinitionLevel <= 0) {\n      return null;\n    }\n","sourceCodeStart":268,"sourceCodeEnd":304,"githubUrl":"https://github.com/apache/iceberg/blob/86d9c8fc543e7c56c9f624eb725f76c9baff9570/parquet/src/main/java/org/apache/iceberg/parquet/ParquetValueReaders.java#L268-L304","documentation":"Thrown by ParquetValueReaders.defaultReader() when a required (non-optional, non-repeating) struct field in the Parquet schema has no corresponding column reader: it is neither a constant field, an optional field, nor otherwise resolvable. The reader cannot produce values for a required field it has no source for, so it fails fast with the field name.","triggerScenarios":"Building a struct reader via ParquetValueReaders.structs(...) where the expected Iceberg struct type declares a required field but the Parquet file's projected schema omits it or provides it in a form defaultReader cannot map (not constant, not optional, not a primitive/group column).","commonSituations":"Schema evolution: a field was added to the Iceberg table as required after the file was written; projection/pruning dropping a required column; mismatch between the expected type and the file schema when reading files written by older writers.","solutions":["Ensure the required field is present in the Parquet file schema or included in the projection","Make the field optional in the table schema and backfill values (Iceberg requires adding columns as optional)","Use typed constant/default values so the reader maps the missing field to initialDefault"],"exampleFix":"// before: adding a required column to an existing table (old files lack it)\nschema = new Schema(Types.NestedField.required(1, \"new_col\", Types.IntegerType.get()));\n// after: add as optional, then backfill\nschema = new Schema(Types.NestedField.optional(1, \"new_col\", Types.IntegerType.get()));","handlingStrategy":"validation","validationCode":"for (Types.NestedField field : struct.fields()) {\n  if (field.isRequired() && !fileSchemaContainsField(field.name())) {\n    throw new IllegalArgumentException(\"File schema lacks required field: \" + field.name());\n  }\n}","typeGuard":null,"tryCatchPattern":"try {\n  reader = ParquetValueReaders.structs(...);\n} catch (IllegalArgumentException e) {\n  if (e.getMessage().startsWith(\"Missing required field\")) {\n    // fall back to projection including the missing column, or fail with a clear schema-evolution message\n  } else throw e;\n}","preventionTips":["Always add new table columns as optional, never required (Iceberg forbids it)","Include all required columns in the read projection","Backfill old files before enforcing required fields","Compare the file schema against the expected schema before opening readers"],"tags":["parquet","schema-evolution","required-field"],"backgroundTag":"missing-required-argument","analyzedSha":"86d9c8fc543e7c56c9f624eb725f76c9baff9570","analyzedAt":"2026-09-12T00:46:39.097Z","contentChangedAt":"2026-09-12T00:46:39.097Z","schemaVersion":2},"datasetVersion":"2026-09-23T08:17:48.524Z"}