apache/iceberg · error · IllegalArgumentException

Cannot find projected field:

Error message

Cannot find projected field:

What it means

Thrown by SupportsIndexProjection when building an index-based projection mapping: a projected field from the requested schema cannot be located in the source Avro schema fields by field ID. The library cannot construct the position array used to project values, so record reading cannot proceed.

Solutions

  1. Ensure all Avro schema fields carry the correct 'field-id' (iceberg-field-id) attribute matching the Iceberg schema
  2. Verify the data file schema is in sync with the current table schema; rewrite/repair stale files
  3. Make newly added fields optional so missing values can be filled with null instead of failing projection
  4. Regenerate the read schema from the table schema via AvroSchemaUtil rather than hand-editing

Example fix

// before
Schema readSchema = new Schema(Optional.of("t"), required(5, "notes", StringType.get()));
// after
Schema readSchema = new Schema(Optional.of("t"), optional(5, "notes", StringType.get()));
Defensive patterns

Strategy: validation

Validate before calling

Set<Integer> ids = table.schema().columns().stream().map(Types.NestedField::fieldId).collect(Collectors.toSet());
Schema fileSchema = AvroSchemaUtil.readSchema(file).orElseThrow();
for (Types.NestedField f : readSchema.columns()) {
  Object id = fileSchema.asStruct().field(f.name()) != null ? f.fieldId() : null;
  if (id == null || !ids.contains(f.fieldId())) throw new IllegalStateException("field-id missing for " + f.name());
}

Type guard

boolean hasField(Schema schema, Types.NestedField f) { return Arrays.stream(schema.getFields().toArray(new Field[0])).anyMatch(x -> x.getProp(AvroSchemaUtil.FIELD_ID_PROP) != null); }

Try / catch

try (CloseableIterable<Record> rows = Avro.read(file).project(table.schema()).build()) { rows.forEach(...); } catch (IllegalArgumentException e) { if (e.getMessage().startsWith("Cannot find projected field")) { /* resync schema / rewrite files */ } else throw e; }

Prevention

When it happens

Trigger: Reading an Avro data or manifest file whose schema lacks a field that the table schema (or a projection of it) declares; typically a stale data file written by an older schema after the table added a required field.

Common situations: Schema evolution (field added as required) combined with old data files; custom Avro schemas missing field IDs; manually constructed read schemas where field-id properties were stripped by tools.

Understand the failure class

Background: Schema validation failed / invalid input schema: payload rejected because its shape doesn't match the expected schema — this error's family across 28 libraries.

Related errors

AI-assisted analysis of apache/iceberg@86d9c8fc54 (2026-09-12). Data as JSON: /api/errors/1fa6b288cd87ac69. Report an issue: GitHub.

Appendix: source

Thrown at core/src/main/java/org/apache/iceberg/avro/SupportsIndexProjection.java:53

  }

  /** Base constructor for building the type mapping */
  protected SupportsIndexProjection(Types.StructType baseType, Types.StructType projectionType) {
    List<Types.NestedField> allFields = baseType.fields();
    List<Types.NestedField> fields = projectionType.fields();

    this.fromProjectionPos = new int[fields.size()];
    for (int i = 0; i < fromProjectionPos.length; i += 1) {
      boolean found = false;
      for (int j = 0; j < allFields.size(); j += 1) {
        if (fields.get(i).fieldId() == allFields.get(j).fieldId()) {
          found = true;
          fromProjectionPos[i] = j;
        }
      }

      if (!found) {
        throw new IllegalArgumentException("Cannot find projected field: " + fields.get(i));
      }
    }
  }

  /** Constructor with a precomputed position mapping */
  protected SupportsIndexProjection(int[] fromProjectionPos) {
    this.fromProjectionPos = fromProjectionPos;
  }

  /** Copy constructor */
  protected SupportsIndexProjection(SupportsIndexProjection toCopy) {
    this.fromProjectionPos = toCopy.fromProjectionPos;
  }

  protected abstract <T> T internalGet(int pos, Class<T> javaClass);

  protected abstract <T> void internalSet(int pos, T value);

View on GitHub (pinned to 86d9c8fc54)