apache/iceberg · error

Provided schema:...Problems: (IllegalArgumentException with

Error message

Provided schema:...Problems: (IllegalArgumentException with schema and error list)

What it means

TypeUtil.checkSchemaCompatibility verifies a provided schema is compatible with an expected schema and collects all mismatches. If any errors are found it throws an IllegalArgumentException whose message embeds the full provided schema and a bulleted list of problems. It is the gate behind validateWriteSchema and validateSchema.

Source

Thrown at api/src/main/java/org/apache/iceberg/types/TypeUtil.java:562

    } else {
      errors = CheckCompatibility.typeCompatibilityErrors(schema, providedSchema, checkOrdering);
    }

    if (!errors.isEmpty()) {
      StringBuilder sb = new StringBuilder();
      sb.append(errMsg)
          .append("\n")
          .append(schema)
          .append("\n")
          .append("Provided schema:")
          .append("\n")
          .append(providedSchema)
          .append("\n")
          .append("Problems:");
      for (String error : errors) {
        sb.append("\n* ").append(error);
      }
      throw new IllegalArgumentException(sb.toString());
    }
  }

  /**
   * Estimates the number of bytes a value for a given field may occupy in memory.
   *
   * <p>This method approximates the memory size based on heuristics and the internal Java
   * representation defined by {@link Type.TypeID}. It is important to note that the actual size
   * might differ from this estimation. The method is designed to handle a variety of data types,
   * including primitive types, strings, and nested types such as structs, maps, and lists.
   *
   * @param field a field for which to estimate the size
   * @return the estimated size in bytes of the field's value in memory
   */
  public static int estimateSize(Types.NestedField field) {
    return estimateSize(field.type());
  }

View on GitHub (pinned to 86d9c8fc54)

Solutions

1. Read the 'Problems:' list in the message and fix each listed field/type/nullability mismatch in the provided schema.
2. Cast or convert the incoming data to match the expected schema before writing.
3. If extra columns are expected to be tolerated, project the provided schema to the expected one before validation.
4. Relax checks only deliberately: set checkNullability=false if optional-vs-required differences are acceptable.

Example fix

// before
table.validateWriteSchema(expected, provided, true, true); // fails on int vs long
// after
Types.StructType cast = TypeUtil.join(provided.asStruct(), expected.asStruct());
DataFrame fixed = data.select(col("id").cast("long"), col("data"));
table.validateWriteSchema(expected, fixed.schema(), true, true);

List<String> mismatches = new ArrayList<>();
for (Types.NestedField expectedField : expected.asStruct().fields()) {
  Types.NestedField providedField = provided.asStruct().field(expectedField.name());
  if (providedField == null) mismatches.add("missing field: " + expectedField.name());
  else if (!providedField.type().equals(expectedField.type())) mismatches.add("type mismatch: " + expectedField.name());
}
if (!mismatches.isEmpty()) { /* reconcile schema before writing */ }

boolean compatible = expected.asStruct().fields().stream()
    .allMatch(f -> {
      Types.NestedField pf = provided.asStruct().field(f.name());
      return pf != null && pf.type().equals(f.type());
    });

try {
  Schema.validateWriteSchema(expected, provided, true, true);
} catch (IllegalArgumentException e) {
  // log e.getMessage(); reconcile/cast data schema before retry
}

When it happens

Trigger: Calling Schema.validateWriteSchema(expected, provided, checkNullability, checkCompatibility) or TypeUtil.validateSchema when the provided schema is missing fields, has incompatible types, or (with checkNullability) fields that are optional where required.

Common situations: Writing a DataFrame/dataset whose columns don't match the Iceberg table schema (missing column, wrong type like string vs long, nullability differences); evolving writer code after the table schema changed; case-sensitivity mismatches in field names.

Understand the failure class

Background: Schema validation failed / invalid input schema: payload rejected because its shape doesn't match the expected schema — this error's family across 28 libraries.

Related errors

• Unsorted order ID must be 0
• Field + field.name() + not found in source schema
• Replaced and created manifests must have the same number of
• Cannot delete element type from list:
• Cannot delete map keys:

AI-assisted analysis of apache/iceberg@86d9c8fc54 (2026-09-12). Data as JSON: /api/errors/b2ab09907781a107. Report an issue: GitHub.