What causes "Invalid primary key '%s'. Column '%s' does not exist." in apache/iceberg?

Align the PRIMARY KEY column names with the actual columns in the table definition.

apache/iceberg · error · org.apache.flink.table.api.ValidationException

Invalid primary key '%s'. Column '%s' does not exist.

Error message

Invalid primary key '%s'. Column '%s' does not exist.

What it means

validatePrimaryKey checks each primary-key column against the table's column lookup; if the named column is absent from the schema, a ValidationException is thrown. The primary key must reference existing (by-name) columns of the Flink schema being converted.

Source

Thrown at flink/v1.20/flink/src/main/java/org/apache/iceberg/flink/FlinkSchemaUtil.java:358

    final Map<String, Column> columnsByNameLookup =
        columns.stream().collect(Collectors.toMap(Column::getName, Function.identity()));

    final Set<String> duplicateColumns =
        primaryKey.getColumns().stream()
            .filter(name -> Collections.frequency(primaryKey.getColumns(), name) > 1)
            .collect(Collectors.toSet());

    if (!duplicateColumns.isEmpty()) {
      throw new ValidationException(
          String.format(
              "Invalid primary key '%s'. A primary key must not contain duplicate columns. Found: %s",
              primaryKey.getName(), duplicateColumns));
    }

    for (String columnName : primaryKey.getColumns()) {
      Column column = columnsByNameLookup.get(columnName);
      if (column == null) {
        throw new ValidationException(
            String.format(
                "Invalid primary key '%s'. Column '%s' does not exist.",
                primaryKey.getName(), columnName));
      }

      if (!column.isPhysical()) {
        throw new ValidationException(
            String.format(
                "Invalid primary key '%s'. Column '%s' is not a physical column.",
                primaryKey.getName(), columnName));
      }

      final LogicalType columnType = column.getDataType().getLogicalType();
      if (columnType.isNullable()) {
        throw new ValidationException(
            String.format(
                "Invalid primary key '%s'. Column '%s' is nullable.",
                primaryKey.getName(), columnName));

View on GitHub (pinned to 86d9c8fc54)

Solutions

Align the PRIMARY KEY column names with the actual columns in the table definition.
Update the key after schema changes (ALTER that renames/drops columns).
Validate programmatically: check each key column exists in the schema's columns before constructing the ResolvedSchema.

Example fix

// before
CREATE TABLE t (id BIGINT, data STRING, PRIMARY KEY (uid) NOT ENFORCED);
// after
CREATE TABLE t (id BIGINT, data STRING, PRIMARY KEY (id) NOT ENFORCED);

Defensive patterns

Strategy: validation

Validate before calling

for (String col : primaryKey.getColumns()) {
  if (schema.getColumn(col) == null) {
    throw new IllegalArgumentException("PK column missing in schema: " + col);
  }
}

Type guard

null

Try / catch

try {
  FlinkSchemaUtil.toResolvedSchema(schema, partitionKeys, primaryKey);
} catch (ValidationException e) {
  LOG.error("PK references unknown column: {}", e.getMessage());
  throw e;
}

Prevention

Update PRIMARY KEY clauses whenever columns are renamed or dropped.
Validate key columns against the schema in tests after every schema change.
Avoid copy-pasting DDL between tables without rechecking key columns.

When it happens

Trigger: PRIMARY KEY referencing a column that was renamed, dropped, or never defined in the CREATE TABLE; mismatch between the constraint built in code and the columns passed to toResolvedSchema.

Common situations: Schema evolution changed column names while the key clause stayed stale; copying DDL between tables; typo in the key column name.

Understand the failure class

Background: Schema validation failed / invalid input schema: payload rejected because its shape doesn't match the expected schema — this error's family across 28 libraries.

Related errors

AI-assisted analysis of apache/iceberg@86d9c8fc54 (2026-09-12). Data as JSON: /api/errors/abab7b021d33f2ae. Report an issue: GitHub.