apache/seatunnel · critical · IllegalStateException

Schema evolution: partition column [" + col + "] is missing…

Error message

Schema evolution: partition column [" + col + "] is missing from the post-ALTER schema. Dropping or renaming a partition column is not supported. Available columns: " + java.util.Arrays.toString(fieldNames)

What it means

After applying a schema change, the write strategy rebuilds the mapping from partition columns to their indices in the new schema. If a partition column no longer exists in the post-ALTER schema (it was dropped or renamed), the rebuild fails with IllegalStateException. Dropping or renaming a partition column would corrupt the output directory layout, so it is explicitly rejected.

Solutions

1. Restore the dropped/renamed partition column (re-add it or rename it back) on the source table
2. Update the sink's partition_by config to use a column that survives the ALTER, and restart the job
3. If the partition column must be dropped, stop the pipeline, remove it from partition_by, and resubmit the job

Example fix

// before
sink {
  LocalFile {
    partition_by = ["dept_id"]   # dept_id gets dropped upstream
    schema_evolution_enabled = true
  }
}
// after
sink {
  LocalFile {
    partition_by = ["country"]   # stable column that is not dropped
    schema_evolution_enabled = true
  }
}

// before enabling schema evolution on a partitioned file sink
List<String> partitionBy = sinkConfig.partitionBy;
boolean allPresent = partitionBy.stream()
    .allMatch(col -> Arrays.stream(catalogTable.getTableSchema().toPhysicalRowDataType().getFieldNames())
        .anyMatch(f -> f.equalsIgnoreCase(col)));
if (!allPresent) throw new IllegalStateException("partition_by columns must exist and must never be dropped/renamed upstream");

try { sink.applySchemaChange(event); } catch (IllegalStateException e) { log.error("Partition column removed by ALTER: {}", e.getMessage()); /* restore column or repartition and restart */ }

When it happens

Trigger: applySchemaChange on the file sink processes an ALTER that DROPPED or RENAMED a column which is listed in the sink's partition_by config; rebuildPartitionFieldsIndex then cannot find that column in the new fieldNames.

Common situations: A DBA renames or drops a partitioned column on the CDC source table while a schema-evolution-enabled file sink is running; the sink partition_by was configured against an older table schema that no longer matches.

Understand the failure class

Background: Schema validation failed / invalid input schema: payload rejected because its shape doesn't match the expected schema — this error's family across 28 libraries.

Related errors

• Received AlterTableEvent but schema_evolution_enabled=false…
• DebeziumJson file does not support this compress type
• Failed to resolve schemaChangeEvent, just skip.
• [FileSchemaEvolution] DROP event references unknown column
• FORMAT_NOT_SUPPORT

AI-assisted analysis of apache/seatunnel@cf67b549a7 (2026-09-10). Data as JSON: /api/errors/c2da93d3f2197f03. Report an issue: GitHub.

Appendix: source

Thrown at seatunnel-connectors-v2/connector-file/connector-file-base/src/main/java/org/apache/seatunnel/connectors/seatunnel/file/sink/writer/AbstractWriteStrategy.java:441

     * <p>Throws if a configured partition column has been dropped from the schema — a dropped
     * partition key would corrupt the partition tree, so we fail fast rather than silently writing
     * to the wrong directory.
     */
    private List<Integer> rebuildPartitionFieldsIndex() {
        List<String> partitionFieldList = fileSinkConfig.getPartitionFieldList();
        if (partitionFieldList == null || partitionFieldList.isEmpty()) {
            return new ArrayList<>();
        }
        String[] fieldNames = seaTunnelRowType.getFieldNames();
        Map<String, Integer> nameToIndex = new HashMap<>(fieldNames.length);
        for (int i = 0; i < fieldNames.length; i++) {
            nameToIndex.put(fieldNames[i].toLowerCase(), i);
        }
        List<Integer> newIndex = new ArrayList<>(partitionFieldList.size());
        for (String col : partitionFieldList) {
            Integer idx = nameToIndex.get(col.toLowerCase());
            if (idx == null) {
                throw new IllegalStateException(
                        "Schema evolution: partition column ["
                                + col
                                + "] is missing from the post-ALTER schema. "
                                + "Dropping or renaming a partition column is not supported. "
                                + "Available columns: "
                                + java.util.Arrays.toString(fieldNames));
            }
            newIndex.add(idx);
        }
        return newIndex;
    }

    /**
     * use seaTunnelRow generate partition directory
     *
     * @param seaTunnelRow seaTunnelRow
     * @return the map of partition directory
     */

View on GitHub (pinned to cf67b549a7)