apache/flink · error · UnsupportedOperationException

Schema evolution not supported.

Error message

Schema evolution not supported.

What it means

UnsupportedOperationException from checkSchema() in ParquetColumnarRowSplitReader. For every projected column path that exists in the file schema, the reader compares the file's ColumnDescriptor with the requested one; any difference in primitive type, logical type annotation, or nullability (repetition) is rejected because the vectorized reader performs no schema evolution. The column must match byte-for-byte at the Parquet type level.

Source

Thrown at flink-formats/flink-parquet/src/main/java/org/apache/flink/formats/parquet/vector/ParquetColumnarRowSplitReader.java:242

        }
        return vectors;
    }

    private void checkSchema() throws IOException, UnsupportedOperationException {
        if (selectedTypes.length != requestedSchema.getFieldCount()) {
            throw new RuntimeException(
                    "The quality of field type is incompatible with the request schema!");
        }

        /*
         * Check that the requested schema is supported.
         */
        for (int i = 0; i < requestedSchema.getFieldCount(); ++i) {
            String[] colPath = requestedSchema.getPaths().get(i);
            if (fileSchema.containsPath(colPath)) {
                ColumnDescriptor fd = fileSchema.getColumnDescription(colPath);
                if (!fd.equals(requestedSchema.getColumns().get(i))) {
                    throw new UnsupportedOperationException("Schema evolution not supported.");
                }
            } else {
                if (requestedSchema.getColumns().get(i).getMaxDefinitionLevel() == 0) {
                    // Column is missing in data but the required data is non-nullable. This file is
                    // invalid.
                    throw new IOException(
                            "Required column is missing in data file. Col: "
                                    + Arrays.toString(colPath));
                }
            }
        }
    }

    /**
     * Method used to check if the end of the input is reached.
     *
     * @return True if the end is reached, otherwise false.
     * @throws IOException Thrown, if an I/O error occurred.

View on GitHub (pinned to 2f3c205e92)

Solutions

1. Align the Flink/Hive table column type exactly with the physical Parquet type (inspect with parquet-tools 'parquet schema file.parquet')
2. If files with multiple type versions must be read, partition them and register separate tables, or rewrite the old files to the new type
3. For timestamps, match the writer's unit (MILLIS/MICROS/NANOS logical annotation) in the table definition
4. Upgrade Flink only after checking release notes - this path intentionally does not evolve schemas

Example fix

-- before: table says BIGINT, file has INT32
CREATE TABLE t (k INT, v BIGINT) WITH ('format'='parquet');
-- after: match the physical type
CREATE TABLE t (k INT, v INT) WITH ('format'='parquet');

MessageType fileSchema = ParquetFileReader.readFooter(conf, path).getFileMetaData().getSchema();
MessageType requested = ParquetSchemaConverter.convertToParquetMessageType("flink-parquet", rowType);
for (int i = 0; i < requested.getFieldCount(); i++) {
    String[] colPath = requested.getPaths().get(i);
    if (fileSchema.containsPath(colPath)
            && !fileSchema.getColumnDescription(colPath).equals(requested.getColumns().get(i))) {
        throw new IllegalStateException("Type mismatch for column " + Arrays.toString(colPath));
    }
}

catch (UnsupportedOperationException e) { if ("Schema evolution not supported.".equals(e.getMessage())) { /* align types or rewrite file */ } else throw e; }

When it happens

Trigger: Reading a column whose Parquet ColumnDescriptor in the file differs from the requested schema's descriptor - e.g. file has INT32 but table declares BIGINT, file has INT32/timestamp millis but table expects timestamp micros, or optional vs required repetition mismatch.

Common situations: Hive/Spark schema evolution where a column type was changed after files were written; tables created with a手动 DDL that guesses types differently from the writer; timestamp precision (MILLIS vs MICROS) or decimal precision/scale mismatches between writer and reader.

Related errors

• Required column is missing in data file. Col: {}
• Could not stop with a savepoint job "{}".
• Could not stop with a detached savepoint job "{}".
• Failed to trigger a savepoint for the job {}.
• JAR file is not a file: {}

AI-assisted analysis of apache/flink@2f3c205e92 (2026-08-14). Data as JSON: /api/errors/7c10ad95ab5e7dff. Report an issue: GitHub.