apache/flink · error · IOException

Required column is missing in data file. Col: {}

Error message

Required column is missing in data file. Col: {}

What it means

IOException from checkSchema() in ParquetColumnarRowSplitReader: a requested column path is absent from the file schema AND its requested ColumnDescriptor has maxDefinitionLevel == 0, meaning the column is declared non-nullable (REQUIRED). A missing nullable column can be padded with nulls, but a missing required column makes the file invalid for the requested schema, so reading is refused up front.

Source

Thrown at flink-formats/flink-parquet/src/main/java/org/apache/flink/formats/parquet/vector/ParquetColumnarRowSplitReader.java:248

            throw new RuntimeException(
                    "The quality of field type is incompatible with the request schema!");
        }

        /*
         * Check that the requested schema is supported.
         */
        for (int i = 0; i < requestedSchema.getFieldCount(); ++i) {
            String[] colPath = requestedSchema.getPaths().get(i);
            if (fileSchema.containsPath(colPath)) {
                ColumnDescriptor fd = fileSchema.getColumnDescription(colPath);
                if (!fd.equals(requestedSchema.getColumns().get(i))) {
                    throw new UnsupportedOperationException("Schema evolution not supported.");
                }
            } else {
                if (requestedSchema.getColumns().get(i).getMaxDefinitionLevel() == 0) {
                    // Column is missing in data but the required data is non-nullable. This file is
                    // invalid.
                    throw new IOException(
                            "Required column is missing in data file. Col: "
                                    + Arrays.toString(colPath));
                }
            }
        }
    }

    /**
     * Method used to check if the end of the input is reached.
     *
     * @return True if the end is reached, otherwise false.
     * @throws IOException Thrown, if an I/O error occurred.
     */
    public boolean reachedEnd() throws IOException {
        return !ensureBatch();
    }

    public ColumnarRowData nextRecord() {

View on GitHub (pinned to 2f3c205e92)

Solutions

1. Make the column nullable in the table/row type so maxDefinitionLevel > 0 and the reader can emit nulls for old files
2. Backfill or rewrite the old files so they contain the required column
3. Exclude the missing column from the read projection if it is not needed

Example fix

-- before: new required column absent in old files
CREATE TABLE t (k INT, new_col STRING NOT NULL) ...;
-- after: allow nulls so missing column reads as NULL
CREATE TABLE t (k INT, new_col STRING) ...;

MessageType fileSchema = ParquetFileReader.readFooter(conf, path).getFileMetaData().getSchema();
for (int i = 0; i < requested.getFieldCount(); i++) {
    String[] colPath = requested.getPaths().get(i);
    if (!fileSchema.containsPath(colPath) && requested.getColumns().get(i).getMaxDefinitionLevel() == 0) {
        throw new IllegalStateException("Required column missing in file: " + Arrays.toString(colPath) + " - make it nullable or backfill");
    }
}

When it happens

Trigger: Projecting a NOT NULL / required column that does not exist in the Parquet file being read (fileSchema.containsPath(colPath) is false and requestedSchema.getColumns().get(i).getMaxDefinitionLevel() == 0).

Common situations: Schema evolution that ADDED a required column after older files were written; Hive tables where new partitions have the column but old ones do not; table DDL marking a column NOT NULL while source files never contained it.

Related errors

• Schema evolution not supported.
• Failed to deserialize Avro record.
• Could not register schema in registry
• Only support seek at first.
• A stream against this file was already created.

AI-assisted analysis of apache/flink@2f3c205e92 (2026-08-14). Data as JSON: /api/errors/271b54ba57440eaf. Report an issue: GitHub.