{"record":{"id":"271b54ba57440eaf","repo":"apache/flink","slug":"required-column-is-missing-in-data-file-col-271b54","errorCode":null,"errorMessage":"Required column is missing in data file. Col: {}","messagePattern":"Required column is missing in data file\\. Col: (.+?)","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"flink-formats/flink-parquet/src/main/java/org/apache/flink/formats/parquet/vector/ParquetColumnarRowSplitReader.java","lineNumber":248,"sourceCode":"            throw new RuntimeException(\n                    \"The quality of field type is incompatible with the request schema!\");\n        }\n\n        /*\n         * Check that the requested schema is supported.\n         */\n        for (int i = 0; i < requestedSchema.getFieldCount(); ++i) {\n            String[] colPath = requestedSchema.getPaths().get(i);\n            if (fileSchema.containsPath(colPath)) {\n                ColumnDescriptor fd = fileSchema.getColumnDescription(colPath);\n                if (!fd.equals(requestedSchema.getColumns().get(i))) {\n                    throw new UnsupportedOperationException(\"Schema evolution not supported.\");\n                }\n            } else {\n                if (requestedSchema.getColumns().get(i).getMaxDefinitionLevel() == 0) {\n                    // Column is missing in data but the required data is non-nullable. This file is\n                    // invalid.\n                    throw new IOException(\n                            \"Required column is missing in data file. Col: \"\n                                    + Arrays.toString(colPath));\n                }\n            }\n        }\n    }\n\n    /**\n     * Method used to check if the end of the input is reached.\n     *\n     * @return True if the end is reached, otherwise false.\n     * @throws IOException Thrown, if an I/O error occurred.\n     */\n    public boolean reachedEnd() throws IOException {\n        return !ensureBatch();\n    }\n\n    public ColumnarRowData nextRecord() {","sourceCodeStart":230,"sourceCodeEnd":266,"githubUrl":"https://github.com/apache/flink/blob/2f3c205e9266cb30240eb7f4fdab15cad629a70f/flink-formats/flink-parquet/src/main/java/org/apache/flink/formats/parquet/vector/ParquetColumnarRowSplitReader.java#L230-L266","documentation":"IOException from checkSchema() in ParquetColumnarRowSplitReader: a requested column path is absent from the file schema AND its requested ColumnDescriptor has maxDefinitionLevel == 0, meaning the column is declared non-nullable (REQUIRED). A missing nullable column can be padded with nulls, but a missing required column makes the file invalid for the requested schema, so reading is refused up front.","triggerScenarios":"Projecting a NOT NULL / required column that does not exist in the Parquet file being read (fileSchema.containsPath(colPath) is false and requestedSchema.getColumns().get(i).getMaxDefinitionLevel() == 0).","commonSituations":"Schema evolution that ADDED a required column after older files were written; Hive tables where new partitions have the column but old ones do not; table DDL marking a column NOT NULL while source files never contained it.","solutions":["Make the column nullable in the table/row type so maxDefinitionLevel > 0 and the reader can emit nulls for old files","Backfill or rewrite the old files so they contain the required column","Exclude the missing column from the read projection if it is not needed"],"exampleFix":"-- before: new required column absent in old files\nCREATE TABLE t (k INT, new_col STRING NOT NULL) ...;\n-- after: allow nulls so missing column reads as NULL\nCREATE TABLE t (k INT, new_col STRING) ...;","handlingStrategy":"validation","validationCode":"MessageType fileSchema = ParquetFileReader.readFooter(conf, path).getFileMetaData().getSchema();\nfor (int i = 0; i < requested.getFieldCount(); i++) {\n    String[] colPath = requested.getPaths().get(i);\n    if (!fileSchema.containsPath(colPath) && requested.getColumns().get(i).getMaxDefinitionLevel() == 0) {\n        throw new IllegalStateException(\"Required column missing in file: \" + Arrays.toString(colPath) + \" - make it nullable or backfill\");\n    }\n}","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Prefer nullable columns when adding new fields to evolving Parquet tables","Backfill old files when introducing NOT NULL columns"],"tags":["parquet","schema-evolution","required-column","flink"],"backgroundTag":null,"analyzedSha":"2f3c205e9266cb30240eb7f4fdab15cad629a70f","analyzedAt":"2026-08-14T08:48:24.518Z","schemaVersion":2},"datasetVersion":"2026-08-14T10:17:34.591Z"}