{"record":{"id":"02869250b883eb45","repo":"apache/flink","slug":"corrupted-parquet-schema","errorCode":null,"errorMessage":"Corrupted Parquet schema","messagePattern":"Corrupted Parquet schema","errorType":"validation","errorClass":"InvalidSchemaException","httpStatus":null,"severity":"error","filePath":"flink-formats/flink-parquet/src/main/java/org/apache/flink/formats/parquet/vector/ParquetSplitReaderUtil.java","lineNumber":293,"sourceCode":"            case TIMESTAMP_WITHOUT_TIME_ZONE:\n                HeapTimestampVector tv = new HeapTimestampVector(batchSize);\n                if (value == null) {\n                    tv.fillWithNulls();\n                } else {\n                    tv.fill(TimestampData.fromLocalDateTime((LocalDateTime) value));\n                }\n                return tv;\n            default:\n                throw new UnsupportedOperationException(\"Unsupported type: \" + type);\n        }\n    }\n\n    private static List<ColumnDescriptor> getAllColumnDescriptorByType(\n            int depth, Type type, List<ColumnDescriptor> columns) throws ParquetRuntimeException {\n        List<ColumnDescriptor> res = new ArrayList<>();\n        for (ColumnDescriptor descriptor : columns) {\n            if (depth >= descriptor.getPath().length) {\n                throw new InvalidSchemaException(\"Corrupted Parquet schema\");\n            }\n            if (type.getName().equals(descriptor.getPath()[depth])) {\n                res.add(descriptor);\n            }\n        }\n\n        // If doesn't find the type descriptor in corresponding depth, throw exception\n        if (res.isEmpty()) {\n            throw new InvalidSchemaException(\n                    \"Failed to find related Parquet column descriptor with type \" + type);\n        }\n        return res;\n    }\n\n    public static ColumnReader createColumnReader(\n            boolean isUtcTimestamp,\n            LogicalType fieldType,\n            Type type,","sourceCodeStart":275,"sourceCodeEnd":311,"githubUrl":"https://github.com/apache/flink/blob/2f3c205e9266cb30240eb7f4fdab15cad629a70f/flink-formats/flink-parquet/src/main/java/org/apache/flink/formats/parquet/vector/ParquetSplitReaderUtil.java#L275-L311","documentation":"InvalidSchemaException from getAllColumnDescriptorByType in ParquetSplitReaderUtil. While walking the requested schema tree against the flattened list of Parquet ColumnDescriptors, the code indexes descriptor.getPath()[depth]; if depth is at or beyond a descriptor's path length, the requested schema is deeper than the physical column paths, which only happens with inconsistent or corrupt schema mapping.","triggerScenarios":"getAllColumnDescriptorByType(depth, type, columns) called with a depth >= some descriptor.getPath().length - i.e. the requested nested schema descends into a level where the file's column paths end (a primitive column where a group was expected).","commonSituations":"Table schema declares nesting (row/map/array) where the Parquet file has a flat primitive column with the same name; mismatched files behind one table definition; partially corrupted schema metadata.","solutions":["Dump both schemas (parquet-tools schema vs table DDL) and align the nesting structure exactly","Point the table at files whose schema matches, or rewrite the files with the expected nested structure","If types were changed by upstream evolution, rewrite old files to the current schema"],"exampleFix":null,"handlingStrategy":"validation","validationCode":"// assert every requested nested path exists in the file's flattened column paths\nSet<List<String>> filePaths = fileSchema.getPaths().stream().map(Arrays::asList).collect(Collectors.toSet());\nfor (int i = 0; i < requested.getFieldCount(); i++) {\n    if (!filePaths.contains(Arrays.asList(requested.getPaths().get(i)))) {\n        throw new IllegalStateException(\"Requested path absent in file: \" + Arrays.toString(requested.getPaths().get(i)));\n    }\n}","typeGuard":null,"tryCatchPattern":"catch (ParquetRuntimeException e) { if (\"Corrupted Parquet schema\".equals(e.getMessage())) { /* quarantine file, alert on schema drift */ } else throw e; }","preventionTips":["Make table nesting match file nesting exactly - verify with parquet-tools schema","Reject schema-drifted files at ingestion time"],"tags":["parquet","schema","corruption","nested-types","flink"],"backgroundTag":null,"analyzedSha":"2f3c205e9266cb30240eb7f4fdab15cad629a70f","analyzedAt":"2026-08-14T08:48:24.518Z","schemaVersion":2},"datasetVersion":"2026-08-14T10:17:34.591Z"}