{"record":{"id":"1db0f2293e08d907","repo":"apache/flink","slug":"parquet-with-case-insensitive-mode-should-have-no-1db0f2","errorCode":null,"errorMessage":"Parquet with case insensitive mode should have no duplicate key: {}","messagePattern":"Parquet with case insensitive mode should have no duplicate key: (.+?)","errorType":"exception","errorClass":"FlinkRuntimeException","httpStatus":null,"severity":"error","filePath":"flink-formats/flink-parquet/src/main/java/org/apache/flink/formats/parquet/vector/ParquetColumnarRowSplitReader.java","lineNumber":178,"sourceCode":"    private static MessageType clipParquetSchema(\n            GroupType parquetSchema, String[] fieldNames, boolean caseSensitive) {\n        Type[] types = new Type[fieldNames.length];\n        if (caseSensitive) {\n            for (int i = 0; i < fieldNames.length; ++i) {\n                String fieldName = fieldNames[i];\n                if (parquetSchema.getFieldIndex(fieldName) < 0) {\n                    throw new IllegalArgumentException(fieldName + \" does not exist\");\n                }\n                types[i] = parquetSchema.getType(fieldName);\n            }\n        } else {\n            Map<String, Type> caseInsensitiveFieldMap = new HashMap<>();\n            for (Type type : parquetSchema.getFields()) {\n                caseInsensitiveFieldMap.compute(\n                        type.getName().toLowerCase(Locale.ROOT),\n                        (key, previousType) -> {\n                            if (previousType != null) {\n                                throw new FlinkRuntimeException(\n                                        \"Parquet with case insensitive mode should have no duplicate key: \"\n                                                + key);\n                            }\n                            return type;\n                        });\n            }\n            for (int i = 0; i < fieldNames.length; ++i) {\n                Type type = caseInsensitiveFieldMap.get(fieldNames[i].toLowerCase(Locale.ROOT));\n                if (type == null) {\n                    throw new IllegalArgumentException(fieldNames[i] + \" does not exist\");\n                }\n                // TODO clip for array,map,row types.\n                types[i] = type;\n            }\n        }\n\n        return Types.buildMessage().addFields(types).named(\"flink-parquet\");\n    }","sourceCodeStart":160,"sourceCodeEnd":196,"githubUrl":"https://github.com/apache/flink/blob/2f3c205e9266cb30240eb7f4fdab15cad629a70f/flink-formats/flink-parquet/src/main/java/org/apache/flink/formats/parquet/vector/ParquetColumnarRowSplitReader.java#L160-L196","documentation":"Thrown by ParquetColumnarRowSplitReader when building the requested schema in case-insensitive mode (the default for Flink/Hive catalogs). The reader folds every field of the Parquet file schema into a map keyed by lowercased name; if two file columns collide (e.g. 'Id' and 'ID'), a FlinkRuntimeException is thrown because the reader cannot decide which physical column a query name refers to. This is detected purely from the file schema, before any data is read.","triggerScenarios":"Reading a Parquet file whose top-level schema contains two field names that are equal after toLowerCase(Locale.ROOT), while the reader was constructed with caseSensitive=false (ParquetColumnarRowSplitReader builds caseInsensitiveFieldMap and finds previousType != null).","commonSituations":"Tables written by tools that preserve mixed-case column names (Spark with spark.sql.caseSensitive=true, Avro-to-Parquet conversions, hand-crafted files); Hive tables defined case-insensitively over such files; merging data from sources with inconsistent column casing.","solutions":["Rewrite/rename the duplicate columns in the Parquet file so their lowercased names are unique","Set the table/format option to case-sensitive name matching (e.g. Flink Hive table with parquet column names matching exactly, or read via a projection that avoids ambiguity is NOT enough - the check runs over the whole file schema)","If the table is external, recreate the underlying data with normalized lower-case column names","As a last resort, read the file with a case-sensitive reader path (ParquetVectorReader with caseSensitive=true) where fieldNames[i] must match exactly"],"exampleFix":"// file schema: id: INT, ID: STRING  -> case-insensitive collision\n// fix: rewrite file with unique names\n// before (file):  message flink { required INT32 id; required BINARY ID (UTF8); }\n// after (file):   message flink { required INT32 id; required BINARY id_str (UTF8); }","handlingStrategy":"validation","validationCode":"MessageType fileSchema = ParquetFileReader.readFooter(conf, path).getFileMetaData().getSchema();\nSet<String> seen = new HashSet<>();\nfor (Type f : fileSchema.getFields()) {\n    if (!seen.add(f.getName().toLowerCase(Locale.ROOT))) {\n        throw new IllegalStateException(\"Case-insensitive duplicate column in file: \" + f.getName());\n    }\n}","typeGuard":null,"tryCatchPattern":"catch (FlinkRuntimeException e) { if (e.getMessage() != null && e.getMessage().contains(\"no duplicate key\")) { /* normalize/rename columns in file */ } else throw e; }","preventionTips":["Standardize all column names to lower case in upstream writers","Validate incoming Parquet files against the catalog schema in an ingestion test job before registering partitions"],"tags":["parquet","schema","case-insensitive","flink"],"backgroundTag":null,"analyzedSha":"2f3c205e9266cb30240eb7f4fdab15cad629a70f","analyzedAt":"2026-08-14T08:48:24.518Z","schemaVersion":2},"datasetVersion":"2026-08-14T10:17:34.591Z"}