apache/flink · error · IllegalArgumentException

{} does not exist

Error message

{} does not exist

What it means

ParquetColumnarRowSplitReader clips the parquet schema to the requested field names. In case-sensitive mode, if a requested field name has no match in the file's parquet schema (getFieldIndex < 0), it throws IllegalArgumentException(fieldName + ' does not exist').

Source

Thrown at flink-formats/flink-parquet/src/main/java/org/apache/flink/formats/parquet/vector/ParquetColumnarRowSplitReader.java:167

        this.writableVectors = createWritableVectors();
        this.columnarBatch = generator.generate(createReadableVectors());
        this.row = new ColumnarRowData(columnarBatch);

        MessageColumnIO columnIO = new ColumnIOFactory().getColumnIO(requestedSchema);
        RowType selectedType = RowType.of(selectedTypes, selectedFieldNames);
        this.fieldList =
                buildFieldsList(selectedType.getFields(), selectedType.getFieldNames(), columnIO);
    }

    /** Clips `parquetSchema` according to `fieldNames`. */
    private static MessageType clipParquetSchema(
            GroupType parquetSchema, String[] fieldNames, boolean caseSensitive) {
        Type[] types = new Type[fieldNames.length];
        if (caseSensitive) {
            for (int i = 0; i < fieldNames.length; ++i) {
                String fieldName = fieldNames[i];
                if (parquetSchema.getFieldIndex(fieldName) < 0) {
                    throw new IllegalArgumentException(fieldName + " does not exist");
                }
                types[i] = parquetSchema.getType(fieldName);
            }
        } else {
            Map<String, Type> caseInsensitiveFieldMap = new HashMap<>();
            for (Type type : parquetSchema.getFields()) {
                caseInsensitiveFieldMap.compute(
                        type.getName().toLowerCase(Locale.ROOT),
                        (key, previousType) -> {
                            if (previousType != null) {
                                throw new FlinkRuntimeException(
                                        "Parquet with case insensitive mode should have no duplicate key: "
                                                + key);
                            }
                            return type;
                        });
            }
            for (int i = 0; i < fieldNames.length; ++i) {

View on GitHub (pinned to 2f3c205e92)

Solutions

1. Align the requested field names with the parquet file's actual column names (check with parquet-tools/parquet-schema)
2. If only casing differs, enable case-insensitive mode ('table.exec.sink.upsert-insert-extract' style options aside: use the reader's caseSensitive=false path / corresponding connector option)
3. Add missing columns to the files or remove them from the projection; use schema evolution support in the catalog

Example fix

-- before
SELECT userId FROM parquet_table; -- file column is `UserId`

-- after
SELECT UserId FROM parquet_table;
-- or configure case-insensitive reading if the connector supports it

for (String f : requestedFields) {
  if (parquetSchema.getFieldIndex(f) < 0) throw new IllegalArgumentException("field '" + f + "' missing in " + parquetSchema.getName());
}

try { reader = new ParquetColumnarRowSplitReader(...); } catch (IllegalArgumentException e) { if (e.getMessage().endsWith("does not exist")) { /* reconcile projection with file schema */ } throw e; }

When it happens

Trigger: Reading with a projected schema whose field names don't match the parquet file - e.g. case mismatch (userId vs UserId), renamed columns, or a DDL/table schema referencing columns absent from the file.

Common situations: Schema evolution where the table schema has newer names than old files; case-sensitive flag set while files use different casing; typos in the SELECT/projection list; mixing files from different writers.

Related errors

• Error while waiting for job to be initialized
• Translator {} cannot translate the given pipeline {}.
• Could not get job jar and dependencies from JAR file: {}
• Could not build the program from JAR file: {}
• Failed to retrieve job list.

AI-assisted analysis of apache/flink@2f3c205e92 (2026-08-14). Data as JSON: /api/errors/f257b3185840862d. Report an issue: GitHub.