apache/flink · error · InvalidSchemaException

Corrupted Parquet schema

Error message

Corrupted Parquet schema

What it means

InvalidSchemaException from getAllColumnDescriptorByType in ParquetSplitReaderUtil. While walking the requested schema tree against the flattened list of Parquet ColumnDescriptors, the code indexes descriptor.getPath()[depth]; if depth is at or beyond a descriptor's path length, the requested schema is deeper than the physical column paths, which only happens with inconsistent or corrupt schema mapping.

Source

Thrown at flink-formats/flink-parquet/src/main/java/org/apache/flink/formats/parquet/vector/ParquetSplitReaderUtil.java:293

            case TIMESTAMP_WITHOUT_TIME_ZONE:
                HeapTimestampVector tv = new HeapTimestampVector(batchSize);
                if (value == null) {
                    tv.fillWithNulls();
                } else {
                    tv.fill(TimestampData.fromLocalDateTime((LocalDateTime) value));
                }
                return tv;
            default:
                throw new UnsupportedOperationException("Unsupported type: " + type);
        }
    }

    private static List<ColumnDescriptor> getAllColumnDescriptorByType(
            int depth, Type type, List<ColumnDescriptor> columns) throws ParquetRuntimeException {
        List<ColumnDescriptor> res = new ArrayList<>();
        for (ColumnDescriptor descriptor : columns) {
            if (depth >= descriptor.getPath().length) {
                throw new InvalidSchemaException("Corrupted Parquet schema");
            }
            if (type.getName().equals(descriptor.getPath()[depth])) {
                res.add(descriptor);
            }
        }

        // If doesn't find the type descriptor in corresponding depth, throw exception
        if (res.isEmpty()) {
            throw new InvalidSchemaException(
                    "Failed to find related Parquet column descriptor with type " + type);
        }
        return res;
    }

    public static ColumnReader createColumnReader(
            boolean isUtcTimestamp,
            LogicalType fieldType,
            Type type,

View on GitHub (pinned to 2f3c205e92)

Solutions

1. Dump both schemas (parquet-tools schema vs table DDL) and align the nesting structure exactly
2. Point the table at files whose schema matches, or rewrite the files with the expected nested structure
3. If types were changed by upstream evolution, rewrite old files to the current schema

// assert every requested nested path exists in the file's flattened column paths
Set<List<String>> filePaths = fileSchema.getPaths().stream().map(Arrays::asList).collect(Collectors.toSet());
for (int i = 0; i < requested.getFieldCount(); i++) {
    if (!filePaths.contains(Arrays.asList(requested.getPaths().get(i)))) {
        throw new IllegalStateException("Requested path absent in file: " + Arrays.toString(requested.getPaths().get(i)));
    }
}

catch (ParquetRuntimeException e) { if ("Corrupted Parquet schema".equals(e.getMessage())) { /* quarantine file, alert on schema drift */ } else throw e; }

When it happens

Trigger: getAllColumnDescriptorByType(depth, type, columns) called with a depth >= some descriptor.getPath().length - i.e. the requested nested schema descends into a level where the file's column paths end (a primitive column where a group was expected).

Common situations: Table schema declares nesting (row/map/array) where the Parquet file has a flat primitive column with the same name; mismatched files behind one table definition; partially corrupted schema metadata.

Related errors

• Failed to find related Parquet column descriptor with type {
• Can not find column io for parquet reader.
• Offset is not monotonically ascending. offsets[%s]=%s, offse
• Parquet with case insensitive mode should have no duplicate
• The quality of field type is incompatible with the request s

AI-assisted analysis of apache/flink@2f3c205e92 (2026-08-14). Data as JSON: /api/errors/02869250b883eb45. Report an issue: GitHub.