apache/flink · error · IllegalArgumentException

Offset is not monotonically ascending. offsets[%s]=%s, offse

Error message

Offset is not monotonically ascending. offsets[%s]=%s, offsets[%s]=%s

What it means

NestedPositionUtil.calculateLengthByOffsets derives each collection element's length from consecutive offsets. If arrayOffsets is not monotonically non-decreasing (a later offset is smaller than the previous), the computed length is negative and it throws IllegalArgumentException naming the offending indices.

Source

Thrown at flink-formats/flink-parquet/src/main/java/org/apache/flink/formats/parquet/utils/NestedPositionUtil.java:157

        if (nullValuesCount == 0) {
            return new CollectionPosition(null, offsetsArray, length, valueCount);
        }
        return new CollectionPosition(
                nullCollectionFlags.toArray(), offsetsArray, length, valueCount);
    }

    public static boolean isOptionalFieldValueNull(int definitionLevel, int maxDefinitionLevel) {
        return definitionLevel == maxDefinitionLevel - 1;
    }

    public static long[] calculateLengthByOffsets(
            boolean[] collectionIsEmpty, long[] arrayOffsets) {
        LongArrayList lengthList = new LongArrayList(arrayOffsets.length);
        for (int i = 0; i < arrayOffsets.length - 1; i++) {
            long offset = arrayOffsets[i];
            long length = arrayOffsets[i + 1] - offset;
            if (length < 0) {
                throw new IllegalArgumentException(
                        format(
                                "Offset is not monotonically ascending. offsets[%s]=%s, offsets[%s]=%s",
                                i, arrayOffsets[i], i + 1, arrayOffsets[i + 1]));
            }
            if (collectionIsEmpty[i]) {
                length = 0;
            }
            lengthList.add(length);
        }
        return lengthList.toArray();
    }

    private static int getNextCollectionStartIndex(
            int[] repetitionLevels, int maxRepetitionLevel, int elementIndex) {
        do {
            elementIndex++;
        } while (hasMoreElements(repetitionLevels, elementIndex)
                && isNotCollectionBeginningMarker(

View on GitHub (pinned to 2f3c205e92)

Solutions

1. Validate/re-write the offending parquet file (parquet-tools / spark read of the same file usually confirms corruption)
2. If the file reads fine elsewhere, update Flink to pick up fixes in the nested-column vectorized reader
3. Quarantine the bad input split and let the job continue with clean files

try { readSplit(split); } catch (IllegalArgumentException e) { if (e.getMessage().contains("monotonically ascending")) { quarantine(split); /* skip bad file, alert */ } else throw e; }

When it happens

Trigger: Reading nested (array/map) parquet columns where the reconstructed offset array from definition/repetition levels is malformed - typically a corrupt file, a reader bug in level processing, or offsets produced by a writer violating the row-data offset contract.

Common situations: Corrupted parquet files (truncated network copy, bad HDFS block), files written by non-conforming writers, or version skew in the vectorized reading path for nested types.

Related errors

• Corrupted Parquet schema
• expecting more rows but reached last block. Read {} out of {
• Failed to find related Parquet column descriptor with type {
• Unknown ColumnIO, %s
• Can not find column io for parquet reader.

AI-assisted analysis of apache/flink@2f3c205e92 (2026-08-14). Data as JSON: /api/errors/882f80e50dd7976d. Report an issue: GitHub.