apache/flink · error · IllegalArgumentException

Current version of AvroParquetRecordFormat is not…

Error message

Current version of AvroParquetRecordFormat is not splittable, but found split end (%d) different from file length (%d)

What it means

AvroParquetRecordFormat is not splittable: it must read a parquet file from offset 0 to its end because parquet metadata/footer handling in this format requires the whole file. checkNotSplit throws IllegalArgumentException when the split end does not equal the file length.

Solutions

Configure the source to emit one split per file (unsplittable), so splitEnd == fileLen
Increase parallelism via more/smaller files rather than splitting a single file
Use ParquetVectorizedInputFormat-based formats (e.g. ParquetRowData) when splittable parquet reading is required

Example fix

// before
FileSource.forRecordStreamFormat(format, path) // with a splitter producing partial splits

// after
// keep splits whole: one split per file, e.g.
FileSource.forRecordStreamFormat(format, path)
          .withSplitSize(Long.MAX_VALUE); // effectively one split per file

Defensive patterns

Strategy: validation

Validate before calling

// before reading, assert the split covers the whole file
if (split.getEndOffset() != split.getFileSize()) { throw new IllegalArgumentException("need whole-file split"); }

Prevention

Use one split per file for AvroParquetRecordFormat sources
Size input files for parallelism instead of splitting large ones
Prefer ParquetRowData formats when split parallelism matters

When it happens

Trigger: Using AvroParquetRecordFormat with a FileSource configured with a split assigner or custom splitter that produces splits whose end offset != fileLen (e.g. nonDefaultMinSplitSize, custom FileStoreSplit, or compressed-file splitting logic).

Common situations: Large parquet files where users expect parallelism via splitting; using the avro-parquet stream format with a source (e.g. file store connectors) that hands out partial-file splits.

Related errors

AI-assisted analysis of apache/flink@2f3c205e92 (2026-08-14). Data as JSON: /api/errors/2300c575393991b0. Report an issue: GitHub.

Appendix: source

Thrown at flink-formats/flink-parquet/src/main/java/org/apache/flink/formats/parquet/avro/AvroParquetRecordFormat.java:135

    /** Current version does not support splitting. */
    @Override
    public boolean isSplittable() {
        return false;
    }

    /**
     * Gets the type produced by this format. This type will be the type produced by the file source
     * as a whole.
     */
    @Override
    public TypeInformation<E> getProducedType() {
        return type;
    }

    private static void checkNotSplit(long fileLen, long splitEnd) {
        if (splitEnd != fileLen) {
            throw new IllegalArgumentException(
                    String.format(
                            "Current version of AvroParquetRecordFormat is not splittable, "
                                    + "but found split end (%d) different from file length (%d)",
                            splitEnd, fileLen));
        }
    }

    /**
     * {@link StreamFormat.Reader} implementation. Using {@link ParquetReader} internally to read
     * avro {@link GenericRecord} from parquet {@link InputFile}.
     */
    private static class AvroParquetRecordReader<E> implements StreamFormat.Reader<E> {

        private final ParquetReader<E> parquetReader;

        private long skipCount;
        private final boolean checkpointed;

View on GitHub (pinned to 2f3c205e92)