apache/flink · error · IllegalArgumentException

Current version of AvroParquetRecordFormat is not splittable

Error message

Current version of AvroParquetRecordFormat is not splittable, but found split end (%d) different from file length (%d)

What it means

AvroParquetRecordFormat is not splittable: it must read a parquet file from offset 0 to its end because parquet metadata/footer handling in this format requires the whole file. checkNotSplit throws IllegalArgumentException when the split end does not equal the file length.

Source

Thrown at flink-formats/flink-parquet/src/main/java/org/apache/flink/formats/parquet/avro/AvroParquetRecordFormat.java:135

    /** Current version does not support splitting. */
    @Override
    public boolean isSplittable() {
        return false;
    }

    /**
     * Gets the type produced by this format. This type will be the type produced by the file source
     * as a whole.
     */
    @Override
    public TypeInformation<E> getProducedType() {
        return type;
    }

    private static void checkNotSplit(long fileLen, long splitEnd) {
        if (splitEnd != fileLen) {
            throw new IllegalArgumentException(
                    String.format(
                            "Current version of AvroParquetRecordFormat is not splittable, "
                                    + "but found split end (%d) different from file length (%d)",
                            splitEnd, fileLen));
        }
    }

    /**
     * {@link StreamFormat.Reader} implementation. Using {@link ParquetReader} internally to read
     * avro {@link GenericRecord} from parquet {@link InputFile}.
     */
    private static class AvroParquetRecordReader<E> implements StreamFormat.Reader<E> {

        private final ParquetReader<E> parquetReader;

        private long skipCount;
        private final boolean checkpointed;

View on GitHub (pinned to 2f3c205e92)

Solutions

1. Configure the source to emit one split per file (unsplittable), so splitEnd == fileLen
2. Increase parallelism via more/smaller files rather than splitting a single file
3. Use ParquetVectorizedInputFormat-based formats (e.g. ParquetRowData) when splittable parquet reading is required

Example fix

// before
FileSource.forRecordStreamFormat(format, path) // with a splitter producing partial splits

// after
// keep splits whole: one split per file, e.g.
FileSource.forRecordStreamFormat(format, path)
          .withSplitSize(Long.MAX_VALUE); // effectively one split per file

// before reading, assert the split covers the whole file
if (split.getEndOffset() != split.getFileSize()) { throw new IllegalArgumentException("need whole-file split"); }

When it happens

Trigger: Using AvroParquetRecordFormat with a FileSource configured with a split assigner or custom splitter that produces splits whose end offset != fileLen (e.g. nonDefaultMinSplitSize, custom FileStoreSplit, or compressed-file splitting logic).

Common situations: Large parquet files where users expect parallelism via splitting; using the avro-parquet stream format with a source (e.g. file store connectors) that hands out partial-file splits.

Related errors

• Please use AvroParquetReaders.forSpecificRecord(Class<T>) fo
• Please use AvroParquetReaders.forGenericRecord(Class<T>) for
• Error while waiting for job to be initialized
• Translator {} cannot translate the given pipeline {}.
• Could not get job jar and dependencies from JAR file: {}

AI-assisted analysis of apache/flink@2f3c205e92 (2026-08-14). Data as JSON: /api/errors/2300c575393991b0. Report an issue: GitHub.