{"record":{"id":"2300c575393991b0","repo":"apache/flink","slug":"current-version-of-avroparquetrecordformat-is-not","errorCode":null,"errorMessage":"Current version of AvroParquetRecordFormat is not splittable, but found split end (%d) different from file length (%d)","messagePattern":"Current version of AvroParquetRecordFormat is not splittable, but found split end \\((.+?)\\) different from file length \\((.+?)\\)","errorType":"validation","errorClass":"IllegalArgumentException","httpStatus":null,"severity":"error","filePath":"flink-formats/flink-parquet/src/main/java/org/apache/flink/formats/parquet/avro/AvroParquetRecordFormat.java","lineNumber":135,"sourceCode":"\n    /** Current version does not support splitting. */\n    @Override\n    public boolean isSplittable() {\n        return false;\n    }\n\n    /**\n     * Gets the type produced by this format. This type will be the type produced by the file source\n     * as a whole.\n     */\n    @Override\n    public TypeInformation<E> getProducedType() {\n        return type;\n    }\n\n    private static void checkNotSplit(long fileLen, long splitEnd) {\n        if (splitEnd != fileLen) {\n            throw new IllegalArgumentException(\n                    String.format(\n                            \"Current version of AvroParquetRecordFormat is not splittable, \"\n                                    + \"but found split end (%d) different from file length (%d)\",\n                            splitEnd, fileLen));\n        }\n    }\n\n    /**\n     * {@link StreamFormat.Reader} implementation. Using {@link ParquetReader} internally to read\n     * avro {@link GenericRecord} from parquet {@link InputFile}.\n     */\n    private static class AvroParquetRecordReader<E> implements StreamFormat.Reader<E> {\n\n        private final ParquetReader<E> parquetReader;\n\n        private long skipCount;\n        private final boolean checkpointed;\n","sourceCodeStart":117,"sourceCodeEnd":153,"githubUrl":"https://github.com/apache/flink/blob/2f3c205e9266cb30240eb7f4fdab15cad629a70f/flink-formats/flink-parquet/src/main/java/org/apache/flink/formats/parquet/avro/AvroParquetRecordFormat.java#L117-L153","documentation":"AvroParquetRecordFormat is not splittable: it must read a parquet file from offset 0 to its end because parquet metadata/footer handling in this format requires the whole file. checkNotSplit throws IllegalArgumentException when the split end does not equal the file length.","triggerScenarios":"Using AvroParquetRecordFormat with a FileSource configured with a split assigner or custom splitter that produces splits whose end offset != fileLen (e.g. nonDefaultMinSplitSize, custom FileStoreSplit, or compressed-file splitting logic).","commonSituations":"Large parquet files where users expect parallelism via splitting; using the avro-parquet stream format with a source (e.g. file store connectors) that hands out partial-file splits.","solutions":["Configure the source to emit one split per file (unsplittable), so splitEnd == fileLen","Increase parallelism via more/smaller files rather than splitting a single file","Use ParquetVectorizedInputFormat-based formats (e.g. ParquetRowData) when splittable parquet reading is required"],"exampleFix":"// before\nFileSource.forRecordStreamFormat(format, path) // with a splitter producing partial splits\n\n// after\n// keep splits whole: one split per file, e.g.\nFileSource.forRecordStreamFormat(format, path)\n          .withSplitSize(Long.MAX_VALUE); // effectively one split per file","handlingStrategy":"validation","validationCode":"// before reading, assert the split covers the whole file\nif (split.getEndOffset() != split.getFileSize()) { throw new IllegalArgumentException(\"need whole-file split\"); }","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Use one split per file for AvroParquetRecordFormat sources","Size input files for parallelism instead of splitting large ones","Prefer ParquetRowData formats when split parallelism matters"],"tags":["parquet","avro","splitting","file-source"],"backgroundTag":null,"analyzedSha":"2f3c205e9266cb30240eb7f4fdab15cad629a70f","analyzedAt":"2026-08-14T08:48:24.518Z","schemaVersion":2},"datasetVersion":"2026-08-14T10:17:34.591Z"}