{"record":{"id":"7b826b52413c8cac","repo":"apache/flink","slug":"simplestreamformat-is-not-splittable-but-found-sp","errorCode":null,"errorMessage":"SimpleStreamFormat is not splittable, but found split end (%d) different from file length (%d)","messagePattern":"SimpleStreamFormat is not splittable, but found split end \\((.+?)\\) different from file length \\((.+?)\\)","errorType":"exception","errorClass":"IllegalArgumentException","httpStatus":null,"severity":"error","filePath":"flink-connectors/flink-connector-files/src/main/java/org/apache/flink/connector/file/src/reader/SimpleStreamFormat.java","lineNumber":109,"sourceCode":"    }\n\n    @Override\n    public final Reader<T> restoreReader(\n            final Configuration config,\n            final FSDataInputStream stream,\n            final long restoredOffset,\n            final long fileLen,\n            final long splitEnd)\n            throws IOException {\n\n        checkNotSplit(fileLen, splitEnd);\n        stream.seek(restoredOffset);\n        return createReader(config, stream);\n    }\n\n    private static void checkNotSplit(long fileLen, long splitEnd) {\n        if (splitEnd != fileLen) {\n            throw new IllegalArgumentException(\n                    String.format(\n                            \"SimpleStreamFormat is not splittable, but found split end (%d) different from file length (%d)\",\n                            splitEnd, fileLen));\n        }\n    }\n}\n","sourceCodeStart":91,"sourceCodeEnd":116,"githubUrl":"https://github.com/apache/flink/blob/2f3c205e9266cb30240eb7f4fdab15cad629a70f/flink-connectors/flink-connector-files/src/main/java/org/apache/flink/connector/file/src/reader/SimpleStreamFormat.java#L91-L116","documentation":"SimpleStreamFormat is a non-splittable format that reads an entire file from start to end as a single stream. The checkNotSplit method validates that the split end equals the file length; if they differ, it throws IllegalArgumentException because the format cannot handle partial-file reads. Splittable formats (BulkFormat-based) support arbitrary split boundaries; SimpleStreamFormat does not.","triggerScenarios":"A SimpleStreamFormat subclass is used as the reader format for a FileSource where the file is split into multiple parts (e.g. the source created splits where splitEnd < fileLen). This can happen when the file is large enough to be split but the format does not support it.","commonSituations":"Using a non-splittable format (like a custom SimpleStreamFormat or certain compression formats) with a file source that attempts to split large files. Configuring a small split size with a format that can only read whole files. Reading compressed files (gzip) which are inherently non-splittable but split size is set low.","solutions":["Switch to a BulkFormat-based reader (splittable) if the data format supports it (e.g. TextLineInputFormat for plain text).","Configure the FileSource to not split files when using SimpleStreamFormat — set the split size larger than the largest expected file.","Use a splittable compression format (e.g. bzip2) or uncompressed format if splitting is required."],"exampleFix":"// before\nFileSource.forRecordStreamFormat(new MySimpleStreamFormat(), path)\n    // large file gets split -> IllegalArgumentException\n\n// after\nFileSource.forBulkFileFormat(new MyBulkFormat(), path)\n    // BulkFormat supports arbitrary split boundaries","handlingStrategy":"validation","validationCode":"// Check if format is splittable before creating splits\nif (format instanceof SimpleStreamFormat) {\n    // ensure files are not split; use a single split per file\n    LOG.warn(\"SimpleStreamFormat is non-splittable; each file will be a single split\");\n}","typeGuard":"boolean isSplittable(StreamFormat<?> format) {\n    return !(format instanceof SimpleStreamFormat);\n}","tryCatchPattern":null,"preventionTips":["Use BulkFormat for large files that need splitting; reserve SimpleStreamFormat for whole-file reads.","If using SimpleStreamFormat, set split size larger than the largest file to avoid splitting.","Avoid gzip-compressed files with splitting enabled; use bzip2 or uncompressed for splittable reads."],"tags":["simple-stream-format","non-splittable","file-source","flink"],"backgroundTag":null,"analyzedSha":"2f3c205e9266cb30240eb7f4fdab15cad629a70f","analyzedAt":"2026-08-14T08:48:24.518Z","schemaVersion":2},"datasetVersion":"2026-08-14T10:17:34.591Z"}