apache/flink · error · IllegalArgumentException

SimpleStreamFormat is not splittable, but found split end (%

Error message

SimpleStreamFormat is not splittable, but found split end (%d) different from file length (%d)

What it means

SimpleStreamFormat is a non-splittable format that reads an entire file from start to end as a single stream. The checkNotSplit method validates that the split end equals the file length; if they differ, it throws IllegalArgumentException because the format cannot handle partial-file reads. Splittable formats (BulkFormat-based) support arbitrary split boundaries; SimpleStreamFormat does not.

Source

Thrown at flink-connectors/flink-connector-files/src/main/java/org/apache/flink/connector/file/src/reader/SimpleStreamFormat.java:109

    }

    @Override
    public final Reader<T> restoreReader(
            final Configuration config,
            final FSDataInputStream stream,
            final long restoredOffset,
            final long fileLen,
            final long splitEnd)
            throws IOException {

        checkNotSplit(fileLen, splitEnd);
        stream.seek(restoredOffset);
        return createReader(config, stream);
    }

    private static void checkNotSplit(long fileLen, long splitEnd) {
        if (splitEnd != fileLen) {
            throw new IllegalArgumentException(
                    String.format(
                            "SimpleStreamFormat is not splittable, but found split end (%d) different from file length (%d)",
                            splitEnd, fileLen));
        }
    }
}

View on GitHub (pinned to 2f3c205e92)

Solutions

Switch to a BulkFormat-based reader (splittable) if the data format supports it (e.g. TextLineInputFormat for plain text).
Configure the FileSource to not split files when using SimpleStreamFormat — set the split size larger than the largest expected file.
Use a splittable compression format (e.g. bzip2) or uncompressed format if splitting is required.

Example fix

// before
FileSource.forRecordStreamFormat(new MySimpleStreamFormat(), path)
    // large file gets split -> IllegalArgumentException

// after
FileSource.forBulkFileFormat(new MyBulkFormat(), path)
    // BulkFormat supports arbitrary split boundaries

Defensive patterns

Strategy: validation

Validate before calling

// Check if format is splittable before creating splits
if (format instanceof SimpleStreamFormat) {
    // ensure files are not split; use a single split per file
    LOG.warn("SimpleStreamFormat is non-splittable; each file will be a single split");
}

Type guard

boolean isSplittable(StreamFormat<?> format) {
    return !(format instanceof SimpleStreamFormat);
}

Prevention

Use BulkFormat for large files that need splitting; reserve SimpleStreamFormat for whole-file reads.
If using SimpleStreamFormat, set split size larger than the largest file to avoid splitting.
Avoid gzip-compressed files with splitting enabled; use bzip2 or uncompressed for splittable reads.

When it happens

Trigger: A SimpleStreamFormat subclass is used as the reader format for a FileSource where the file is split into multiple parts (e.g. the source created splits where splitEnd < fileLen). This can happen when the file is large enough to be split but the format does not support it.

Common situations: Using a non-splittable format (like a custom SimpleStreamFormat or certain compression formats) with a file source that attempts to split large files. Configuring a small split size with a format that can only read whole files. Reading compressed files (gzip) which are inherently non-splittable but split size is set low.

Related errors

AI-assisted analysis of apache/flink@2f3c205e92 (2026-08-14). Data as JSON: /api/errors/7b826b52413c8cac. Report an issue: GitHub.