apache/flink · error · UnsupportedOperationException

Currently only block sizes up to Integer.MAX_VALUE are suppo

Error message

Currently only block sizes up to Integer.MAX_VALUE are supported

What it means

BinaryInputFormat stores and indexes block boundaries using int arithmetic internally, so it caps the block size at Integer.MAX_VALUE (~2 GB). setBlockSize throws UnsupportedOperationException for any value above Integer.MAX_VALUE even though the field is a long. This is a hard implementation limit of the legacy binary block format.

Source

Thrown at flink-core/src/main/java/org/apache/flink/api/common/io/BinaryInputFormat.java:93

    /**
     * The number of records already read from the block. This is used to decide if the end of the
     * block has been reached.
     */
    private long readRecords = 0;

    @Override
    public void configure(Configuration parameters) {
        super.configure(parameters);
    }

    public void setBlockSize(long blockSize) {
        if (blockSize < 1 && blockSize != NATIVE_BLOCK_SIZE) {
            throw new IllegalArgumentException(
                    "The block size parameter must be set and larger than 0.");
        }
        if (blockSize > Integer.MAX_VALUE) {
            throw new UnsupportedOperationException(
                    "Currently only block sizes up to Integer.MAX_VALUE are supported");
        }
        this.blockSize = blockSize;
    }

    public long getBlockSize() {
        return this.blockSize;
    }

    @Override
    public FileInputSplit[] createInputSplits(int minNumSplits) throws IOException {
        final List<FileStatus> files = this.getFiles();

        final List<FileInputSplit> inputSplits = new ArrayList<FileInputSplit>(minNumSplits);
        for (FileStatus file : files) {
            final FileSystem fs = file.getPath().getFileSystem();
            final long blockSize =
                    this.blockSize == NATIVE_BLOCK_SIZE ? fs.getDefaultBlockSize() : this.blockSize;

View on GitHub (pinned to 2f3c205e92)

Solutions

1. Use a block size <= Integer.MAX_VALUE (<= 2147483647 bytes, ~2 GB).
2. Use BinaryInputFormat.NATIVE_BLOCK_SIZE if you want the format to derive block size from the filesystem.
3. If you genuinely need > 2 GB blocks, switch to a modern source (FLIP-27 FileSource / flink-connector-files) that does not have this limit.

Example fix

// before
format.setBlockSize(3L * 1024 * 1024 * 1024); // 3 GB -> throws

// after
format.setBlockSize(Integer.MAX_VALUE); // ~2 GB ceiling
// or
format.setBlockSize(BinaryInputFormat.NATIVE_BLOCK_SIZE);

long bs = configuredBlockSize;
if (bs > Integer.MAX_VALUE) {
    throw new IllegalArgumentException("block size must be <= Integer.MAX_VALUE (~2GB), got " + bs);
}
format.setBlockSize(bs);

When it happens

Trigger: Calling format.setBlockSize with a value > Integer.MAX_VALUE (e.g. 3L * 1024 * 1024 * 1024 for 3 GB, or any long exceeding 2147483647).

Common situations: Migrating to very large file blocks (multi-GB) typical of modern object stores; computing block size from a total file size and exceeding 2 GB; treating the long-typed parameter as supporting full 64-bit ranges.

Related errors

• The block size parameter must be set and larger than 0.
• You must have forgotten to call open() on your input format.
• Can't deploy a standalone cluster.
• Application Mode not supported by standalone deployments.
• Cannot terminate a standalone cluster.

AI-assisted analysis of apache/flink@2f3c205e92 (2026-08-14). Data as JSON: /api/errors/f7df65fc822c5329. Report an issue: GitHub.