{"record":{"id":"41fefac9b959e229","repo":"apache/seatunnel","slug":"binary-chunk-size-too-large-max-100mb-got-s","errorCode":null,"errorMessage":"Binary chunk size too large (max 100MB), got: %s","messagePattern":"Binary chunk size too large \\(max 100MB\\), got: (.+?)","errorType":"exception","errorClass":"IllegalArgumentException","httpStatus":null,"severity":"error","filePath":"seatunnel-connectors-v2/connector-file/connector-file-base/src/main/java/org/apache/seatunnel/connectors/seatunnel/file/source/reader/BinaryReadStrategy.java","lineNumber":82,"sourceCode":"            basePathIsFile = hadoopFileSystemProxy.isFile(basePath);\n        } catch (IOException e) {\n            throw new FileConnectorException(\n                    SeaTunnelAPIErrorCode.CONFIG_VALIDATION_FAILED,\n                    \"Failed to determine whether file source path is a file or directory: \"\n                            + basePath,\n                    e);\n        }\n\n        // Load binary chunk size configuration\n        if (pluginConfig.hasPath(FileBaseSourceOptions.BINARY_CHUNK_SIZE.key())) {\n            binaryChunkSize = pluginConfig.getInt(FileBaseSourceOptions.BINARY_CHUNK_SIZE.key());\n            // Validate chunk size - should be positive and reasonable\n            if (binaryChunkSize <= 0) {\n                throw new IllegalArgumentException(\n                        \"Binary chunk size must be positive, got: \" + binaryChunkSize);\n            }\n            if (binaryChunkSize > 100 * 1024 * 1024) { // 100MB limit\n                throw new IllegalArgumentException(\n                        \"Binary chunk size too large (max 100MB), got: \" + binaryChunkSize);\n            }\n        }\n\n        // Load complete file mode configuration\n        if (pluginConfig.hasPath(FileBaseSourceOptions.BINARY_COMPLETE_FILE_MODE.key())) {\n            completeFileMode =\n                    pluginConfig.getBoolean(FileBaseSourceOptions.BINARY_COMPLETE_FILE_MODE.key());\n        }\n    }\n\n    @Override\n    public void read(String path, String tableId, Collector<SeaTunnelRow> output)\n            throws IOException, FileConnectorException {\n        MessageDigest digest = createSha256Digest();\n        lastReadFingerprint = null;\n        try (InputStream inputStream =\n                new DigestTrackingInputStream(hadoopFileSystemProxy.getInputStream(path), digest)) {","sourceCodeStart":64,"sourceCodeEnd":100,"githubUrl":"https://github.com/apache/seatunnel/blob/cf67b549a7a6c35fa0beb12d83c62892427ea919/seatunnel-connectors-v2/connector-file/connector-file-base/src/main/java/org/apache/seatunnel/connectors/seatunnel/file/source/reader/BinaryReadStrategy.java#L64-L100","documentation":"BinaryReadStrategy.init caps binary_chunk_size at 100MB (100 * 1024 * 1024 bytes) to prevent oversized in-memory buffers. A larger configured value throws IllegalArgumentException with the configured number. This protects worker heap memory from a single huge allocation.","triggerScenarios":"Setting binary_chunk_size greater than 104857600, e.g. binary_chunk_size = 268435456 (256MB), when initializing the binary read strategy.","commonSituations":"Users reading very large binary files assuming a bigger chunk is faster; unit confusion (setting a GB value thinking it is MB); copied configs from other tools without the cap.","solutions":["Reduce binary_chunk_size to at most 104857600 bytes.","Remove the option to use the default chunk size.","If throughput is the goal, increase source parallelism instead of chunk size."],"exampleFix":"// before\nbinary_chunk_size = 268435456  // 256MB\n// after\nbinary_chunk_size = 104857600  // max 100MB","handlingStrategy":"validation","validationCode":"// Java\nint chunk = config.getInt(\"binary_chunk_size\", DEFAULT_CHUNK);\nint MAX = 100 * 1024 * 1024;\nif (chunk > MAX) {\n    throw new IllegalArgumentException(\"binary_chunk_size max is 100MB, got: \" + chunk);\n}","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Cap chunk size at 104857600 bytes in your config generator.","Scale read parallelism rather than chunk size for throughput.","Document the 100MB limit wherever the option is templated."],"tags":["config-validation","binary","memory"],"backgroundTag":"value-out-of-range","analyzedSha":"cf67b549a7a6c35fa0beb12d83c62892427ea919","analyzedAt":"2026-09-10T21:44:55.265Z","contentChangedAt":"2026-09-10T21:44:55.265Z","schemaVersion":2},"datasetVersion":"2026-09-23T08:17:48.524Z"}