{"record":{"id":"cec05f565ff55de5","repo":"prestodb/presto","slug":"snappy-requires-buffer-s-larger-than-max-size","errorCode":null,"errorMessage":"Snappy requires buffer (%s) larger than max size (%s)","messagePattern":"Snappy requires buffer \\((.+?)\\) larger than max size \\((.+?)\\)","errorType":"exception","errorClass":"OrcCorruptionException","httpStatus":null,"severity":"error","filePath":"presto-orc/src/main/java/com/facebook/presto/orc/OrcSnappyDecompressor.java","lineNumber":42,"sourceCode":"{\n    private final OrcDataSourceId orcDataSourceId;\n    private final int maxBufferSize;\n    private final SnappyDecompressor decompressor = new SnappyDecompressor();\n\n    public OrcSnappyDecompressor(OrcDataSourceId orcDataSourceId, int maxBufferSize)\n    {\n        this.orcDataSourceId = requireNonNull(orcDataSourceId, \"orcDataSourceId is null\");\n        this.maxBufferSize = maxBufferSize;\n    }\n\n    @Override\n    public int decompress(byte[] input, int offset, int length, OutputBuffer output)\n            throws OrcCorruptionException\n    {\n        try {\n            int uncompressedLength = SnappyDecompressor.getUncompressedLength(input, offset);\n            if (uncompressedLength > maxBufferSize) {\n                throw new OrcCorruptionException(orcDataSourceId, \"Snappy requires buffer (%s) larger than max size (%s)\", uncompressedLength, maxBufferSize);\n            }\n\n            // Snappy decompressor is more efficient if there's at least a long's worth of extra space\n            // in the output buffer\n            byte[] buffer = output.initialize(uncompressedLength + SIZE_OF_LONG);\n            return decompressor.decompress(input, offset, length, buffer, 0, buffer.length);\n        }\n        catch (MalformedInputException e) {\n            throw new OrcCorruptionException(e, orcDataSourceId, \"Invalid compressed stream\");\n        }\n    }\n\n    @Override\n    public String toString()\n    {\n        return \"snappy\";\n    }\n}","sourceCodeStart":24,"sourceCodeEnd":60,"githubUrl":"https://github.com/prestodb/presto/blob/55bb57d202de3b926896fa966c2c4a44c779634e/presto-orc/src/main/java/com/facebook/presto/orc/OrcSnappyDecompressor.java#L24-L60","documentation":"OrcSnappyDecompressor.decompress first reads the declared uncompressed length from the Snappy stream. If it exceeds maxBufferSize, allocating the output buffer would violate memory limits, so OrcCorruptionException is thrown.","triggerScenarios":"Decompressing a Snappy ORC chunk whose header-declared uncompressedLength exceeds the configured maxBufferSize (OrcReader's maxBufferSize config).","commonSituations":"Very large stripe/chunk sizes written with big compression buffers while reader maxBufferSize is small; corrupted Snappy header claiming huge lengths.","solutions":["Increase the reader's max-buffer-size configuration to at least the chunk's uncompressed size","Rewrite the file with a smaller compression chunk size (e.g. 64KB–1MB)","Validate the file integrity if the declared length seems implausibly large","Align writer compression chunk size with reader buffer limits"],"exampleFix":"// before\nreader max-buffer-size=8MB, file written with 64MB chunks\n// after\nincrease reader config: orc.max-buffer-size=256MB\n(or rewrite with writer .setCompressionMaxBufferSize(sizedDataSize(1, MB)))","handlingStrategy":"validation","validationCode":"// know your writer's max chunk size and size the reader accordingly\nDataSize writerChunk = sizedReader(1, MEGABYTE);\nDataSize readerMax = new DataSize(256, MEGABYTE); // reader max-buffer-size >= writer chunk","typeGuard":null,"tryCatchPattern":"try { reader = new OrcReader(...); }\ncatch (OrcCorruptionException e) {\n    if (e.getMessage().startsWith(\"Snappy requires buffer\")) {\n        /* rebuild reader with larger maxBufferSize or reject the file */\n    }\n}","preventionTips":["Set reader max-buffer-size well above the writer's compression chunk size","Standardize compression chunk size across the pipeline","Reject files whose headers declare implausible sizes"],"tags":["orc","snappy","decompression","memory-limit","configuration"],"backgroundTag":"decompression-buffer-too-small","analyzedSha":"55bb57d202de3b926896fa966c2c4a44c779634e","analyzedAt":"2026-09-04T12:50:26.162Z","contentChangedAt":"2026-09-04T12:50:26.162Z","schemaVersion":2},"datasetVersion":"2026-09-11T21:17:09.523Z"}