prestodb/presto · error · OrcCorruptionException

Snappy requires buffer (%s) larger than max size (%s)

Error message

Snappy requires buffer (%s) larger than max size (%s)

What it means

OrcSnappyDecompressor.decompress first reads the declared uncompressed length from the Snappy stream. If it exceeds maxBufferSize, allocating the output buffer would violate memory limits, so OrcCorruptionException is thrown.

Source

Thrown at presto-orc/src/main/java/com/facebook/presto/orc/OrcSnappyDecompressor.java:42

{
    private final OrcDataSourceId orcDataSourceId;
    private final int maxBufferSize;
    private final SnappyDecompressor decompressor = new SnappyDecompressor();

    public OrcSnappyDecompressor(OrcDataSourceId orcDataSourceId, int maxBufferSize)
    {
        this.orcDataSourceId = requireNonNull(orcDataSourceId, "orcDataSourceId is null");
        this.maxBufferSize = maxBufferSize;
    }

    @Override
    public int decompress(byte[] input, int offset, int length, OutputBuffer output)
            throws OrcCorruptionException
    {
        try {
            int uncompressedLength = SnappyDecompressor.getUncompressedLength(input, offset);
            if (uncompressedLength > maxBufferSize) {
                throw new OrcCorruptionException(orcDataSourceId, "Snappy requires buffer (%s) larger than max size (%s)", uncompressedLength, maxBufferSize);
            }

            // Snappy decompressor is more efficient if there's at least a long's worth of extra space
            // in the output buffer
            byte[] buffer = output.initialize(uncompressedLength + SIZE_OF_LONG);
            return decompressor.decompress(input, offset, length, buffer, 0, buffer.length);
        }
        catch (MalformedInputException e) {
            throw new OrcCorruptionException(e, orcDataSourceId, "Invalid compressed stream");
        }
    }

    @Override
    public String toString()
    {
        return "snappy";
    }
}

View on GitHub (pinned to 55bb57d202)

Solutions

1. Increase the reader's max-buffer-size configuration to at least the chunk's uncompressed size
2. Rewrite the file with a smaller compression chunk size (e.g. 64KB–1MB)
3. Validate the file integrity if the declared length seems implausibly large
4. Align writer compression chunk size with reader buffer limits

Example fix

// before
reader max-buffer-size=8MB, file written with 64MB chunks
// after
increase reader config: orc.max-buffer-size=256MB
(or rewrite with writer .setCompressionMaxBufferSize(sizedDataSize(1, MB)))

// know your writer's max chunk size and size the reader accordingly
DataSize writerChunk = sizedReader(1, MEGABYTE);
DataSize readerMax = new DataSize(256, MEGABYTE); // reader max-buffer-size >= writer chunk

try { reader = new OrcReader(...); }
catch (OrcCorruptionException e) {
    if (e.getMessage().startsWith("Snappy requires buffer")) {
        /* rebuild reader with larger maxBufferSize or reject the file */
    }
}

When it happens

Trigger: Decompressing a Snappy ORC chunk whose header-declared uncompressedLength exceeds the configured maxBufferSize (OrcReader's maxBufferSize config).

Common situations: Very large stripe/chunk sizes written with big compression buffers while reader maxBufferSize is small; corrupted Snappy header claiming huge lengths.

Related errors

• Invalid compressed stream
• HIVE_UNSUPPORTED_FORMAT
• NOT_SUPPORTED
• NOT_SUPPORTED
• Unsupported compression for verification:

AI-assisted analysis of prestodb/presto@55bb57d202 (2026-09-04). Data as JSON: /api/errors/cec05f565ff55de5. Report an issue: GitHub.