prestodb/presto · error · OrcEncryptionException

Encrypted data size %s exceeds limit of 2^23

Error message

Encrypted data size %s exceeds limit of 2^23

What it means

In OrcOutputBuffer.writeChunkToOutputStream, DWRF encryption runs on a chunk before writing; each chunk must fit in a 3-byte length header (max 2^23 = 8388608 bytes). If the encrypted data exceeds that, OrcEncryptionException is thrown because the resulting ORC chunk cannot be encoded.

Source

Thrown at presto-orc/src/main/java/com/facebook/presto/orc/OrcOutputBuffer.java:567

                compressionBuffer = compressionBufferPool.checkOut(minCompressionBufferSize);
                int compressedSize = compressor.compress(chunk, offset, length, compressionBuffer, 0, compressionBuffer.length);
                if (compressedSize < length) {
                    if (verifyDecompressor != null) {
                        verifyCompressedChunk(verifyDecompressor, verifyDecompressionBufferPool, chunk, offset, length, compressionBuffer, compressedSize);
                    }
                    isCompressed = true;
                    chunk = compressionBuffer;
                    length = compressedSize;
                    offset = 0;
                }
            }
            if (dwrfEncryptor.isPresent()) {
                chunk = dwrfEncryptor.get().encrypt(chunk, offset, length);
                length = chunk.length;
                offset = 0;
                // size after encryption should not exceed what the 3 byte header can hold (2^23)
                if (length > 8388608) {
                    throw new OrcEncryptionException("Encrypted data size %s exceeds limit of 2^23", length);
                }
            }

            int header = isCompressed ? length << 1 : (length << 1) + 1;
            writeChunkedOutput(chunk, offset, length, header);
        }
        finally {
            if (compressionBuffer != null) {
                compressionBufferPool.checkIn(compressionBuffer);
            }
        }
    }

    private void writeChunkedOutput(byte[] chunk, int offset, int length, int header)
    {
        compressedOutputStream.ensureAvailable(3, length + 3);
        compressedOutputStream.writeHeader(header);
        compressedOutputStream.writeBytes(chunk, offset, length);

View on GitHub (pinned to 55bb57d202)

Solutions

1. Reduce the ORC writer's strip/buffer size so raw chunks stay safely below 2^23 before encryption.
2. Use a compression setting that shrinks chunks below the limit (or verify compression is actually applied before encryption).
3. Upgrade to a Presto version that splits oversized chunks instead of failing, if available.
4. If using a custom DWRF encryptor, fix encryption overhead so ciphertext length <= plaintext length.
5. Split the input so a single column chunk cannot exceed ~8MB of encrypted output.

Example fix

// before: huge buffer leading to >2^23 encrypted chunks
OrcWriterOptions.options().withMaxBufferSize(16MB)
// after
OrcWriterOptions.options().withMaxBufferSize(4MB)

// before writing an encrypted DWRF chunk
if (chunk.length > 8388608) {
    throw new IllegalArgumentException("Chunk too large for encrypted ORC output: " + chunk.length + " > 8388608");
}

boolean fitsEncryptedHeaderLimit(byte[] encryptedChunk) { return encryptedChunk.length <= 8388608; }

try { orcWriter.writeChunk(chunk); }
catch (OrcEncryptionException e) {
    log.error("Encrypted chunk exceeded 2^23; reduce writer buffer size: %s", e.getMessage());
    throw new IOException("Rewrite with smaller strip/buffer size", e);
}

When it happens

Trigger: Writing a large chunk to a DWRF-encrypted ORC output stream where encryption expands the data past 8388608 bytes (length > 8388608 check after dwrfEncryptor.encrypt).

Common situations: Very wide/huge row groups producing large raw chunks; encryptor with padding/overhead pushing a chunk just under 8MB over the limit after encryption; writers configured with overly large buffer/compression settings; defective encryptor implementation that bloats data.

Related errors

• Stripe encryption keys are missing, but file is encrypted
• Number of stripe encryption keys did not match number of enc
• Failed to read or decrypt FileStatistics for node %s
• GENERIC_INTERNAL_ERROR
• HIVE_INVALID_ENCRYPTION_METADATA

AI-assisted analysis of prestodb/presto@55bb57d202 (2026-09-04). Data as JSON: /api/errors/f6f129d84ceb8c58. Report an issue: GitHub.