{"record":{"id":"f6f129d84ceb8c58","repo":"prestodb/presto","slug":"encrypted-data-size-s-exceeds-limit-of-2-23","errorCode":null,"errorMessage":"Encrypted data size %s exceeds limit of 2^23","messagePattern":"Encrypted data size (.+?) exceeds limit of 2\\^23","errorType":"exception","errorClass":"OrcEncryptionException","httpStatus":null,"severity":"error","filePath":"presto-orc/src/main/java/com/facebook/presto/orc/OrcOutputBuffer.java","lineNumber":567,"sourceCode":"                compressionBuffer = compressionBufferPool.checkOut(minCompressionBufferSize);\n                int compressedSize = compressor.compress(chunk, offset, length, compressionBuffer, 0, compressionBuffer.length);\n                if (compressedSize < length) {\n                    if (verifyDecompressor != null) {\n                        verifyCompressedChunk(verifyDecompressor, verifyDecompressionBufferPool, chunk, offset, length, compressionBuffer, compressedSize);\n                    }\n                    isCompressed = true;\n                    chunk = compressionBuffer;\n                    length = compressedSize;\n                    offset = 0;\n                }\n            }\n            if (dwrfEncryptor.isPresent()) {\n                chunk = dwrfEncryptor.get().encrypt(chunk, offset, length);\n                length = chunk.length;\n                offset = 0;\n                // size after encryption should not exceed what the 3 byte header can hold (2^23)\n                if (length > 8388608) {\n                    throw new OrcEncryptionException(\"Encrypted data size %s exceeds limit of 2^23\", length);\n                }\n            }\n\n            int header = isCompressed ? length << 1 : (length << 1) + 1;\n            writeChunkedOutput(chunk, offset, length, header);\n        }\n        finally {\n            if (compressionBuffer != null) {\n                compressionBufferPool.checkIn(compressionBuffer);\n            }\n        }\n    }\n\n    private void writeChunkedOutput(byte[] chunk, int offset, int length, int header)\n    {\n        compressedOutputStream.ensureAvailable(3, length + 3);\n        compressedOutputStream.writeHeader(header);\n        compressedOutputStream.writeBytes(chunk, offset, length);","sourceCodeStart":549,"sourceCodeEnd":585,"githubUrl":"https://github.com/prestodb/presto/blob/55bb57d202de3b926896fa966c2c4a44c779634e/presto-orc/src/main/java/com/facebook/presto/orc/OrcOutputBuffer.java#L549-L585","documentation":"In OrcOutputBuffer.writeChunkToOutputStream, DWRF encryption runs on a chunk before writing; each chunk must fit in a 3-byte length header (max 2^23 = 8388608 bytes). If the encrypted data exceeds that, OrcEncryptionException is thrown because the resulting ORC chunk cannot be encoded.","triggerScenarios":"Writing a large chunk to a DWRF-encrypted ORC output stream where encryption expands the data past 8388608 bytes (length > 8388608 check after dwrfEncryptor.encrypt).","commonSituations":"Very wide/huge row groups producing large raw chunks; encryptor with padding/overhead pushing a chunk just under 8MB over the limit after encryption; writers configured with overly large buffer/compression settings; defective encryptor implementation that bloats data.","solutions":["Reduce the ORC writer's strip/buffer size so raw chunks stay safely below 2^23 before encryption.","Use a compression setting that shrinks chunks below the limit (or verify compression is actually applied before encryption).","Upgrade to a Presto version that splits oversized chunks instead of failing, if available.","If using a custom DWRF encryptor, fix encryption overhead so ciphertext length <= plaintext length.","Split the input so a single column chunk cannot exceed ~8MB of encrypted output."],"exampleFix":"// before: huge buffer leading to >2^23 encrypted chunks\nOrcWriterOptions.options().withMaxBufferSize(16MB)\n// after\nOrcWriterOptions.options().withMaxBufferSize(4MB)","handlingStrategy":"validation","validationCode":"// before writing an encrypted DWRF chunk\nif (chunk.length > 8388608) {\n    throw new IllegalArgumentException(\"Chunk too large for encrypted ORC output: \" + chunk.length + \" > 8388608\");\n}","typeGuard":"boolean fitsEncryptedHeaderLimit(byte[] encryptedChunk) { return encryptedChunk.length <= 8388608; }","tryCatchPattern":"try { orcWriter.writeChunk(chunk); }\ncatch (OrcEncryptionException e) {\n    log.error(\"Encrypted chunk exceeded 2^23; reduce writer buffer size: %s\", e.getMessage());\n    throw new IOException(\"Rewrite with smaller strip/buffer size\", e);\n}","preventionTips":["Size writer buffers so encrypted output stays well under 8MB per chunk","Verify compression is reducing chunks before encryption","Test with maximum-width rows when enabling DWRF encryption","Upgrade Presto if newer versions handle oversized encrypted chunks gracefully"],"tags":["orc","dwrf","encryption","buffer-limit"],"backgroundTag":"chunk-size-limit-exceeded","analyzedSha":"55bb57d202de3b926896fa966c2c4a44c779634e","analyzedAt":"2026-09-04T12:50:26.162Z","contentChangedAt":"2026-09-04T12:50:26.162Z","schemaVersion":2},"datasetVersion":"2026-09-11T21:17:09.523Z"}