{"record":{"id":"15ab59ff8fe45b64","repo":"apache/druid","slug":"output-buffer-too-small-please-allocate-more-spac","errorCode":null,"errorMessage":"Output buffer too small, please allocate more space. %s required.","messagePattern":"Output buffer too small, please allocate more space\\. (.+?) required\\.","errorType":"exception","errorClass":"RuntimeException","httpStatus":null,"severity":"error","filePath":"processing/src/main/java/org/apache/druid/segment/data/CompressionStrategy.java","lineNumber":403,"sourceCode":"      return inBuffer;\n    }\n\n    @Override\n    public ByteBuffer allocateOutBuffer(int inputSize, Closer closer)\n    {\n      ByteBuffer outBuffer = ByteBuffer.allocateDirect((int) Zstd.compressBound(inputSize));\n      closer.register(() -> ByteBufferUtils.free(outBuffer));\n      return outBuffer;\n    }\n\n    @Override\n    public ByteBuffer compress(ByteBuffer in, ByteBuffer out)\n    {\n      int position = in.position();\n      out.clear();\n      long sizeNeeded = Zstd.compressBound(in.remaining());\n      if (out.remaining() < sizeNeeded) {\n        throw new RuntimeException(\"Output buffer too small, please allocate more space. \" + sizeNeeded + \" required.\");\n      }\n      Zstd.compress(out, in, Zstd.maxCompressionLevel());\n      in.position(position);\n      out.flip();\n      return out;\n    }\n  }\n\n  public static class ZstdDecompressor implements Decompressor\n  {\n    private static final ZstdDecompressor DEFAULT_COMPRESSOR = new ZstdDecompressor();\n\n    @Override\n    public void decompress(ByteBuffer in, int numBytes, ByteBuffer out)\n    {\n      out.clear();\n      if (!in.isDirect() || !out.isDirect()) {\n        // fall back to heap byte arrays if both buffers are not direct","sourceCodeStart":385,"sourceCodeEnd":421,"githubUrl":"https://github.com/apache/druid/blob/9b90983fd291f26935af934383ce360473179e4d/processing/src/main/java/org/apache/druid/segment/data/CompressionStrategy.java#L385-L421","documentation":"The Zstd Compressor's compress(ByteBuffer, ByteBuffer) checks Zstd.compressBound(input) against the output buffer's remaining space before compressing. If the destination buffer cannot hold the worst-case compressed size, it throws RuntimeException telling the caller how many bytes are required. Zstd compression can expand small inputs, so the output buffer must be sized generously.","triggerScenarios":"Calling compress with an output ByteBuffer whose remaining() is less than Zstd.compressBound(in.remaining()).","commonSituations":"Allocating output buffers equal to input size; reusing small pooled buffers for larger inputs; compressing many tiny blocks with fixed-size buffers.","solutions":["Size the output buffer with Zstd.compressBound(in.remaining()) before each compress call","Grow or reallocate the out buffer to the reported required size and retry","Use the allocate-more-space pattern: catch the message, parse the required size, allocate, retry"],"exampleFix":"// before\nByteBuffer out = ByteBuffer.allocate(in.remaining());\nout = compressor.compress(in, out);\n// after\nByteBuffer out = ByteBuffer.allocate((int) Zstd.compressBound(in.remaining()));\nout = compressor.compress(in, out);","handlingStrategy":"validation","validationCode":"long needed = Zstd.compressBound(in.remaining());\nif (out.remaining() < needed) { out = ByteBuffer.allocate((int) needed); }","typeGuard":null,"tryCatchPattern":"try {\n  out = compressor.compress(in, out);\n} catch (RuntimeException e) {\n  if (e.getMessage().startsWith(\"Output buffer too small\")) {\n    out = ByteBuffer.allocate((int) Zstd.compressBound(in.remaining()));\n    out = compressor.compress(in, out);\n  } else throw e;\n}","preventionTips":["Always allocate output buffers using Zstd.compressBound","Avoid reusing pooled buffers smaller than the bound","Account for expansion of tiny incompressible blocks"],"tags":["java","compression","zstd","buffer"],"backgroundTag":"buffer-too-small","analyzedSha":"9b90983fd291f26935af934383ce360473179e4d","analyzedAt":"2026-09-07T13:32:30.957Z","contentChangedAt":"2026-09-07T13:32:30.957Z","schemaVersion":2},"datasetVersion":"2026-09-17T15:17:12.973Z"}