apache/druid · error · RuntimeException

Output buffer too small, please allocate more space.

Error message

Output buffer too small, please allocate more space. %s required.

What it means

The Zstd Compressor's compress(ByteBuffer, ByteBuffer) checks Zstd.compressBound(input) against the output buffer's remaining space before compressing. If the destination buffer cannot hold the worst-case compressed size, it throws RuntimeException telling the caller how many bytes are required. Zstd compression can expand small inputs, so the output buffer must be sized generously.

Solutions

Size the output buffer with Zstd.compressBound(in.remaining()) before each compress call
Grow or reallocate the out buffer to the reported required size and retry
Use the allocate-more-space pattern: catch the message, parse the required size, allocate, retry

Example fix

// before
ByteBuffer out = ByteBuffer.allocate(in.remaining());
out = compressor.compress(in, out);
// after
ByteBuffer out = ByteBuffer.allocate((int) Zstd.compressBound(in.remaining()));
out = compressor.compress(in, out);

Defensive patterns

Strategy: validation

Validate before calling

long needed = Zstd.compressBound(in.remaining());
if (out.remaining() < needed) { out = ByteBuffer.allocate((int) needed); }

Try / catch

try {
  out = compressor.compress(in, out);
} catch (RuntimeException e) {
  if (e.getMessage().startsWith("Output buffer too small")) {
    out = ByteBuffer.allocate((int) Zstd.compressBound(in.remaining()));
    out = compressor.compress(in, out);
  } else throw e;
}

Prevention

Always allocate output buffers using Zstd.compressBound
Avoid reusing pooled buffers smaller than the bound
Account for expansion of tiny incompressible blocks

When it happens

Trigger: Calling compress with an output ByteBuffer whose remaining() is less than Zstd.compressBound(in.remaining()).

Common situations: Allocating output buffers equal to input size; reusing small pooled buffers for larger inputs; compressing many tiny blocks with fixed-size buffers.

Related errors

AI-assisted analysis of apache/druid@9b90983fd2 (2026-09-07). Data as JSON: /api/errors/15ab59ff8fe45b64. Report an issue: GitHub.

Appendix: source

Thrown at processing/src/main/java/org/apache/druid/segment/data/CompressionStrategy.java:403

      return inBuffer;
    }

    @Override
    public ByteBuffer allocateOutBuffer(int inputSize, Closer closer)
    {
      ByteBuffer outBuffer = ByteBuffer.allocateDirect((int) Zstd.compressBound(inputSize));
      closer.register(() -> ByteBufferUtils.free(outBuffer));
      return outBuffer;
    }

    @Override
    public ByteBuffer compress(ByteBuffer in, ByteBuffer out)
    {
      int position = in.position();
      out.clear();
      long sizeNeeded = Zstd.compressBound(in.remaining());
      if (out.remaining() < sizeNeeded) {
        throw new RuntimeException("Output buffer too small, please allocate more space. " + sizeNeeded + " required.");
      }
      Zstd.compress(out, in, Zstd.maxCompressionLevel());
      in.position(position);
      out.flip();
      return out;
    }
  }

  public static class ZstdDecompressor implements Decompressor
  {
    private static final ZstdDecompressor DEFAULT_COMPRESSOR = new ZstdDecompressor();

    @Override
    public void decompress(ByteBuffer in, int numBytes, ByteBuffer out)
    {
      out.clear();
      if (!in.isDirect() || !out.isDirect()) {
        // fall back to heap byte arrays if both buffers are not direct

View on GitHub (pinned to 9b90983fd2)