apache/druid · error · IllegalArgumentException

Cannot copy [%,d] bytes, only have [%,d] buffered

Error message

Cannot copy [%,d] bytes, only have [%,d] buffered

What it means

copyFromQueuedChunks copies a requested number of bytes out of the buffered chunks. If fewer bytes are buffered than requested, it throws this IAE — an internal invariant failure, since callers should only request fully-buffered byte counts. It indicates a bug or external misuse (e.g., non-synchronized access) rather than a normal runtime condition.

Solutions

1. Ensure all interactions with the channel occur through its public thread-safe API, not by reaching into internals.
2. Check for concurrent access from multiple threads to the same channel instance.
3. Upgrade Druid if reproducible on a stock path (likely an internal bug to report).
4. Add logging around addChunk/read to capture the interleaving.

try { rac = channel.read(); } catch (IAE e) { if (e.getMessage().startsWith("Cannot copy")) { reportInternalBug(e); } else { throw e; } }

When it happens

Trigger: Calling copyFromQueuedChunks(n) when bytesBuffered < n — possible via direct misuse of the channel internals or a race where chunks were consumed/deleted before the copy.

Common situations: Custom code subclassing or wrapping the channel without holding lock; concurrent addChunk/deleteFromQueuedChunks interleaving due to broken external synchronization; Druid internal bugs.

Understand the failure class

Background: "This is a bug, please report it": internal invariant violations, unreachable panics, and SNH errors explained — this error's family across 47 libraries.

Related errors

• Cannot delete [%,d] bytes, only have [%,d] buffered
• bitmapEncoding[ ] does not match indexSpec.bitmap[ ]
• Cannot read frame of size [%,d] bytes
• Cleaned partition map
• Frame of size [%,d] not yet ready to read

AI-assisted analysis of apache/druid@9b90983fd2 (2026-09-07). Data as JSON: /api/errors/f9b1017c1eabdb63. Report an issue: GitHub.

Appendix: source

Thrown at processing/src/main/java/org/apache/druid/frame/channel/ReadableByteChunksFrameChannel.java:474

  @GuardedBy("lock")
  private boolean canReadError()
  {
    return chunks.size() > 0 && chunks.get(0).isError();
  }

  @GuardedBy("lock")
  private boolean canReadFrame()
  {
    return nextCompressedFrameLength != UNKNOWN_LENGTH
           && bytesBuffered >= FRAME_MARKER_AND_COMPRESSED_ENVELOPE_BYTES + nextCompressedFrameLength;
  }

  @GuardedBy("lock")
  private Memory copyFromQueuedChunks(final int numBytes)
  {
    if (bytesBuffered < numBytes) {
      throw new IAE("Cannot copy [%,d] bytes, only have [%,d] buffered", numBytes, bytesBuffered);
    }

    final WritableMemory buf = WritableMemory.allocate(numBytes, ByteOrder.LITTLE_ENDIAN);

    int bufPos = 0;
    for (int chunkNumber = 0; chunkNumber < chunks.size(); chunkNumber++) {
      final byte[] chunk = chunks.get(chunkNumber).valueOrThrow();
      final int chunkPosition = chunkNumber == 0 ? positionInFirstChunk : 0;
      final int len = Math.min(chunk.length - chunkPosition, numBytes - bufPos);

      buf.putByteArray(bufPos, chunk, chunkPosition, len);
      bufPos += len;

      if (bufPos == numBytes) {
        break;
      }
    }

View on GitHub (pinned to 9b90983fd2)