apache/druid · error · IllegalStateException

Cannot read frame of size [%,d] bytes

Error message

Cannot read frame of size [%,d] bytes

What it means

During frame decoding, the announced compressed frame length exceeds the maximum representable payload that can fit in an int-sized buffer after subtracting marker and compression-envelope overhead. This guard prevents an absurd or corrupted length field from being used to allocate/copy a huge buffer. It almost always indicates corrupted stream data or a bad length read from the stream.

Solutions

1. Verify the frame file/stream was produced by the same Druid version and format (FrameFileWriter MAGIC check passes).
2. Re-run the stage/query to regenerate the frame data; inspect underlying storage for corruption.
3. Check that the file was not truncated or modified after writing (compare sizes/checksums).
4. If reproducible, capture the id and frame header bytes and report with the full query log.

// regenerate data; no pre-call check available to users beyond trusting source integrity
if (!frameFile.exists() || frameFile.length() == 0) { throw new IllegalStateException("missing/incomplete frame file"); }

try { rac = channel.read(); } catch (ISE e) { if (e.getMessage().startsWith("Cannot read frame of size")) { failStage("corrupted frame stream", e); } else { throw e; } }

When it happens

Trigger: nextCompressedFrameLength (read from 8 bytes of the buffered frame header) is greater than Integer.MAX_VALUE - FRAME_MARKER_BYTES - FrameCompression.COMPRESSED_DATA_ENVELOPE_SIZE, i.e. a corrupt or misaligned frame header.

Common situations: Corrupted spooled frame files or inter-process streams (disk corruption, truncated/overwritten files); reading a stream written by an incompatible Druid version with a different frame format; stream desynchronization after earlier mis-parse.

Understand the failure class

Background: Schema validation failed / invalid input schema: payload rejected because its shape doesn't match the expected schema — this error's family across 28 libraries.

Related errors

• Invalid frame size (size = %,d B)
• Invalid midstream marker
• Invalid stream header
• Unexpected marker type
• Cannot copy [%,d] bytes, only have [%,d] buffered

AI-assisted analysis of apache/druid@9b90983fd2 (2026-09-07). Data as JSON: /api/errors/67a3e273ddd4e08a. Report an issue: GitHub.

Appendix: source

Thrown at processing/src/main/java/org/apache/druid/frame/channel/ReadableByteChunksFrameChannel.java:360

  long getBytesBuffered()
  {
    synchronized (lock) {
      return bytesBuffered;
    }
  }

  private RowsAndColumns nextRAC()
  {
    final Memory compressedMemory;
    final byte markerType;

    synchronized (lock) {
      if (!canReadFrame()) {
        throw new ISE("Frame of size [%,d] not yet ready to read", nextCompressedFrameLength);
      }

      if (nextCompressedFrameLength > Integer.MAX_VALUE - FRAME_MARKER_BYTES - FrameCompression.COMPRESSED_DATA_ENVELOPE_SIZE) {
        throw new ISE("Cannot read frame of size [%,d] bytes", nextCompressedFrameLength);
      }

      markerType = nextMarkerType;
      final int numBytes = Ints.checkedCast(FRAME_MARKER_AND_COMPRESSED_ENVELOPE_BYTES + nextCompressedFrameLength);
      compressedMemory = copyFromQueuedChunks(numBytes).region(
          FRAME_MARKER_BYTES,
          FRAME_MARKER_AND_COMPRESSED_ENVELOPE_BYTES + nextCompressedFrameLength - FRAME_MARKER_BYTES
      );
      deleteFromQueuedChunks(numBytes);
      updateStreamState();
    }

    final byte[] decompressedBytes =
        FrameCompression.decompress(compressedMemory, 0, compressedMemory.getCapacity());

    switch (markerType) {
      case FrameFileWriter.MARKER_FRAME:
        final Frame frame = Frame.wrap(decompressedBytes);

View on GitHub (pinned to 9b90983fd2)