apache/druid · error · IllegalStateException

Invalid frame size (size = %,d B)

Error message

Invalid frame size (size = %,d B)

What it means

When a frame marker is read, the 8-byte length following it must be positive and below MAX_FRAME_SIZE_BYTES. A zero, negative, or oversized length means the stream is corrupted or misaligned, so the reader throws instead of attempting an unsafe allocation. This guards against corrupt or desynchronized frame headers.

Solutions

1. Regenerate the frame data; check the writer's max frame size configuration.
2. Verify stream/file integrity (no truncation or overwrite).
3. Ensure reader and writer formats/versions match.
4. If length is consistently >= MAX_FRAME_SIZE_BYTES, confirm MAX_FRAME_SIZE_BYTES is not exceeded by data written by an older version with different limits.

try { rac = channel.read(); } catch (ISE e) { if (e.getMessage().startsWith("Invalid frame size")) { failStage("corrupt or oversized frame header", e); } else { throw e; } }

When it happens

Trigger: Reading a frame length from the buffered header where nextCompressedFrameLength <= 0 or >= MAX_FRAME_SIZE_BYTES (approximately 2 GiB by default, the FrameFileWriter frame size limit).

Common situations: Frame files larger than the configured max frame size written by misconfigured producers; corrupted or truncated streams; reading at a wrong byte offset after a prior parse error; byte-order/format mismatch.

Understand the failure class

Background: "value must be between 0 and 1" / "out of range" / "must not be negative" errors: fixing range-validation failures across open-source libraries — this error's family across 42 libraries.

Related errors

• Cannot read frame of size [%,d] bytes
• Invalid midstream marker
• Invalid stream header
• Unexpected marker type
• Cannot copy [%,d] bytes, only have [%,d] buffered

AI-assisted analysis of apache/druid@9b90983fd2 (2026-09-07). Data as JSON: /api/errors/79f1e199938f87b0. Report an issue: GitHub.

Appendix: source

Thrown at processing/src/main/java/org/apache/druid/frame/channel/ReadableByteChunksFrameChannel.java:427

    }

    if (streamPart == StreamPart.FRAMES) {
      if (bytesBuffered >= Byte.BYTES) {
        final Memory memory = copyFromQueuedChunks(1);
        final byte markerByte = memory.getByte(0);

        if (markerByte == FrameFileWriter.MARKER_FRAME || markerByte == FrameFileWriter.MARKER_RAC) {
          // Read nextFrameLength if needed; otherwise do nothing.
          // Both MARKER_FRAME and MARKER_RAC use the same compression envelope format.
          final int bytesRequiredToReadLength = FRAME_MARKER_BYTES + FrameCompression.COMPRESSED_DATA_HEADER_SIZE;

          if (nextCompressedFrameLength == UNKNOWN_LENGTH && bytesBuffered >= bytesRequiredToReadLength) {
            nextMarkerType = markerByte;
            nextCompressedFrameLength = copyFromQueuedChunks(bytesRequiredToReadLength)
                .getLong(FRAME_MARKER_BYTES + Byte.BYTES /* Compression strategy byte */);

            if (nextCompressedFrameLength <= 0 || nextCompressedFrameLength >= MAX_FRAME_SIZE_BYTES) {
              throw new ISE("Invalid frame size (size = %,d B)", nextCompressedFrameLength);
            }
          }
        } else if (markerByte == FrameFileWriter.MARKER_NO_MORE_FRAMES) {
          streamPart = StreamPart.FOOTER;
          nextCompressedFrameLength = UNKNOWN_LENGTH;
          nextMarkerType = NO_MARKER;
        } else {
          throw new ISE("Invalid midstream marker (id = %s, position = %d)", id, bytesAdded - bytesBuffered);
        }
      }
    }

    if (streamPart == StreamPart.FOOTER) {
      if (bytesBuffered > 0) {
        // Footer is discarded: it isn't useful when reading frame files as streams. (It contains pointers
        // for random access of frames.)
        deleteFromQueuedChunks(bytesBuffered);
      }

View on GitHub (pinned to 9b90983fd2)