apache/druid · error · IllegalStateException

Invalid stream header

Error message

Invalid stream header (id = %s, position = %d)

What it means

The channel expects a stream to begin with FrameFileWriter.MAGIC. If the first bytes buffered do not equal that magic, the stream is not a valid frame stream, so updateStreamState throws with the channel id and stream position. This detects reading a non-frame file or garbage stream.

Solutions

1. Verify the file/channel source is actually a frame file written by FrameFileWriter.
2. Check that the writer fully completed before the reader started (await writer completion).
3. Ensure both sides run compatible Druid versions.
4. Regenerate the data and check storage integrity.

if (!frameFile.exists() || frameFile.length() < FrameFileWriter.MAGIC.length) { throw new IllegalStateException("not a frame file: " + file); }

try { rac = channel.read(); } catch (ISE e) { if (e.getMessage().startsWith("Invalid stream header")) { failStage("source is not a frame stream", e); } else { throw e; } }

When it happens

Trigger: addChunk()/nextRAC() triggering updateStreamState while streamPart is HEADER and the first FrameFileWriter.MAGIC.length bytes do not match MAGIC.

Common situations: Pointing the channel at the wrong file (not a frame file); reading a partially written/zero-length file; version mismatch where the writer format changed; corrupted first block on disk.

Understand the failure class

Background: "Invalid ... format", "must be in format X", "does not look like a ..." — invalid argument format errors across CLI tools and libraries — this error's family across 17 libraries.

Related errors

• Cannot read frame of size [%,d] bytes
• Invalid frame size (size = %,d B)
• Invalid midstream marker
• Unexpected marker type
• Cannot copy [%,d] bytes, only have [%,d] buffered

AI-assisted analysis of apache/druid@9b90983fd2 (2026-09-07). Data as JSON: /api/errors/68648bbdaa84dd48. Report an issue: GitHub.

Appendix: source

Thrown at processing/src/main/java/org/apache/druid/frame/channel/ReadableByteChunksFrameChannel.java:406

        return rac;

      default:
        throw new ISE("Unexpected marker type[%d]", markerType);
    }
  }

  @GuardedBy("lock")
  private void updateStreamState()
  {
    if (streamPart == StreamPart.MAGIC) {
      if (bytesBuffered >= FrameFileWriter.MAGIC.length) {
        final Memory memory = copyFromQueuedChunks(FrameFileWriter.MAGIC.length);

        if (memory.equalTo(0, Memory.wrap(FrameFileWriter.MAGIC), 0, FrameFileWriter.MAGIC.length)) {
          streamPart = StreamPart.FRAMES;
          deleteFromQueuedChunks(FrameFileWriter.MAGIC.length);
        } else {
          throw new ISE("Invalid stream header (id = %s, position = %d)", id, bytesAdded - bytesBuffered);
        }
      }
    }

    if (streamPart == StreamPart.FRAMES) {
      if (bytesBuffered >= Byte.BYTES) {
        final Memory memory = copyFromQueuedChunks(1);
        final byte markerByte = memory.getByte(0);

        if (markerByte == FrameFileWriter.MARKER_FRAME || markerByte == FrameFileWriter.MARKER_RAC) {
          // Read nextFrameLength if needed; otherwise do nothing.
          // Both MARKER_FRAME and MARKER_RAC use the same compression envelope format.
          final int bytesRequiredToReadLength = FRAME_MARKER_BYTES + FrameCompression.COMPRESSED_DATA_HEADER_SIZE;

          if (nextCompressedFrameLength == UNKNOWN_LENGTH && bytesBuffered >= bytesRequiredToReadLength) {
            nextMarkerType = markerByte;
            nextCompressedFrameLength = copyFromQueuedChunks(bytesRequiredToReadLength)
                .getLong(FRAME_MARKER_BYTES + Byte.BYTES /* Compression strategy byte */);

View on GitHub (pinned to 9b90983fd2)