{"record":{"id":"a03e9f4f46214fab","repo":"apache/druid","slug":"invalid-midstream-marker-id-s-position-d","errorCode":null,"errorMessage":"Invalid midstream marker (id = %s, position = %d)","messagePattern":"Invalid midstream marker \\(id = (.+?), position = (.+?)\\)","errorType":"exception","errorClass":"IllegalStateException","httpStatus":null,"severity":"error","filePath":"processing/src/main/java/org/apache/druid/frame/channel/ReadableByteChunksFrameChannel.java","lineNumber":435,"sourceCode":"          // Read nextFrameLength if needed; otherwise do nothing.\n          // Both MARKER_FRAME and MARKER_RAC use the same compression envelope format.\n          final int bytesRequiredToReadLength = FRAME_MARKER_BYTES + FrameCompression.COMPRESSED_DATA_HEADER_SIZE;\n\n          if (nextCompressedFrameLength == UNKNOWN_LENGTH && bytesBuffered >= bytesRequiredToReadLength) {\n            nextMarkerType = markerByte;\n            nextCompressedFrameLength = copyFromQueuedChunks(bytesRequiredToReadLength)\n                .getLong(FRAME_MARKER_BYTES + Byte.BYTES /* Compression strategy byte */);\n\n            if (nextCompressedFrameLength <= 0 || nextCompressedFrameLength >= MAX_FRAME_SIZE_BYTES) {\n              throw new ISE(\"Invalid frame size (size = %,d B)\", nextCompressedFrameLength);\n            }\n          }\n        } else if (markerByte == FrameFileWriter.MARKER_NO_MORE_FRAMES) {\n          streamPart = StreamPart.FOOTER;\n          nextCompressedFrameLength = UNKNOWN_LENGTH;\n          nextMarkerType = NO_MARKER;\n        } else {\n          throw new ISE(\"Invalid midstream marker (id = %s, position = %d)\", id, bytesAdded - bytesBuffered);\n        }\n      }\n    }\n\n    if (streamPart == StreamPart.FOOTER) {\n      if (bytesBuffered > 0) {\n        // Footer is discarded: it isn't useful when reading frame files as streams. (It contains pointers\n        // for random access of frames.)\n        deleteFromQueuedChunks(bytesBuffered);\n      }\n\n      assert bytesBuffered == 0 && chunks.isEmpty() && nextCompressedFrameLength == UNKNOWN_LENGTH;\n    }\n\n    if (addChunkBackpressureFuture != null && (bytesBuffered < bytesLimit || !canReadFrame())) {\n      // Release backpressure.\n      addChunkBackpressureFuture.set(null);\n      addChunkBackpressureFuture = null;","sourceCodeStart":417,"sourceCodeEnd":453,"githubUrl":"https://github.com/apache/druid/blob/9b90983fd291f26935af934383ce360473179e4d/processing/src/main/java/org/apache/druid/frame/channel/ReadableByteChunksFrameChannel.java#L417-L453","documentation":"In the FRAMES stream part, every byte read at a marker position must be a valid frame marker (a data frame marker or MARKER_NO_MORE_FRAMES). Any other byte indicates the stream is not aligned to frame boundaries, so the reader throws with the channel id and byte position. This catches desynchronization or corruption mid-stream.","triggerScenarios":"updateStreamState reads a marker byte in StreamPart.FRAMES that is neither a frame marker nor FrameFileWriter.MARKER_NO_MORE_FRAMES.","commonSituations":"Corrupted spool files on disk; two writers appending to the same stream; wrong file passed to ReadableByteChunksFrameChannel; byte offset drift due to earlier skipped/failed read.","solutions":["Regenerate the stream/file and retry the stage.","Confirm only one writer owns the channel and files aren't shared across tasks.","Check disk/filesystem integrity for the spool directory.","Compare Druid versions of writer and reader."],"exampleFix":null,"handlingStrategy":"try-catch","validationCode":null,"typeGuard":null,"tryCatchPattern":"try { rac = channel.read(); } catch (ISE e) { if (e.getMessage().startsWith(\"Invalid midstream marker\")) { failStage(\"frame stream misaligned\", e); } else { throw e; } }","preventionTips":["Single writer per stream file; never append from multiple tasks.","Read frames strictly through the channel API to keep byte alignment.","Verify spool storage integrity periodically."],"tags":["druid","frame-channel","corruption"],"backgroundTag":"invalid-argument-format","analyzedSha":"9b90983fd291f26935af934383ce360473179e4d","analyzedAt":"2026-09-07T13:32:30.957Z","contentChangedAt":"2026-09-07T13:32:30.957Z","schemaVersion":2},"datasetVersion":"2026-09-17T15:17:12.973Z"}