quickwit-oss/quickwit · error

serializing PartialHit should never fail

Error message

serializing PartialHit should never fail

What it means

ScrollKeyAndStartOffset's Display impl serializes the embedded PartialHit (search_after) into a fixed 28-byte payload buffer via serde_json::to_writer and expects success, then base64-encodes it. Since PartialHit contains only simple serializable fields (numbers/strings), serialization is infallible in practice; the expect asserts this invariant. A panic indicates a payload buffer too small for the JSON or a PartialHit containing a non-serializable type after a schema change.

Source

Thrown at quickwit/quickwit-search/src/scroll_context.rs:231

            self.max_hits_per_page = 0;
        }
        self.search_after = last_hit;
        self
    }

    pub fn scroll_key(&self) -> [u8; 16] {
        u128::from(self.scroll_ulid).to_le_bytes()
    }
}

impl fmt::Display for ScrollKeyAndStartOffset {
    fn fmt(&self, formatter: &mut fmt::Formatter) -> fmt::Result {
        let mut payload = vec![0u8; 28];
        payload[..16].copy_from_slice(&u128::from(self.scroll_ulid).to_le_bytes());
        payload[16..24].copy_from_slice(&self.start_offset.to_le_bytes());
        payload[24..28].copy_from_slice(&self.max_hits_per_page.to_le_bytes());
        serde_json::to_writer(&mut payload, &self.search_after)
            .expect("serializing PartialHit should never fail");
        let b64_payload = BASE64_STANDARD.encode(payload);
        write!(formatter, "{b64_payload}")
    }
}

impl FromStr for ScrollKeyAndStartOffset {
    type Err = &'static str;

    fn from_str(scroll_id_str: &str) -> Result<Self, Self::Err> {
        let base64_decoded: Vec<u8> = BASE64_STANDARD
            .decode(scroll_id_str)
            .map_err(|_| "scroll id is invalid base64.")?;
        if base64_decoded.len() <= 16 + 8 + 4 {
            return Err("scroll id payload is truncated");
        }
        let (scroll_ulid_bytes, from_bytes, max_hits_bytes) = (
            &base64_decoded[..16],
            &base64_decoded[16..24],

View on GitHub (pinned to a39730c5cd)

Solutions

1. If hit, increase the payload buffer: serialize to a Vec first (serde_json::to_vec) instead of a fixed 28-byte array.
2. Audit recent PartialHit struct changes for new fields that grow or break JSON serialization.
3. Better: replace the fixed-size buffer with to_vec + extend so capacity is always sufficient.

Example fix

// before
let mut payload = vec![0u8; 28];
...
serde_json::to_writer(&mut payload, &self.search_after)
    .expect("serializing PartialHit should never fail");
// after
let mut payload = vec![0u8; 28];
...
serde_json::to_writer(&mut payload, &self.search_after)
    .expect("serializing PartialHit should never fail");
// (if PartialHit grows, switch to)
let json = serde_json::to_vec(&self.search_after)
    .expect("serializing PartialHit should never fail");
payload.extend_from_slice(&json);

let json = serde_json::to_vec(&search_after)?;
assert!(json.len() <= 28, "search_after JSON exceeds scroll key buffer");

When it happens

Trigger: Formatting a ScrollKeyAndStartOffset (scroll key) where the JSON encoding of search_after exceeds the 28-byte buffer capacity beyond the 24 reserved bytes, or where search_after contains values serde_json cannot write (theoretically impossible with current PartialHit types).

Common situations: Essentially unreachable with current code; would appear after someone widens PartialHit (e.g. larger sort values, nested objects) without growing the payload buffer, surfacing as a panic when a scroll/next-page key is rendered into a search response.

Understand the failure class

Background: json.Marshal / "failed to marshal" errors in Go: why "unsupported type" happens and how to fix it — this error's family across 22 libraries.

Related errors

• `doc_batch` should not be empty
• node not found in pending
• OTP logs or traces do not support VRL transforms
• position of a Kafka partition should never be EOF
• position of a Kinesis shard should never be EOF

AI-assisted analysis of quickwit-oss/quickwit@a39730c5cd (2026-09-08). Data as JSON: /api/errors/6814acd09c1f5642. Report an issue: GitHub.