risingwavelabs/risingwave · error · BackupError

prost payload length

Error message

prost payload length {} exceeds remaining buffer {}

What it means

decode_prost_message reads a u32 length prefix then requires that many bytes in the buffer; if buf.remaining() < len it throws "prost payload length {} exceeds remaining buffer {}". The declared protobuf message size exceeds the data actually available, so the snapshot is truncated or its length field is corrupt/foreign.

Solutions

  1. Restore from an intact backup — compare stored object size and checksum with the manifest.
  2. Verify the parse offset: ensure read_snapshot_header and prior section skips consumed the correct byte counts before this decode.
  3. Check the snapshot format_version matches the decoder version in use.
  4. Rebuild test payloads with the writer helpers so length prefixes are consistent with the payload.

Example fix

// before: decoding from a mis-offset buffer
let len = read_u32_le(buf)? as usize;
let v = buf[..len].to_vec();
// after: keep offsets in sync with section header
let section = read_section_with_header(buf)?; // consumes length too
let msg = decode_prost_message::<Model>(&mut &section[..])?;
Defensive patterns

Strategy: validation

Validate before calling

// ensure the declared length fits before decoding a prost message
fn fits(buf: &[u8], declared: usize) -> bool { buf.len() >= declared }
// e.g. after reading len: if !fits(&buf[..], len) { bail!("truncated protobuf section"); }

Type guard

fn has_payload(buf: &[u8], len: usize) -> bool { buf.len() >= len }

Try / catch

match decode_section(&mut buf) {
    Ok(model) => model,
    Err(e) if e.to_string().contains("exceeds remaining buffer") => {
        log::error!("snapshot truncated or framing desynced at offset");
        return Err(anyhow!("snapshot protobuf section truncated"));
    }
    Err(e) => return Err(e.into()),
}

Prevention

When it happens

Trigger: Decoding any prost-encoded metadata list inside the snapshot (tables, hummock sequences, system parameters) when the section bytes end before the declared length — corrupted buffer, truncated object, or misparsed framing causing a bogus length.

Common situations: A backup object truncated mid-section; a snapshot body parsed with a wrong offset so random bytes become a huge length prefix; endianness/framing mismatch from hand-crafted test data; older-format sections decoded by a newer reader.

Understand the failure class

Background: "cannot parse invalid wire-format data", "cannot unmarshal", "failed unmarshalling": protobuf unmarshal errors explained — this error's family across 10 libraries.

Related errors

AI-assisted analysis of risingwavelabs/risingwave@6469eb736d (2026-09-11). Data as JSON: /api/errors/7d9e7ae0a9a0b26e. Report an issue: GitHub.

Appendix: source

Thrown at src/storage/backup/src/meta_snapshot_v1.rs:238

            cluster_id,
            subscription,
            secret,
        })
    }

    fn encode_prost_message(message: &impl prost::Message, buf: &mut impl BufMut) {
        let encoded_message = message.encode_to_vec();
        buf.put_u32_le(encoded_message.len() as u32);
        buf.put_slice(&encoded_message);
    }

    fn decode_prost_message<T>(buf: &mut &[u8]) -> BackupResult<T>
    where
        T: prost::Message + Default,
    {
        let len = read_u32_le(buf)? as usize;
        if buf.remaining() < len {
            return Err(BackupError::Other(anyhow::anyhow!(
                "prost payload length {} exceeds remaining buffer {}",
                len,
                buf.remaining()
            )));
        }
        let v = buf[..len].to_vec();
        buf.advance(len);
        T::decode(v.as_slice()).map_err(|e| BackupError::Decoding(e.into()))
    }

    fn encode_prost_message_list(messages: &[&impl prost::Message], buf: &mut impl BufMut) {
        buf.put_u32_le(messages.len() as u32);
        for message in messages {
            Self::encode_prost_message(*message, buf);
        }
    }

    fn decode_prost_message_list<T>(buf: &mut &[u8]) -> BackupResult<Vec<T>>

View on GitHub (pinned to 6469eb736d)