risingwavelabs/risingwave · error · BackupError

metadata JSON payload length exceeds usize

Error message

metadata JSON payload length exceeds usize

What it means

This error is thrown when reading the length prefix of a metadata JSON payload in a v2 metadata snapshot. The wire format stores the length as a u32 when it fits, or as a marker 0 followed by a full u64 for larger payloads. When the u64 form is used, its value must fit in usize; otherwise the payload length is meaningless on this platform and the snapshot is treated as corrupt.

Source

Thrown at src/storage/backup/src/meta_snapshot_v2.rs:339

async fn skip_metadata_list(reader: &mut SnapshotPayloadReader) -> BackupResult<()> {
    let n = reader.read_u32_le().await? as usize;
    for _ in 0..n {
        skip_with_len_prefix(reader).await?;
    }
    Ok(())
}

async fn skip_with_len_prefix(reader: &mut SnapshotPayloadReader) -> BackupResult<()> {
    let len = read_len_prefix(reader).await?;
    reader.skip_exact(len).await
}

async fn read_len_prefix(reader: &mut SnapshotPayloadReader) -> BackupResult<usize> {
    match reader.read_u32_le().await? {
        0 => {
            reader.read_u64_le().await?.try_into().map_err(|_| {
                BackupError::Other(anyhow!("metadata JSON payload length exceeds usize"))
            })
        }
        len => Ok(len as usize),
    }
}

async fn write_with_len_prefix<T: Serialize>(
    writer: &mut SnapshotPayloadWriter,
    data: &T,
) -> BackupResult<()> {
    let b = serde_json::to_vec(data)?;
    // Any valid JSON value serialized by serde_json is non-empty, such as `null`, `{}`, or `[]`.
    assert!(!b.is_empty());
    let len_prefix_len = if u32::try_from(b.len()).is_ok() {
        size_of::<u32>()
    } else {
        size_of::<u32>() + size_of::<u64>()
    };

View on GitHub (pinned to 6469eb736d)

Solutions

1. Verify the snapshot file integrity (checksum) against the backup manifest and re-upload/re-create the backup if it is corrupted
2. Ensure the payload reader is correctly positioned — a misaligned stream can make garbage bytes decode as a huge length; fix upstream parsing so offsets stay aligned
3. If running on a 32-bit platform, restore on a 64-bit host where realistic metadata sizes always fit in usize

fn validate_len_prefix(bytes: &[u8]) -> Result<usize, String> {
    if bytes.len() < 4 { return Err("buffer too short for u32 length".into()); }
    let prefix = u32::from_le_bytes(bytes[..4].try_into().unwrap());
    if prefix == 0 {
        if bytes.len() < 12 { return Err("buffer too short for u64 length".into()); }
        let len = u64::from_le_bytes(bytes[4..12].try_into().unwrap());
        usize::try_from(len).map_err(|_| "u64 length exceeds usize".to_string())
    } else {
        Ok(prefix as usize)
    }
}

fn fits_usize(len: u64) -> bool { usize::try_from(len).is_ok() }

When it happens

Trigger: read_len_prefix reads a u32 length-prefix of 0, then reads a u64 whose value exceeds usize::MAX (e.g. a corrupted or maliciously crafted snapshot file, or a truncated/misaligned stream where unrelated bytes are interpreted as a huge u64 length).

Common situations: Restoring from a corrupted or partially written backup snapshot file; byte-offset misalignment after earlier parse errors on 32-bit targets where u64 lengths never fit; hand-edited or bit-rotted snapshot payloads.

Understand the failure class

Background: "value must be between 0 and 1" / "out of range" / "must not be negative" errors: fixing range-validation failures across open-source libraries — this error's family across 42 libraries.

Related errors

• Encoding error: {0}
• Decoding error: {0}
• Checksum mismatch: expected {expected}, found: {found}
• metadata snapshot is truncated while reading u64
• metadata JSON payload length {} exceeds remaining buffer {}

AI-assisted analysis of risingwavelabs/risingwave@6469eb736d (2026-09-11). Data as JSON: /api/errors/248945da2ba397e4. Report an issue: GitHub.