risingwavelabs/risingwave · error · BackupError

unexpected bytes after meta snapshot checksum

Error message

unexpected bytes after meta snapshot checksum

What it means

MetaSnapshot::finish verifies the trailing checksum then reads one extra byte to confirm nothing follows it; if the reader still returns bytes it throws "unexpected bytes after meta snapshot checksum". The snapshot stream contains trailing data beyond the defined format, so the object is not a well-formed meta snapshot.

Solutions

  1. Upgrade the RisingWave/risectl binary to a version matching (or newer, with V2 section skipping) the backup producer before restoring.
  2. Verify the snapshot object is exactly the writer's output (compare byte length/checksum from the backup manifest).
  3. Recreate the backup without post-processing (no compression-on-write tricks, no concatenation).
  4. If intentional forward-compat data is present, port the reader to skip unknown trailing sections like decode_hummock_sequences_from_stream does.

Example fix

// before: strict full read
snapshot.finish().await?;
// after: skip unknown trailing sections when tolerated, or ensure exact match
let n = reader.read(&mut trailing).await?;
if n != 0 && !allow_trailing { snapshot.finish().await?; }
Defensive patterns

Strategy: try-catch

Validate before calling

let size = storage.get_object_size(&key).await?;
let expected = manifest.snapshot_size; // written at backup time
if size != expected {
    return Err(anyhow!("snapshot size mismatch: {} != {}", size, expected));
}

Type guard

fn is_exact_size(actual: u64, expected: u64) -> bool { actual == expected }

Try / catch

match snapshot.finish().await {
    Ok(()) => {},
    Err(e) if e.to_string().contains("unexpected bytes after") => {
        log::warn!("trailing data in snapshot; producer likely newer version");
        // handle per policy: upgrade reader, or reject
    }
    Err(e) => return Err(e.into()),
}

Prevention

When it happens

Trigger: Calling finish() on a reader where bytes remain after the checksum — e.g. the snapshot object has appended/future-incompatible sections written by a newer tool, or the object wraps the snapshot with extra framing/headers.

Common situations: A newer RisingWave version wrote extra sections into the snapshot and an older binary tries to read it; an export script concatenated metadata after the snapshot; double-reading a stream that was not consumed exactly once; manual edits to a backup file.

Related errors

AI-assisted analysis of risingwavelabs/risingwave@6469eb736d (2026-09-11). Data as JSON: /api/errors/d10b5b203ef9ce05. Report an issue: GitHub.

Appendix: source

Thrown at src/storage/backup/src/meta_snapshot.rs:236

        if tail.len() != size_of::<u64>() {
            return Err(BackupError::Decoding(
                anyhow::anyhow!("meta snapshot is missing checksum").into(),
            ));
        }
        Self::verify_checksum_with_hasher(&hasher, &tail)
    }

    pub async fn finish(mut self) -> BackupResult<()> {
        let mut checksum = [0; size_of::<u64>()];
        self.reader.read_exact(&mut checksum).await?;
        self.verify_checksum(&checksum)?;

        let mut trailing = [0; 1];
        let n = self.reader.read(&mut trailing).await?;
        if n != 0 {
            return Err(BackupError::Decoding(
                anyhow::anyhow!("unexpected bytes after meta snapshot checksum").into(),
            ));
        }
        Ok(())
    }

    fn verify_checksum(&self, checksum: &[u8]) -> BackupResult<()> {
        Self::verify_checksum_with_hasher(&self.hasher, checksum)
    }

    fn verify_checksum_with_hasher(hasher: &XxHash64, checksum: &[u8]) -> BackupResult<()> {
        let expected = u64::from_le_bytes(checksum.try_into().expect("u64 length"));
        let found = hasher.finish();
        if expected != found {
            return Err(BackupError::ChecksumMismatch { expected, found });
        }
        Ok(())
    }
}

View on GitHub (pinned to 6469eb736d)