risingwavelabs/risingwave · critical

checkpoint checksum mismatch: expected

Error message

checkpoint checksum mismatch: expected {:#x}, got {:#x}

What it means

While decoding a Hummock version checkpoint, the stored envelope's xxhash64 checksum is compared against a freshly computed checksum of its payload. A mismatch means the checkpoint bytes are corrupted or were written by an incompatible writer, so decode aborts rather than trusting the data.

Solutions

  1. Verify the object in the object store (re-upload/restore from a healthy backup)
  2. Check object store integrity (ETag/checksum features) and disk health
  3. Rebuild state by re-checkpointing from a healthy meta node
  4. If persistent, treat the checkpoint as lost and restore meta state from a previous consistent snapshot
Defensive patterns

Strategy: try-catch

Type guard

fn is_checksum_mismatch(err: &anyhow::Error) -> bool {
    err.to_string().contains("checkpoint checksum mismatch")
}

Try / catch

match try_read_checkpoint(object_store, id).await {
    Err(e) if e.to_string().contains("checkpoint checksum mismatch") => {
        tracing::error!("corrupt checkpoint for {id}: {e:#}; restore from backup");
        // fall back to previous checkpoint / re-bootstrap meta state
    }
    other => other?,
}

Prevention

When it happens

Trigger: decode_checkpoint_data reads an envelope from the object store whose checksum field differs from the computed checksum of envelope.payload — i.e. bit rot, truncated write, or tampered/partially-updated object.

Common situations: Object store data corruption or incomplete upload; restore from an inconsistent backup; modifying checkpoint data with an older/newer tool that doesn't preserve checksums; disk errors on the object store side.

Understand the failure class

Background: Checksum mismatch errors: "checksum verification failed", "digest mismatch", "expected vs actual checksum" — what they mean and how to fix them — this error's family across 41 libraries.

Related errors

AI-assisted analysis of risingwavelabs/risingwave@6469eb736d (2026-09-11). Data as JSON: /api/errors/9089133254196a0d. Report an issue: GitHub.

Appendix: source

Thrown at src/meta/src/hummock/manager/checkpoint.rs:133

/// 1. Try to decode as `HummockVersionCheckpointEnvelope`
/// 2. If `checksum.is_some()`:
///    - Verify xxhash64 checksum
///    - Decompress payload according to `compression_algorithm`
///    - Decode decompressed bytes as `PbHummockVersionCheckpoint`
/// 3. If decode fails or `checksum.is_none()`:
///    - Decode bytes directly as legacy `PbHummockVersionCheckpoint`
fn decode_checkpoint_data(data: bytes::Bytes) -> Result<PbHummockVersionCheckpoint> {
    use anyhow::Context;
    use prost::Message;

    let data_size = data.len();

    if let Ok(envelope) = PbHummockVersionCheckpointEnvelope::decode(data.clone())
        && let Some(expected) = envelope.checksum
    {
        let actual = xxhash64_checksum(&envelope.payload);
        if actual != expected {
            return Err(anyhow::anyhow!(
                "checkpoint checksum mismatch: expected {:#x}, got {:#x}",
                expected,
                actual
            )
            .into());
        }

        let algo = CheckpointCompressionAlgorithm::try_from(envelope.compression_algorithm)
            .with_context(|| {
                format!(
                    "unknown checkpoint compression algorithm: {}",
                    envelope.compression_algorithm
                )
            })?;

        let decompressed = decompress_payload(algo, &envelope.payload)?;
        let ckpt = PbHummockVersionCheckpoint::decode(decompressed.as_ref())
            .context("failed to decode checkpoint envelope payload")?;

View on GitHub (pinned to 6469eb736d)