{"record":{"id":"9089133254196a0d","repo":"risingwavelabs/risingwave","slug":"checkpoint-checksum-mismatch-expected-x-got","errorCode":null,"errorMessage":"checkpoint checksum mismatch: expected {:#x}, got {:#x}","messagePattern":"checkpoint checksum mismatch: expected (.+?), got (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"critical","filePath":"src/meta/src/hummock/manager/checkpoint.rs","lineNumber":133,"sourceCode":"/// 1. Try to decode as `HummockVersionCheckpointEnvelope`\n/// 2. If `checksum.is_some()`:\n///    - Verify xxhash64 checksum\n///    - Decompress payload according to `compression_algorithm`\n///    - Decode decompressed bytes as `PbHummockVersionCheckpoint`\n/// 3. If decode fails or `checksum.is_none()`:\n///    - Decode bytes directly as legacy `PbHummockVersionCheckpoint`\nfn decode_checkpoint_data(data: bytes::Bytes) -> Result<PbHummockVersionCheckpoint> {\n    use anyhow::Context;\n    use prost::Message;\n\n    let data_size = data.len();\n\n    if let Ok(envelope) = PbHummockVersionCheckpointEnvelope::decode(data.clone())\n        && let Some(expected) = envelope.checksum\n    {\n        let actual = xxhash64_checksum(&envelope.payload);\n        if actual != expected {\n            return Err(anyhow::anyhow!(\n                \"checkpoint checksum mismatch: expected {:#x}, got {:#x}\",\n                expected,\n                actual\n            )\n            .into());\n        }\n\n        let algo = CheckpointCompressionAlgorithm::try_from(envelope.compression_algorithm)\n            .with_context(|| {\n                format!(\n                    \"unknown checkpoint compression algorithm: {}\",\n                    envelope.compression_algorithm\n                )\n            })?;\n\n        let decompressed = decompress_payload(algo, &envelope.payload)?;\n        let ckpt = PbHummockVersionCheckpoint::decode(decompressed.as_ref())\n            .context(\"failed to decode checkpoint envelope payload\")?;","sourceCodeStart":115,"sourceCodeEnd":151,"githubUrl":"https://github.com/risingwavelabs/risingwave/blob/6469eb736d691e8e9b8a419a57edd6429ca77417/src/meta/src/hummock/manager/checkpoint.rs#L115-L151","documentation":"While decoding a Hummock version checkpoint, the stored envelope's xxhash64 checksum is compared against a freshly computed checksum of its payload. A mismatch means the checkpoint bytes are corrupted or were written by an incompatible writer, so decode aborts rather than trusting the data.","triggerScenarios":"decode_checkpoint_data reads an envelope from the object store whose checksum field differs from the computed checksum of envelope.payload — i.e. bit rot, truncated write, or tampered/partially-updated object.","commonSituations":"Object store data corruption or incomplete upload; restore from an inconsistent backup; modifying checkpoint data with an older/newer tool that doesn't preserve checksums; disk errors on the object store side.","solutions":["Verify the object in the object store (re-upload/restore from a healthy backup)","Check object store integrity (ETag/checksum features) and disk health","Rebuild state by re-checkpointing from a healthy meta node","If persistent, treat the checkpoint as lost and restore meta state from a previous consistent snapshot"],"exampleFix":null,"handlingStrategy":"try-catch","validationCode":null,"typeGuard":"fn is_checksum_mismatch(err: &anyhow::Error) -> bool {\n    err.to_string().contains(\"checkpoint checksum mismatch\")\n}\n","tryCatchPattern":"match try_read_checkpoint(object_store, id).await {\n    Err(e) if e.to_string().contains(\"checkpoint checksum mismatch\") => {\n        tracing::error!(\"corrupt checkpoint for {id}: {e:#}; restore from backup\");\n        // fall back to previous checkpoint / re-bootstrap meta state\n    }\n    other => other?,\n}\n","preventionTips":["Enable object-store integrity features (checksums/ETags)","Verify backups before restoring meta state","Never edit checkpoint objects in place","Monitor hardware/disk health of the object store"],"tags":["rust","hummock","checkpoint","integrity"],"backgroundTag":"checksum-mismatch","analyzedSha":"6469eb736d691e8e9b8a419a57edd6429ca77417","analyzedAt":"2026-09-11T21:06:21.487Z","contentChangedAt":"2026-09-11T21:06:21.487Z","schemaVersion":2},"datasetVersion":"2026-09-23T08:17:48.524Z"}