risingwavelabs/risingwave · critical · HummockError

Checksum mismatch: expected

Error message

Checksum mismatch: expected {expected}, found: {found}

What it means

Hummock verified a data block's checksum after reading it from the object store and the computed value did not match the stored expected value. This indicates the data bytes were corrupted in transit or at rest. The library throws it to prevent serving corrupt data.

Solutions

  1. Delete/redownload the affected SST and let Hummock re-fetch from the object store.
  2. Verify object integrity in the object store (re-upload from replica or restore from backup) for the listed file.
  3. Check hardware health (disk SMART, memory) and network path if corruption recurs.
  4. Full recovery: restore the cluster's data from backup/snapshot and rebuild the affected tables.

Example fix

// before: keeping suspect object
aws s3 cp s3://bucket/hummock_xx ./corrupt --no-verify
// after
aws s3api head-object s3://bucket/hummock_xx  # compare checksum, then delete object and recreate table from backup
Defensive patterns

Strategy: try-catch

Try / catch

// Rust
match hummock_read(...) {
    Err(e) if e.to_string().starts_with("Checksum mismatch") => {
        // alert on corruption; do not blindly retry same object — delete & recover
    }
    r => r?,
}

Prevention

When it happens

Trigger: Reading a block from an SST (via BlockHeader/checksum validation) where computed checksum != stored expected: expected u64 vs found u64.

Common situations: Faulty disk or S3 object corruption; network bit-flips without TLS/verified uploads; a truncated or partially written SST; manual tampering with object store contents.

Understand the failure class

Background: Checksum mismatch errors: "checksum verification failed", "digest mismatch", "expected vs actual checksum" — what they mean and how to fix them — this error's family across 41 libraries.

Related errors

AI-assisted analysis of risingwavelabs/risingwave@6469eb736d (2026-09-11). Data as JSON: /api/errors/b633ddeefe7389f6. Report an issue: GitHub.

Appendix: source

Thrown at src/storage/src/hummock/error.rs:29

// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.

use risingwave_object_store::object::ObjectError;
use risingwave_pb::id::TableId;
use thiserror::Error;
use thiserror_ext::AsReport;
use tokio::sync::oneshot::error::RecvError;

// TODO(error-handling): should prefer use error types than strings.
#[derive(Error, thiserror_ext::ReportDebug, thiserror_ext::Arc)]
#[thiserror_ext(newtype(name = HummockError, backtrace))]
pub enum HummockErrorInner {
    #[error("Magic number mismatch: expected {expected}, found: {found}")]
    MagicMismatch { expected: u32, found: u32 },
    #[error("Invalid format version: {0}")]
    InvalidFormatVersion(u32),
    #[error("Checksum mismatch: expected {expected}, found: {found}")]
    ChecksumMismatch { expected: u64, found: u64 },
    #[error("Invalid block")]
    InvalidBlock,
    #[error("Encode error: {0}")]
    EncodeError(String),
    #[error("Decode error: {0}")]
    DecodeError(String),
    #[error("ObjectStore failed with IO error: {0}")]
    ObjectIoError(
        #[from]
        #[backtrace]
        ObjectError,
    ),
    #[error("Meta error: {0}")]
    MetaError(String),
    #[error("SharedBuffer error: {0}")]
    SharedBufferError(String),
    #[error("Wait epoch error: {0}")]

View on GitHub (pinned to 6469eb736d)